Match objects by position, not objid, in accuracy benchmarks

[codecov-io]

Codecov Report

Merging #693 into master will increase coverage by 0.31%. The diff coverage is 90%.

@@            Coverage Diff             @@
##           master     #693      +/-   ##
==========================================
+ Coverage   73.59%   73.91%   +0.31%     
==========================================
  Files          36       36              
  Lines        3833     3822      -11     
==========================================
+ Hits         2821     2825       +4     
+ Misses       1012      997      -15

Impacted Files	Coverage Δ
src/AccuracyBenchmark.jl	43.28% <90%> (+2.33%)	:arrow_up:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 5ff7b0d...a833d0d. Read the comment docs.

[jeff-regier]

Do the scores from score_predictions.jl stay nearly the same with this PR?

[kbarbary]

master

Reading 'output/coadd_for_4263_5_119_979ddc39b6.csv'...
Reading 'output/sdss_4263_5_119_primary_5490a686bb.csv'...
Reading 'output/sdss_4263_5_119_predictions_555dc86d0b.csv'...
490 objids in common
14×6 DataFrames.DataFrame
│ Row │ N   │ first     │ second    │ diff       │ diff_sd    │ field                         │
├─────┼─────┼───────────┼───────────┼────────────┼────────────┼───────────────────────────────┤
│ 1   │ 487 │ 0.0       │ 0.0       │ 0.0        │ 0.0        │ is_saturated                  │
│ 2   │ 130 │ 0.0769231 │ 0.338462  │ -0.261538  │ 0.0411744  │ missed_stars                  │
│ 3   │ 357 │ 0.0392157 │ 0.0364146 │ 0.00280112 │ 0.00992791 │ missed_galaxies               │
│ 4   │ 487 │ 0.268566  │ 0.331695  │ -0.0631288 │ 0.00943514 │ position                      │
│ 5   │ 487 │ 0.181593  │ 0.259222  │ -0.0776288 │ 0.0243781  │ reference_band_flux_mag       │
│ 6   │ 487 │ 1.15023   │ 5.9793    │ -4.82907   │ 2.26044    │ reference_band_flux_nmgy      │
│ 7   │ 106 │ 17.1374   │ 15.5699   │ 1.56745    │ 1.32904    │ angle_deg                     │
│ 8   │ 213 │ 0.263027  │ 0.188879  │ 0.0741483  │ 0.0217715  │ de_vaucouleurs_mixture_weight │
│ 9   │ 213 │ 0.201823  │ 0.147634  │ 0.0541886  │ 0.0102885  │ minor_major_axis_ratio        │
│ 10  │ 213 │ 1.30143   │ 0.75272   │ 0.548707   │ 0.336256   │ half_light_radius_px          │
│ 11  │ 375 │ 1.02576   │ 0.579212  │ 0.446548   │ 0.049141   │ color_log_ratio_ug            │
│ 12  │ 478 │ 0.340041  │ 0.177282  │ 0.162759   │ 0.019695   │ color_log_ratio_gr            │
│ 13  │ 486 │ 0.203482  │ 0.125886  │ 0.0775955  │ 0.00996649 │ color_log_ratio_ri            │
│ 14  │ 482 │ 0.389627  │ 0.18643   │ 0.203197   │ 0.0217315  │ color_log_ratio_iz            │

This PR

Reading 'output_new/coadd_for_4263_5_119_979ddc39b6.csv'...
Reading 'output_new/sdss_4263_5_119_primary_77cda8e3ae.csv'...
Reading 'output_new/sdss_4263_5_119_predictions_fdf3b75d01.csv'...
14×6 DataFrames.DataFrame
│ Row │ N   │ first     │ second    │ diff       │ diff_sd    │ field                         │
├─────┼─────┼───────────┼───────────┼────────────┼────────────┼───────────────────────────────┤
│ 1   │ 477 │ 0.0       │ 0.0       │ 0.0        │ 0.0        │ is_saturated                  │
│ 2   │ 129 │ 0.0775194 │ 0.333333  │ -0.255814  │ 0.0411565  │ missed_stars                  │
│ 3   │ 348 │ 0.0402299 │ 0.0373563 │ 0.00287356 │ 0.0101801  │ missed_galaxies               │
│ 4   │ 477 │ 0.265489  │ 0.307073  │ -0.0415845 │ 0.00551372 │ position                      │
│ 5   │ 477 │ 0.18019   │ 0.218313  │ -0.0381237 │ 0.0193626  │ reference_band_flux_mag       │
│ 6   │ 477 │ 1.13582   │ 3.08536   │ -1.94953   │ 0.765904   │ reference_band_flux_nmgy      │
│ 7   │ 101 │ 16.7389   │ 15.34     │ 1.39893    │ 1.31563    │ angle_deg                     │
│ 8   │ 206 │ 0.258738  │ 0.190058  │ 0.0686794  │ 0.0220703  │ de_vaucouleurs_mixture_weight │
│ 9   │ 206 │ 0.202108  │ 0.139348  │ 0.0627607  │ 0.0103371  │ minor_major_axis_ratio        │
│ 10  │ 206 │ 1.30795   │ 0.63601   │ 0.671941   │ 0.340124   │ half_light_radius_px          │
│ 11  │ 367 │ 1.0285    │ 0.582329  │ 0.446171   │ 0.0500921  │ color_log_ratio_ug            │
│ 12  │ 468 │ 0.340985  │ 0.175967  │ 0.165018   │ 0.0199242  │ color_log_ratio_gr            │
│ 13  │ 476 │ 0.201283  │ 0.122948  │ 0.0783353  │ 0.0101615  │ color_log_ratio_ri            │
│ 14  │ 472 │ 0.387015  │ 0.185587  │ 0.201428   │ 0.0220689  │ color_log_ratio_iz            │

[jeff-regier]

Maybe a few filters got emitted? I filter out super bright light sources, for example, even if they aren't marked as saturated. (They're often saturated, just not flagged correctly by sdss.)

[kbarbary]

Sorry, the last comparison was wrong because I wasn't comparing apples to apples: master was run with --full-initialization whereas this PR was not. I've updated the comparison so that both are run without --full-initialization. They largely agree now. See my last post.

If anything, this PR is better, probably due to having 10 fewer matching objects. There are fewer because before, we were matching primary to coadd based on position, then matching based on object ID. The only spatial requirement then was that primary and coadd match within 1 pixel. In this PR, we have a requirement that both primary and prediction match coadd to within 1 pixel, so it's a more stringent requirement. So this all makes sense to me.

[jeff-regier]

Sounds good. Please merge this once the tests pass.