jeff-regier / Celeste.jl

Scalable inference for a generative model of astronomical images
MIT License
183 stars 28 forks source link

Improve scores on synthetic imagery #711

Closed jeff-regier closed 6 years ago

jeff-regier commented 6 years ago

Scores following this PR, for a synthetic catalog and synthetic imagery:

Predictions:

│ Row │ N   │ first     │ field           │
├─────┼─────┼───────────┼─────────────────┤
│ 1   │ 131 │ 0.0610687 │ missed_stars    │
│ 2   │ 347 │ 0.037464  │ missed_galaxies │
│ 3   │ 478 │ 0.144662  │ position        │
│ 4   │ 478 │ 0.143964  │ flux_r_mag      │
│ 5   │ 478 │ 0.309982  │ flux_r_nmgy     │
│ 6   │ 50  │ 8.31035   │ gal_angle_deg   │
│ 7   │ 94  │ 0.141638  │ gal_frac_dev    │
│ 8   │ 94  │ 0.140924  │ gal_axis_ratio  │
│ 9   │ 94  │ 0.624384  │ gal_radius_px   │
│ 10  │ 478 │ 0.406908  │ color_ug        │
│ 11  │ 478 │ 0.156452  │ color_gr        │
│ 12  │ 478 │ 0.124973  │ color_ri        │
│ 13  │ 478 │ 0.206888  │ color_iz        │

Uncertainty quantification:

│ Row │ field           │ within_half_sd │ within_1_sd │ within_2_sd │ within_3_sd │
├─────┼─────────────────┼────────────────┼─────────────┼─────────────┼─────────────┤
│ 1   │ color_gr        │ 0.267223       │ 0.438413    │ 0.7119      │ 0.845511    │
│ 2   │ color_iz        │ 0.292276       │ 0.542797    │ 0.822547    │ 0.924843    │
│ 3   │ color_ri        │ 0.223382       │ 0.43215     │ 0.693111    │ 0.830898    │
│ 4   │ color_ug        │ 0.240084       │ 0.498956    │ 0.793319    │ 0.889353    │
│ 5   │ log_flux_r_nmgy │ 0.183716       │ 0.329854    │ 0.51357     │ 0.686848    │

@andymiller Here are the commands I used to generate these tables, all executed in the benchmark/accuracy directory:

$ ./write_ground_truth_catalog_csv.jl prior
$ ./generate_synthetic_field.jl 4263 5 119 output/prior_<hash1>.csv 
$ export JULIA_NUM_THREADS=4
$ ./run_celeste_on_field.jl --images-jld output/prior_<hash1>_synthetic_<hash2>.jld output/prior_<hash1>.csv --use-full-initialization
$ ./score_predictions.jl output/prior_<hash1>.csv --use-full-initialization output/prior_<hash1>_synthetic_<hash2>_predictions_<hash3>.csv
$ ./score_uncertainty.jl output/prior_<hash1>.csv output/prior_<hash1>_synthetic_<hash2>_predictions_<hash3>.csv