search

dirac-institute / sorcha

An open-source community LSST Solar System Simulator
Other
15 stars 16 forks source link

Check that zero-output results are consistent #986

Closed Gerenjie closed 1 month ago

Gerenjie commented 1 month ago

Zero-row outputs are generated if bright detections are made but don't survive the linking filter, but don't get made if no bright detections are made. Now that I think of it, doesn't the latest version of Sorcha have the linking filter add a column rather than filter detections? Not sure if this will be reproduced on the latest version of Sorcha, sorry... Anyway, the discrepancy I saw follows.

2024-07-28 01:20:34,832 sorcha.sorcha INFO Number of rows BEFORE applying bright limit filter 0 2024-07-28 01:20:34,833 sorcha.sorcha INFO Number of rows AFTER applying bright limit filter 0 2024-07-28 01:20:34,833 sorcha.sorcha INFO No observations left in chunk. Skipping to next chunk... 2024-07-28 01:20:34,833 sorcha.sorcha INFO Sorcha process is completed. versus 2024-07-28 01:21:13,175 sorcha.sorcha INFO Number of rows BEFORE applying bright limit filter 1 2024-07-28 01:21:13,176 sorcha.sorcha INFO Number of rows AFTER applying bright limit filter 1 2024-07-28 01:21:13,176 sorcha.sorcha INFO Applying SSP linking filter... 2024-07-28 01:21:13,176 sorcha.sorcha INFO Number of rows BEFORE applying SSP linking filter: 1 2024-07-28 01:21:13,472 sorcha.sorcha INFO Number of rows AFTER applying SSP linking filter: 0 2024-07-28 01:21:13,472 sorcha.sorcha INFO Post processing completed for this chunk 2024-07-28 01:21:13,472 sorcha.sorcha INFO Output results for this chunk 2024-07-28 01:21:13,473 sorcha.modules.PPOutput INFO Constructing output path... 2024-07-28 01:21:13,473 sorcha.modules.PPOutput INFO Output to CSV file... 2024-07-28 01:21:13,479 sorcha.sorcha INFO Sorcha process is completed.

mschwamb commented 1 month ago

Sounds like we should have a check before we get to calling the output to check if there is anything in the dataframe before going to output. This might be easy for @astronomerritt to tackle. What do you think @astronomerritt ?

astronomerritt commented 1 month ago

Sure. Essentially, if the dataframe is empty just before we go to write out the chunk, we should instead log a warning and skip to the next chunk, right?

mschwamb commented 1 month ago

yep

mschwamb commented 1 month ago

One thing to check is if chunk 0 has no objects detectable but chunk 1 does will the outputs work correctly.