Closed Gerenjie closed 1 month ago
Sounds like we should have a check before we get to calling the output to check if there is anything in the dataframe before going to output. This might be easy for @astronomerritt to tackle. What do you think @astronomerritt ?
Sure. Essentially, if the dataframe is empty just before we go to write out the chunk, we should instead log a warning and skip to the next chunk, right?
yep
One thing to check is if chunk 0 has no objects detectable but chunk 1 does will the outputs work correctly.
Zero-row outputs are generated if bright detections are made but don't survive the linking filter, but don't get made if no bright detections are made. Now that I think of it, doesn't the latest version of Sorcha have the linking filter add a column rather than filter detections? Not sure if this will be reproduced on the latest version of Sorcha, sorry... Anyway, the discrepancy I saw follows.
2024-07-28 01:20:34,832 sorcha.sorcha INFO Number of rows BEFORE applying bright limit filter 0
2024-07-28 01:20:34,833 sorcha.sorcha INFO Number of rows AFTER applying bright limit filter 0
2024-07-28 01:20:34,833 sorcha.sorcha INFO No observations left in chunk. Skipping to next chunk...
2024-07-28 01:20:34,833 sorcha.sorcha INFO Sorcha process is completed.
versus2024-07-28 01:21:13,175 sorcha.sorcha INFO Number of rows BEFORE applying bright limit filter 1
2024-07-28 01:21:13,176 sorcha.sorcha INFO Number of rows AFTER applying bright limit filter 1
2024-07-28 01:21:13,176 sorcha.sorcha INFO Applying SSP linking filter...
2024-07-28 01:21:13,176 sorcha.sorcha INFO Number of rows BEFORE applying SSP linking filter: 1
2024-07-28 01:21:13,472 sorcha.sorcha INFO Number of rows AFTER applying SSP linking filter: 0
2024-07-28 01:21:13,472 sorcha.sorcha INFO Post processing completed for this chunk
2024-07-28 01:21:13,472 sorcha.sorcha INFO Output results for this chunk
2024-07-28 01:21:13,473 sorcha.modules.PPOutput INFO Constructing output path...
2024-07-28 01:21:13,473 sorcha.modules.PPOutput INFO Output to CSV file...
2024-07-28 01:21:13,479 sorcha.sorcha INFO Sorcha process is completed.