search

ovro-eovsa / ovro-lwa-solar-ops

Scripts and codes related to operations of OVRO-LWA for solar studies
MIT License
1 stars 4 forks source link

Slow data recording issue (DO NOT MERGE. NOT TESTED) #41

Closed surajit4444mondal closed 3 months ago

surajit4444mondal commented 4 months ago

DO NOT MERGE. NOT TESTED.

But please look at the strategy I have implemented to ensure that we always for the "delay" time passed if file search is not successfull initially. If this makes sense, I can go ahead and test this code out.

peijin94 commented 4 months ago

There is a sleep(2) after the filename is found, is this the waiting time for the file writing time?

(checked inside folder, for a few cases I checked, the creation time of the files within MS is within 1s)

binchensun commented 4 months ago

I think this functionality needs to be implemented outside the list_msfiles() function, which is solely meant for returning the ms files at whatever time it was told to look for. The realtime pipeline ensures that the image time is always behind the current time by a set delay. See https://github.com/ovro-eovsa/ovro-lwa-solar-ops/blob/main/solar_realtime_pipeline.py#L1462-L1465

surajit4444mondal commented 4 months ago

While, it is true, that the realtime pipeline ensures a delay between the current time and image time, if the files are not found in first try, I wanted it to wait for a bit. However, I realize now that, maybe having a larger delay (4 minutes after Bin's change) and having a fail safe like Peijin suggested is a cleaner way.