search

LCOGT / mop

Microlensing Observation Portal
GNU General Public License v3.0
0 stars 7 forks source link

Uploaded lightcurves not displayed on plot #88

Open rachel3834 opened 10 months ago

rachel3834 commented 10 months ago

The pyDANDIA pipeline is uploading lightcurve datafiles as csv as normal, but they do not seem to be displaying on the lightcurve plot as they were previously.

rachel3834 commented 10 months ago

Strangely, if the datafiles are uploaded manually they do display

rachel3834 commented 10 months ago

Using the pipeline's own module for uploading lightcurve files to MOP from the commandline, I find this uploads without any issue raise, and the data are displayed automatically on the lightcurve plot as normal.

rachel3834 commented 10 months ago

The pipeline's own logs record only normal output and it's similar to the output when the upload is run from the commandline. So I don't see why the plot behaves differently when the data are uploaded programmatically.

rachel3834 commented 10 months ago

Diagnosing this problem in deployment is more difficult because ReducedDatums aren't listed in the TOM's admin interface by default.
Instead I've added a management command that lists all available ReducedDatums for a given Target. I also added logging output to the mop_extras function that gathers data for the plot.

I investigated the example of Gata23ats, for which MOP-prod lists two pipeline-produced lightcurves in addition to Gaia photometry, but displays only the Gaia data and one of the pyDANDIA lightcurves.

Loading the display reported ReducedDatums for the Gaia photometry and one pyDANDIA lightcurve. The fetch_reduceddata tool also does not report ReducedDatums for the second lightcurve.

This suggests that the DataProduct entry was created without ingesting the ReducedDatums from it.

Reviewing the pipeline logs, there was no sign of an issue being reported.

rachel3834 commented 10 months ago

A manual test run of the pipeline to explore why this might have happened produced this error:

Traceback (most recent call last): File "reduction_control.py", line 750, in reduction_control() File "reduction_control.py", line 83, in reduction_control run_automatic_reduction(setup,red_log,params) File "reduction_control.py", line 280, in run_automatic_reduction status = run_existing_reduction(setup, config, red_log) File "reduction_control.py", line 324, in run_existing_reduction upload_lightcurve_tom(setup, config, lc_files, red_log) File "reduction_control.py", line 459, in upload_lightcurve_tom upload_lc_to_tom.upload_lightcurve(setup, payload, log=log) File "/data/robouser/software/robonet_venv3/lib/python3.7/site-packages/pyDANDIA-0.1-py3.7-linux-x86_64.egg/pyDANDIA/upload_lc_to_tom.py", line 33, in upload_lightcurve existing_datafiles = tom.list_dataproducts(config, login, payload, target_pk, log=log) File "/data/robouser/software/robonet_venv3/lib/python3.7/site-packages/pyDANDIA-0.1-py3.7-linux-x86_64.egg/pyDANDIA/tom.py", line 74, in list_dataproducts response = requests.get(dataupload_url, params=ur, auth=login).json() File "/data/robouser/software/robonet_venv3/lib/python3.7/site-packages/requests/models.py", line 897, in json return complexjson.loads(self.text, **kwargs) File "/opt/miniconda3/lib/python3.7/json/init.py", line 348, in loads return _default_decoder.decode(s) File "/opt/miniconda3/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/opt/miniconda3/lib/python3.7/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) You have new mail in /var/spool/mail/robouser

rachel3834 commented 10 months ago

I've added HTML error code handling to this module in the pipeline to help to make it more robust. Running this code for a few examples however, and the log clearly shows errors caused by 504 Gataway timeouts, which are the result of MOP queries hanging.

This also explains the occasional duplication of datafiles for some targets, since the code checks for existing duplicated datafiles and would delete them if it received a response. However, it was continuing with the upload assuming no duplication if no response was received.

Instead, I've added a MOP-live status check to the function that decides whether to proceed with the upload; this should now discontinue the upload if MOP is giving time-outs.

rachel3834 commented 10 months ago

From tests, once MOP successfully completes an upload, the data display normally. Any targets where data products are listed but not displayed should be deleted manually and re-uploaded. Eventually, the pipeline will come along and do this once new data are obtained, but the data can also be uploaded manually using the pipeline's upload_lc_to_tom function.