artemis_subscriber needs to update the data_files table

[rachel3834]

Artemis_subscriber should record the path to the survey datafiles in the DB when a new event is ingested.

[rachel3834]

I'm in the process of implementing this functionality and I have a question.

The DataFile model has the last_obs attribute as a DateTime object. I was planning to obtain this parameter by reading the last entry of the .dat files we rsync from ARTEMiS.

However both OGLE and MOA provide timestamps in HJD format. You can't (easily) reverse-engineer HJD timestamps to UTC datetimes, although conceivably we could.

I looked at the section of update_db_2.run_test2 that populates the DB with test entries and I see that it does a partial conversion - it treats the timestamps as 'JD' rather than 'HJD'. This is fine for testing purposes, but in operation I'd prefer to make sure that the timestamps are what they claim to be.

The easiest solution could be to modify the DataFile model, change the last_obs attribute to a float and simply store the value as an HJD.

This may present complexity further down the line, for example in the DB views, if we humans want to view the last_obs date, its more convenient if its in a human readable form.
A simple way around that later issue is to display the current JD and make sure the display clearly explains what frame each timestamp is in.

Assigning to Yianni for an opinion before I implement this.

[rachel3834]

Also, does anyone know where I get the blend parameter value (g) from in the ARTEMiS files? None of them have keywords in the header, so its hard to be sure...

[rachel3834]

I take it back, actually the .dat files do have headers, but they don't seem to have the blend parameter. I'm guessing the value is in either .model or .align...but those don't have headers...

[ytsapras]

We can also provide the HJD as a separate parameter in the model but for performing datetime querysets maybe we want to keep the datetime information as well. What do you think? I think The blend parameter is in the .align files, the third column (blend ratio g=fbl/f*). The columns are: column 1 is the telescope identifier column 2 is I0 column 3 is the blend ratio

[rachel3834]

Thanks for documenting the align headers. Normally I would say always keep the DATE-OBS information - you can always recalculate an HJD, but not always back-calculate the date. However, in this case, we don't have the DATE-OBS for the survey data, just their HJD timestamps.
So I think my preference would be to change the timestamp parameter in the DataFile model to be a float and clearly document what reference frame the timestamp is in throughout the site. Is that OK?

[ytsapras]

That is okay. I will change the model file to accept a floating point.

[ytsapras]

How do we deal with the entries that are in the DB already?

[ytsapras]

Come to think of it, do we need it to be so accurate for the database? The data files themselves contain the HJD. The datatime information in the database will be accurate to within a few minutes so that should be sufficient. I think the best way to do this is to add an extra column for the HJD, so it can be passed too as an optional parameter.

[ytsapras]

The DataFile model now contains (committed it to github)

HJD of last observation in file

last_hjd = models.DecimalField(max_digits=20,decimal_places=8, null=True, blank=True)

[rachel3834]

I'm really sorry its taken so long to get this issue resolved. I believe artemis_subscriber is now updating the data files table in the DB correctly. Please let me know if this works OK with TAP - so assigning to Markus to check.

[mpgh]

The initial selection of events looked reasonable. There are two open issues. REA TAP has selected an event whitout a KB*.align file. Do we really want to risk setting g=0? In addition, the latest models are older than 24 h. The latest files would alter the selection. The TAP log files tell me that we are still missing a few events, but I haven't figured out, yet, why.

[mpgh]

I have resolved the issue with the "KB" event. Its baseline was set to 0.0 which has lead to a high event rank. The next issue appeared because of an "OB" event which has a blend ratio >300. For preventing thses events to enter the list, I have introduced a threshold of g<300 for valid REA TAP events. That still needs to be revisited and discussed.

[rachel3834]

Marking this issue as resolved as the updates now seem to be working. TAP behaviour still under investigation in a separate thread.