Data Organiser Planning

[thespacedoctor]

As shown at ESO by the pipeline team the data organisation (raw data into 'sets of files' or sof files) can be done with REFLEX. It was suggested that REFLEX could be run from the command-line in an automated fashion to make use of this functionality. While this means we would not have to write our own data-organiser recipe, we would again have reflex as a dependency of the final pipeline. I don't think we want this.

Most of the logic behind the REFLEX organiser can be found (and duplicated) from this paper:

Automated data reduction workflows for astronomy The ESO Reflex environment

This issue will be used to track the planning of the data-organiser.

TASKS

• [x] search for pre-existing python bases workflow to organise & associate FITS frames via keywords that we can use off the shelf
• [ ] read paper above and use to write the logic of the SOXS data-organiser in the overleaf design doc.
• [ ] dfitspy much faster than imageFileCollection -- can I speed up imageFileCollection? Particularly the create_sof script

[thespacedoctor]

Data organiser is now 50% written. I found much of the functionality was needed sooner than later when preparing unit-test data suites.

As other recipes are written I will mature the organiser.

[thespacedoctor]

Data Organiser Planning

The main goals of the DO are:

• create the SOF files for all data in a directory

soxspipe prep

Actions:

• [x] if SQLite database file is missing, add it from the static calibration suite shipped with the pipeline
• [x] move all raw data to folder called 'raw' (how can we tell frame is raw or not?)
• [x] update the 'raw_frames' table in SQLite
  • [x] compare filenames in 'raw_frames' table with those in the raw directory
  • [x] if files in the raw directory NOT in 'raw_frames' table, then read headers and write to 'raw_frames' table
  • [x] if files in 'raw_frames' table and NOT in the raw directory then flag them as deleted in 'raw_frames' table
• [ ] create/update the 'products_table' ... all products that could be potentially be created from the raw_data
• [ ] the sof_map table would link raw_frames (raw_frame_id) with product (parent_product_frame_id)
• [ ] the sof_map table would also link input product_frames (child_product_key) with output product frames (parent_product_frame_id)

The test data can be found here:

~/Dropbox/server-sync/soxs-eso-data/xshooter-pipeline-data/unittest_data/xsh/data-organiser

[thespacedoctor]

beta version of data-organiser has been released