Dropping DB

[ajmejia]

Implementing metadata caching without DB

These changes are focused on keeping track of relevant metadata (mjd, imagetyp, camera, expnum, exptime, etc) to be able to match analog frames and master calibration frames quickly, and to keep track of the reduction stage, status and quality as the frames are being reduced.

The following changes are introduced:

• Implemented metadata caching using HDF5 in place of a SQLite DB
• Implemented low-level functions _* for general metadata manipulation
• Implemented high-level functions that are meant to be run in the following sequence:
  • extract_metadata: to extract header information from frames stored locally
  • add_metadata: add metadata to the HDF5 store
  • del_metadata: remove HDF5 store or a given mjd within it
  • get_metadata: retrieve metadata from the store
  • get_analog_groups: group metadata into analogs for combining into master frames
  • match_master_metadata: match master calibration frames with target frames
  • put_reduction_stage: update the reduction stage of a particular frame
• The matching functions follow the criteria:
  • Analogs are based on their mjd, camera, imagetyp and exptime
  • Calibration-target frames are based on their mjd, camera, imagetyp, exptime, quality, stage and arc lamp
• The module lvmdrp.utils.database is replaced by lvmdrp.utils.metadata

Some bug fixes were also introduced to the runMethod module, but those were only meant for testing the metadata module.

[ajmejia]

The functionality of the metadata module is mostly the same after implementing these requests. In summary:

• The metadata file structure has changed to store the metadata in the reduction paths.
• There will be metadata stores for raw frames for each tileid and mjd.
• There will be one metadata store for master calibration frames at the root of a reduction version.
• To retain the possibility to match multiple calibration frames across tileids and mjds, I implemented store matching with wildcards.
• I dropped the observatory parameter/column in all functions/tables as that information is already in the reduction paths. In the tables the column hemi was added to be able to match frame paths using the path.expand method.

[havok2063]

This branch this work was in should be deleted.