Set up PostgreSQL Butler registry and re-ingest Deep Field

[markgbeckett]

• [x] Set up PostgreSQL database
• [x] Create Butler and point to PostgreSQL d/b
• [x] Ingest (deep-field) pipeline run. Try to just ingest Deep Images (i.e., not the intermediates). (Might need an ECSV or Yaml file)
• [x] Register new Butler instance with RSP (replacing SQLite version)

[markgbeckett]

• Raphael to send a sample SXDS dataset, which Amanda and Mike will use to experiment with ingest.
• May be mismatch between Pipeline running at CSD3 (Ver. 22.0.0) and on Amanda's test deployment (Ver. 24.0). Both Gen 3 Butler, at least, plus may be capacity to set DAC version of Pipeline to 22.0 to match what Raphael has used.

[markgbeckett]

Amanda has data from SXDS, which has been successfully ingested into Butler repo on UK DAC. Also tried to install ObsVISTA onto the Stack (Version 24), though this is failing a test. Amanda will liaise with Raphael as potentially a known issue he is to fix.

[markgbeckett]

Next step is to try setting up PostgreSQL database and re-ingest SXDS into that.

[markgbeckett]

Amanda has ingested SXDS into Butler (Postgres-based), using Yaml summary file. Attempting to ingest just using FITS metadata did not work, potentially due to code in ObsVISTA being commented out.

Next step is to try to link Butler to RSP, and to link to data. Stelios can help with that.

Possibly, SXDS storage is Cinder-based so only mountable on one VM at once. Experiments required.

[markgbeckett]

Stelios is helping Amanda to make Butler service accessible from RSP. In first instance, will use public IP address over Internet. Later move to internal data network (need a gateway node for maintenance access).

[markgbeckett]

Amanda tested Butler access from Notebook service. All good.

[markgbeckett]

Butler confirmed visible from RSP. Need to understand how credentials should work.