Split ingestion in two separate program

[nitrosx]

Currently all the ingestion functionalities are in the same single program. We are going to create two different scripts:

• online main ingestor This program goes into an infinite loop and implements the following functionalities:
  • listen to the wrdn (Done writing) message sent by the file writer
  • creates the entry in SciCat
  • populate minimal metadata the SciCat entry
  • checks that the nexus file is accessible
  • save message file under ingestor folder
  • save json structure received in file under ingestor folder
  • create a background process to run the post-ingestion ingestor
• post-ingestion ingestor This program is run on a SciCat dataset entry and perform the following actions:
  • create b2blake hash of all the files
  • save hashes in file with matching names
  • load instrument metadata configuration
  • extract and prepare metadata
  • insert metadata in scicat dataset entry
  • add hashes to scicat dataset data files with hashes previously computed
  • add hash files to scicat dataset file list

[YooSunYoung]

@nitrosx

Can we use Scitacean instead of pyscicat here?

[nitrosx]

I'm not sure, as we need a lot more flexibility that is not allowed in scitacean. We might need to go directly to direct API calls.

[nitrosx]

I reviewed the requirements to create a dataset in SciCat and I think we should change the functionalities of the two programs as follow:

• main ingestor loop This program connect to kafka and subscribe to the correct topics listening for the done writing message. Everytime a WRDN message is received, save the message in json format with minimal transformation to the correct ingestor folder and run the background ingestor on the nexus file and message file in a new process
• background ingestor This program is a single stand-alone python program that can be run on terminal which accepts as input a nexus file and, if present, the WRDN message in json format file and create the entry in SciCat. Here are the steps that are to be performed in this program:
  • open nexus file
  • extract proposal id
  • retrieve information regarding the proposal
  • read metadata configuration for the matching instrument
  • extract and prepare metadata
  • create b2blake hash of all the files
  • create and populate scicat dataset entry
  • create and populate scicat origdatablock entry with files and hashes previously computed

With this design we are able to separate the kafka communication from the scicat interface, simplifying testing. It will also allow the use of Scitacean.

[nitrosx]

I am going to create a branch for this issue

[nitrosx]

Branch functionalities _segregation created

[nitrosx]

This issue has been addressed.