EIDF workers

[mjaquiery]

Parts of Galvanalyser functionality should be separated out for deployment within the EIDF. A key part of this is monitoring battery data file uploads and processing those files.

This will be done on a cluster by creating individual workers who check out and process individual files, creating records in the Galvanalyser database and sending completed records to the EIDF CKAN service for recording and labelling as related to the original file records.

This process is the Worker component in the diagram below.

• [ ] search for unprocessed uploaded files on CKAN
• [ ] 'checkout' a file by registering it in a database
• [x] process a file to extract its data into the generic Galvanalyser format
• [ ] send processed data to CKAN, linked to the raw data as a related record

[martinjrobins]

Hopefully there shouldn't be much EIDF specific stuff. They are still monitoring a directory (just that this directory is in the data lake). The "checkout" functionality is new, but this would be useful for standard galvanalyser as well I think. Probably the "send processed data to CKAN" is the main EIDF specific stuff, can we make this generic somehow (e.g. set url and payload in .env)?

[mjaquiery]

Yeah, good idea to make this workflow as generic as possible. Currently harvesters send metadata before they send file data (which creates the dataset), so we could move dataset creation earlier as a checkout step and then populate the metadata and data later.