search

SciCatProject / scicat-filewriter-ingest

Python client that connects to a kafka queue and creates new datasets when receiving notification that a file has been written
0 stars 0 forks source link

Background ingestor #39

Closed nitrosx closed 3 weeks ago

nitrosx commented 3 weeks ago

This PR introduces the off-line script to ingest a single file produced by the filewriter. It reduces/eliminates the dependency from the done writing message information. The background ingestor has to open the data file to extract the relevant metadata and allows for custom metadata for each instrument.

nitrosx commented 3 weeks ago

@YooSunYoung this PR is still a work in progress, although I think we can merge it in main and continue working on it from there