askap-vast / vast-pipeline

This repository holds the code of the Radio Transient detection pipeline for the VAST project.
https://vast-survey.org/vast-pipeline/
MIT License
8 stars 3 forks source link

Separate image ingestion from main pipeline run #544

Closed dliptai closed 3 years ago

dliptai commented 3 years ago

Allow the user to simply ingest a set of images (and their measurements) into the database, without running the association steps.

This is done via a separate image ingestion configuration file, which is currently just a subset of the config used for standard pipeline runs. It includes the list images to be ingested, as well as the following four "measurement" options:

There are two new commands:

initingest generates a template "image ingestion configuration" file. ingestimages requires an "image ingestion configuration" as input.

dliptai commented 3 years ago

Note: this does not address all the features discussed/requested in #375 , but it does lay the foundation for most of them.

Currently this "new feature" is backwards compatible, since you still run the pipeline in the same was as before (i.e. the config format is unchanged).

Ideally we'd want to clean things up and completely split off the four options:

I'll leave the removal of selavy_local_rms_fill_value (and using values from the noisemap instead) to another pull request.

For the time being it would be nice to merge what we have, and address each of the above points in new, smaller, pull requests.

dliptai commented 3 years ago

@ajstewart agreed on the docs. I'll update those, and the changelog.

I have not thought about how to incorporate this into the UI. Aside from implementing it in a manner that preservers the current UI behaviour. So yes, if the run meets an image it hasn't ingested previously, it ingests it.