spacepy / dbprocessing

Automated processing controller for heliophysics data
5 stars 4 forks source link

Support specifying products by name not just section #81

Open jtniehof opened 3 years ago

jtniehof commented 3 years ago

addFromConfig works from a config file. Each process and product has a section in the config file, headed by a section name in []. The "type" of section is determined by the section name, so [product_rot13] is a product. The process sections need to specify the inputs (optional and required) and the output product.

The section name is not stored in the database. Each product also has a product_name, which is stored in the db.

Right now the input and output products are specified by section name, not product name. It would be convenient to allow specifying by product name as well. This might also make it easier to have a "partial" config file that adds processes to an existing database.

Relation to an issue

Relevant to discussion in #78, which had to make changes to configFromDB to work around this limitation.

Proposed enhancement

This config file illustrates the difference. It doesn't successfully run through addFromConfig as it uses product names. Lines referencing the section names are included (commented out). product_names.conf.txt

Alternatives

Documentation is probably the best alternative (and needs to happen anyhow). Right now the only documentation is in a comment block at the head of configFromDB, and it implies product name can be used for inputs (but not for output.)

OS, Python version, and dependency version information:

Linux-4.15.0-143-generic-x86_64-with-Ubuntu-18.04-bionic
sys.version_info(major=2, minor=7, micro=17, releaselevel='final', serial=0)
sqlalchemy=1.1.11

Version of dbprocessing

Branch from #78

Closure condition

This issue should be closed when the attached file can be ingested with addFromConfig, and appropriate unit tests are included to verify this.