clamsproject / wfe-pipeline-runner

3 stars 1 forks source link

Adding runtime parameters #14

Closed marcverhagen closed 3 years ago

marcverhagen commented 3 years ago

We need to be able to hand the pipeline script parameters for pipeline components. The beginnings of this are in the pipeline file. This maybe needs to be thought through a wee bit more. We can add the parameters to the configuration file:

services:
  - tokenizer:
      image: clams-nlp-example
      container: pipeline_tokenizer
      parameters:
        pretty: True

When we change the parameters it should be simply a matter of editing this file or maybe a matter of using command line parameters to overrule the ones in the file. Currently, a user-created configuration file is copied to config.yml on the host and the container. That file needs to be edited, not the file that was handed to the start script.

For command line parameters we could have something like

$ python pipeline --parameters tokenizer-pretty=True,spacy-linking=True