Improving configuration passing

[joezuntz]

We need to rationalize the provision of configuration options to stages somewhat - right now it is a bit confused - the pipeline both reads the yaml config file and passes all the options in it on the command line and passes the file itself. It's useful to be able to override parameters, but the current design is just a bit confusing.

Here are a few suggested principles:

• The config yaml file is a required for every stage in the pipeline. It should therefore be an implicit input parameter for each one, and not need to be specified manually.

• A pipeline run should not specify parameters on the command line but instead just pass the yaml file

• If (when testing/debugging) the user wants to pass parameters on the command line then they should override the config file as they do now.

@EiffL do these bullet points make sense to you?

[EiffL]

Yes, agreed. Some parameters may be too complicated to be input on the command line anyway so I suspect some kind of configuration file will always be necessary.

Agreed that Ceci doesn't need to provide any arguments on the command line, everything can be in the config file, but keeping the ability to override arguments from the command line makes pipeline stages compatible with standard CWL runnners :-)

[joezuntz]

@EiffL I've pushed some changes that do this into a branch, but I've not tried to modify the generate_cwl part. Could you have a look at that? I think the only relevant difference is that the config input is now implicit.

[EiffL]

Cool, I'll have a look today

[EiffL]

Done, there was just a few lines to update so that the cwl description of a pipeline stage contain a config file even if it's not explicitly specified in the inputs of the stage. Travis build successful, and still runs with cwltool :-)

[joezuntz]

Great, this is dealt with now