Enhanced Config System

[gwaybio]

In marching towards version 0.1 release, I noticed so much room for improvement in our current config system setup. We were developing the system in real time while dealing with (and developing) the various recipe modules and steps. Now that we are close to version 0.1 - many enhancements became apparent. A couple noted below to revisit before version 0.2

• There are lots of overlapping content between the two config files. We should think about merging the config files into one, or at least cross pollinating.
• There are SO many options and many of which are completely unimportant (most of the time). We should consider splitting off our system into 1) a primary config that contains files that should be edited with each new experiment, and 2) options to be modified only by advanced users in emergency cases.
• A lot of the path manipulation and file name creation happens in the config.py file during config processing. This is very obscure and tough to track down. We should consider enhancements to these operations as well.

In addition to an enhanced config setup in version 0.2, we should also tighten up some of the fragile pieces of the weld, that are closely linked to config solutions.

For posterity, one example is described in broadinstitute/pooled-cell-painting-profiling-recipe#45.

[ErinWeisbart]

There are SO many options and many of which are completely unimportant (most of the time). We should consider splitting off our system into 1) a primary config that contains files that should be edited with each new experiment, and 2) options to be modified only by advanced users in emergency cases.

Yes! I really like this! Will definitely increase general usability.

[gwaybio]

I agree. As you go through #11 and make updates to documentation (as noted in https://github.com/broadinstitute/pooled-cell-painting-profiling-template/pull/11#issuecomment-675108249) it would be great to think about this "advanced" configuration file in real time.

As I'm processing more data (and different experiments) through the weld I am getting more experience with knowing which options are likely to change, but you still have a much clearer picture than me.