search

NCI-CGR / plco-analysis

Primary workflow for the PLCO "Atlas" project
2 stars 3 forks source link

fix config check #7

Closed lightning-auriga closed 3 years ago

lightning-auriga commented 3 years ago

make check-config is broken due to separate added support for deeper level yaml structures in config/*.yaml. especially if outside people are going to be using this pipeline in the near future, they're going to need to rely heavily on that check code to catch their doubtless many yaml formatting errors.

lightning-auriga commented 3 years ago

the issue currently is that there's a check for any yaml structure with too many levels of complexity to be represented in bash associative arrays. but that was a restriction in the old preprocessor, which has since been deprecated in favor of https://github.com/NCI-CGR/initialize_output_directories, which is much, much more sturdy.

lightning-auriga commented 3 years ago

since yaml-cpp supports (to my knowledge) the entire yaml feature set, restrictions on yaml features can be removed entirely. except! i think the check code itself is using bash associative arrays, which is really silly. need to reflect on what to do about this. may need to make a python checker (for pyyaml support) for this one, which is a bit of a pain.

lightning-auriga commented 3 years ago

ok, bash associative arrays are out again, and yaml is enforced once again with pyyaml. this should be replaced in the future with something that isn't trash. note that the checker breaks if someone adds an additional layer of multilayer nonsense almost anywhere in the file. that's fine for the moment, as they have lots of examples to work with, and breaking should still point them in the right direction towards a problematic config file. i'll leave this issue open though, for the reasons listed above.

lightning-auriga commented 3 years ago

i deem this fixed, though all the tests require a good modernization that is well beyond scope. I'll start another issue.