maany
commented
5 years ago Turns out calling the compiler_validate() function multiple times involves loading the schema and all the validators for yamale every single time. It would be a more redundant, hence, expensive compute operation compared to an additional I/O call on the complete schema and data files. Closing this issue in favor of this argument.
Is your feature request related to a problem? Please describe. We can improve the runtime performance of the framework by parsing most files only once instead of augmenting them first with the compiler and them validating them in the validation engine later.
Describe the solution you'd like In order to do so, we can make the validation engine a git submodule of the compiler ()https://github.blog/2016-02-01-working-with-submodules/ ). This helps speed up development. Before release, we must deploy the validation engine on PyPI and update the requirements.txt of the compiler.
We must then implement a compiler_validate() function in the config validation engine repo.
We can then use the compiler_validate(data, schema) function to validate each entry in the site level config file with its type parameter available from the schema. This can happen here: https://github.com/WLCG-Lightweight-Sites/simple_grid_yaml_compiler/blob/master/compiler/processor_config_schemas.py#L79
Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.
Additional context Add any other context or screenshots about the feature request here.