NOAA-EMC / wxflow

Tools for Weather Workflows
https://wxflow.readthedocs.io/en/latest
GNU Lesser General Public License v3.0
0 stars 7 forks source link

Parse config files with input envars #35

Closed DavidHuber-NOAA closed 2 months ago

DavidHuber-NOAA commented 2 months ago

Description

This allows the preconditioned sourcing of scripts based on optional environmental variables passed to the parse_config method within the Configuration class. A new test is added for the feature.

This is a required feature for https://github.com/NOAA-EMC/global-workflow/issues/2693 to be able to source configs with RUN specified in order to get RUN-specific resources.

Type of change

How Has This Been Tested?

Created an xml with global-workflow branch https://github.com/DavidHuber-NOAA/global-workflow/tree/feature/simplify_res

Checklist

codecov[bot] commented 2 months ago

Codecov Report

Attention: Patch coverage is 66.66667% with 3 lines in your changes missing coverage. Please review.

Project coverage is 49.93%. Comparing base (5dad7dd) to head (29c1a3c). Report is 1 commits behind head on develop.

Files Patch % Lines
src/wxflow/configuration.py 66.66% 2 Missing and 1 partial :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## develop #35 +/- ## =========================================== + Coverage 48.23% 49.93% +1.70% =========================================== Files 18 18 Lines 1644 1650 +6 Branches 335 337 +2 =========================================== + Hits 793 824 +31 + Misses 791 765 -26 - Partials 60 61 +1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

aerorahul commented 2 months ago

Is that really appropriate for a method that is supposed to just source a shell script and return the set variables as a dictionary.

This PR extends that from parsing/sourcing shell scripts and adding additional variables to that dictionary.

IMO that should be done after the sourcing and adding/updating the envvars.

DavidHuber-NOAA commented 2 months ago

@aerorahul I agree with your assessment on the capability. However, modifying the variables after sourcing is not enough for the global-workflow resources. They depend on knowing the RUN at the time of sourcing, otherwise those variables will be set based on the default RUN (gfs).

I see one possible solution: Read the variables in the input config(s) and check for the existence of the input envvars. If they do not exist, raise an error. If they do exist, export the input envvars and source again.

An option that seems viable but is not: In the global workflow at setup time, source config.base once, then modify the RUN in memory before sourcing task configs. Unfortunately, this will not work unless the contents of config.base are exported to bash as sourcing the task configs requires knowing the contents of config.base. This leads us back to a wxflow modification.

DavidHuber-NOAA commented 2 months ago

After offline discussion with @aerorahul, found a solution in the global-workflow to handle this case. Closing.

DavidHuber-NOAA commented 2 months ago

After testing of another option, this was found to be the favorable one. Reopening.

DavidHuber-NOAA commented 2 months ago

Pytests were run on WCOSS2 successfully.