NCAR / ccpp-framework

Common Community Physics Package (CCPP)
http://www.dtcenter.org/community-code/common-community-physics-package-ccpp/
Other
26 stars 63 forks source link

Fix ccpp-framework errors with Python 3.11 #456

Closed climbfuji closed 1 year ago

climbfuji commented 1 year ago

Fix Python 3.11 errors as described in https://github.com/NCAR/ccpp-framework/issues/457

With these changes, the error from

cd tests
PYTHONPATH=$PWD/../scripts/parse_tools:$PWD/../scripts:$PYTHONPATH python3.11 test_metadata_parser.py

is fixed. I asked @DusanJovic-NOAA to test this branch with the ufs-weather-model on his Fedora 37 machine, which uses Python 3.11 as default, and I would like to ask @mkavulich @grantfirl @dustinswales to test this branch with their versions of Python in the CCPP-SCM, please.

As noted in https://github.com/NCAR/ccpp-framework/issues/457, I cannot run the unit tests in ccpp-framework that require pytest. I ran the following unit tests successfully with Python 3.11:

cd tests
PYTHONPATH=$PWD/../scripts/parse_tools:$PWD/../scripts:$PYTHONPATH python3.11 test_metadata_parser.py

I then ran all unit tests successfully with the native Python 3.10 installation on Ubuntu 22.04:

cd tests
PYTHONPATH=$PWD/../scripts/parse_tools:$PWD/../scripts:$PYTHONPATH python3.11 test_metadata_parser.py
PYTHONPATH=$PWD/../scripts/parse_tools:$PWD/../scripts:$PYTHONPATH python3.11 test_mkstatic.py
cd ..
cd test
./pylint_test.sh # average code rating about 8.5/10
./run_doctest.sh
./run_tests.sh

User interface changes?: No

Fixes: https://github.com/NCAR/ccpp-framework/issues/457 (to be confirmed)

Testing: see above

climbfuji commented 1 year ago

Update. @DusanJovic-NOAA reported that this branch allows him to compile the ufs-weather-model with Python 3.11.

dustinswales commented 1 year ago

@climbfuji I was able to replicate the error with python3.11, and confirm that this fixes the error:

(scm_py311) framework/tests> setenv PYTHONPATH $PWD/../scripts (scm_py311) framework/tests> setenv PYTHONPATH $PYTHONPATH\:$PWD/../scripts/parse_tools (scm_py311) framework/tests> python test_metadata_parser.py ... File "/glade/work/dswales/conda-envs/scm_py311/lib/python3.11/re/_parser.py", line 841, in _parse raise source.error('global flags not at the start ' re.error: global flags not at the start of the expression at position 3

(scm_py311) framework/tests> cd ../../ (scm_py311) ccpp/framework> git remote add dom https://github.com/climbfuji/ccpp-framework (scm_py311) ccpp/framework> git fetch dom (scm_py311) ccpp/framework> git checkout dom/bugfix/python311_global_flags (scm_py311) ccpp/framework> cd framework/tests (scm_py311) framework/tests> python test_metadata_parser.py (scm_py311) framework/tests>

climbfuji commented 1 year ago

Thanks, @dustinswales!

Dusan also ran the unit tests for the framework on his Fedora system, and they pass just like on my box with Python 3.10. I am currently running the ufs-weather-model regression tests on hera with intel against this branch.

DusanJovic-NOAA commented 1 year ago

I successfully compiled ufs-weather-model on Fedora 37 (which uses python 3.11)

climbfuji commented 1 year ago

Note. Full regression tests for ufs-weather-model passed on Hera with Intel.

climbfuji commented 1 year ago

Thanks for taking care of this!

Good to hear from you :-)

jkbk2004 commented 1 year ago

all tests are done on https://github.com/ufs-community/ufs-weather-model/pull/1522, can we start process to merge in this pr?

climbfuji commented 1 year ago

@jkbk2004 Merged, thanks!