search

NCAR / ccpp-framework

Common Community Physics Package (CCPP)
http://www.dtcenter.org/community-code/common-community-physics-package-ccpp/
Other
26 stars 63 forks source link

ccpp-framework not working with Python 3.11 #457

Closed climbfuji closed 1 year ago

climbfuji commented 1 year ago

Description

@DusanJovic-NOAA reported the following error with the newly relerased Python 3.11 on Fedora 37 (cont'd below):

Traceback (most recent call last):
  File "/home/dusan/test/simple-ufs/src/model/FV3/ccpp/framework/scripts/ccpp_prebuild.py", line 21, in <module>
    from metadata_parser import merge_dictionaries, parse_scheme_tables, parse_variable_tables
  File "/home/dusan/test/simple-ufs/src/model/FV3/ccpp/framework/scripts/metadata_parser.py", line 19, in <module>
    from metadata_table import MetadataTable, parse_metadata_file
  File "/home/dusan/test/simple-ufs/src/model/FV3/ccpp/framework/scripts/metadata_table.py", line 132, in <module>
    from ccpp_state_machine  import CCPP_STATE_MACH
  File "/home/dusan/test/simple-ufs/src/model/FV3/ccpp/framework/scripts/ccpp_state_machine.py", line 15, in <module>
    CCPP_STATE_MACH = StateMachine((('initialize', 'uninitialized',
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dusan/test/simple-ufs/src/model/FV3/ccpp/framework/scripts/state_machine.py", line 81, in __init__
    self.add_transition(trans[0], trans[1], trans[2], trans[3])
  File "/home/dusan/test/simple-ufs/src/model/FV3/ccpp/framework/scripts/state_machine.py", line 88, in add_transition
    self[name] = (init_state, final_state, regex)
    ~~~~^^^^^^
  File "/home/dusan/test/simple-ufs/src/model/FV3/ccpp/framework/scripts/state_machine.py", line 165, in __setitem__
    regex = re.compile(value[2] + r"$")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/re/__init__.py", line 227, in compile
    return _compile(pattern, flags)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/re/__init__.py", line 294, in _compile
    p = _compiler.compile(pattern, flags)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/re/_compiler.py", line 743, in compile
    p = _parser.parse(p, flags)
        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/re/_parser.py", line 980, in parse
    p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/re/_parser.py", line 455, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/re/_parser.py", line 863, in _parse
    p = _parse_sub(source, state, sub_verbose, nested + 1)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/re/_parser.py", line 455, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/re/_parser.py", line 841, in _parse
    raise source.error('global flags not at the start '
re.error: global flags not at the start of the expression at position 3

This is because global inline flags are no longer allowed in the middle of regular expressions, only at the beginning (see https://github.com/fail2ban/fail2ban/issues/3260 for example).

Steps to Reproduce

Find yourself a machine with Python 3.11 (ideally Fedora 37, where it is native) and run one set of unit tests:

cd tests
PYTHONPATH=$PWD/../scripts/parse_tools:$PWD/../scripts:$PYTHONPATH python3.11 test_metadata_parser.py

Additional Context

Another problem is that with Python 3.11 installed on a system that doesn't have Python 3.11 as default, pytest doesn't work (see https://github.com/pytest-dev/py/issues/273). Thus, even after fixing the issue reported above in PR https://github.com/NCAR/ccpp-framework/pull/456, I cannot run all unit tests with Python 3.11.

Output

See above