[Failing Test]: PostCommit Python and PostCommit Python Arm perma red

damondouglas commented 4 months ago

What happened?

PostCommit Python and PostCommit Python Arm in perma red status.

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 2 (backlog / disabled test but we think the product is healthy)

Issue Components

[X] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam YAML
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

shunping commented 4 months ago

I took a look at the logs in PostCommit Python.

The last successful run was PostCommit Python #2719 @ 04/26/2024 1:32PM EDT.
The first failure has occurred since PostCommit Python #2721 @ 04/26/2024 7:32PM EDT.

Here is the traceback:

Traceback (most recent call last):
File "/runner/_work/beam/beam/build/gradleenv/1398941892/bin/pytest", line 8, in <module>
sys.exit(console_main())
File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/_pytest/config/__init__.py", line 192, in console_main
code = main()
File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/_pytest/config/__init__.py", line 150, in main
config = _prepareconfig(args, plugins)
File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/_pytest/config/__init__.py", line 331, in _prepareconfig
config = pluginmanager.hook.pytest_cmdline_parse(
File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/pluggy/_hooks.py", line 513, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/pluggy/_manager.py", line 120, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/pluggy/_callers.py", line 156, in _multicall
teardown[0].send(outcome)
File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/_pytest/helpconfig.py", line 104, in pytest_cmdline_parse
config: Config = outcome.get_result()
File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/pluggy/_result.py", line 100, in get_result
raise exc.with_traceback(exc.__traceback__)
File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/pluggy/_callers.py", line 103, in _multicall
res = hook_impl.function(*args)
File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1075, in pytest_cmdline_parse
self.parse(args)
File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1425, in parse
self._preparse(args, addopts=addopts)
File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1305, in _preparse
self.pluginmanager.load_setuptools_entrypoints("pytest11")
File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/pluggy/_manager.py", line 421, in load_setuptools_entrypoints
plugin = ep.load()
File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/_pytest/assertion/rewrite.py", line 186, in exec_module
exec(co, module.__dict__)
File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/typeguard/__init__.py", line 21, in <module>
from ._importhook import ImportHookManager as ImportHookManager
File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/_pytest/assertion/rewrite.py", line 186, in exec_module
exec(co, module.__dict__)
File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/typeguard/_importhook.py", line 22, in <module>
from typing_extensions import Buffer
ImportError: cannot import name 'Buffer' from 'typing_extensions' (/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/typing_extensions.py)

Note that Buffer was introduced in typing_extensions at 4.6.0, but the version resolved here is 4.5.0.

Successfully installed ...(omitted) typing-extensions-4.5.0 ...(omitted)

During checking the previous log, I found that there was an existing version incompatible problem on typing_extensions even before the recent post commit failures. It looks like a lot of packages depend on a newer version of typing-extensions, while tensorflow requires typing-extensions < 4.6.0.

Collecting typing-extensions<4.6.0,>=3.6.6 (from tensorflow>=2.12.0->-r /runner/_work/beam/beam/sdks/python/apache_beam/ml/inference/tensorflow_tests_requirements.txt (line 19))
Downloading typing_extensions-4.5.0-py3-none-any.whl.metadata (8.5 kB)

...

azure-core 1.30.1 requires typing-extensions>=4.6.0, but you have typing-extensions 4.5.0 which is incompatible.
azure-storage-blob 12.20.0b1 requires typing-extensions>=4.6.0, but you have typing-extensions 4.5.0 which is incompatible.
pydantic 2.7.1 requires typing-extensions>=4.6.1, but you have typing-extensions 4.5.0 which is incompatible.
pydantic-core 2.18.2 requires typing-extensions!=4.7.0,>=4.6.0, but you have typing-extensions 4.5.0 which is incompatible.

However this didn't show up as a PostCommit failure until typeguard was introduced as a dependency from https://github.com/apache/beam/blob/3329edba79f7b99af983b22cf44ab6e77a8ee987/sdks/python/apache_beam/transforms/enrichment_handlers/feast_tests_requirements.txt. (Code submitted on 04/26/2024 2:31PM EDT after the last successful run)

Collecting typeguard>=4.0.0 (from feast[gcp]->-r /runner/_work/beam/beam/sdks/python/apache_beam/transforms/enrichment_handlers/feast_tests_requirements.txt (line 18))

In typeguard, the code tries to import Buffer which causes an import error as shown in the traceback.

Adding @riteshghorse who submitted the above commit to the discussion here.

riteshghorse commented 4 months ago

Thanks for the analysis @shunping! I think I can take this one

damondouglas commented 3 months ago

Numerous errors still showing for PostCommit Python: https://github.com/apache/beam/actions/runs/9447564413 https://github.com/apache/beam/actions/runs/9447157024 https://github.com/apache/beam/actions/runs/9447191775 https://github.com/apache/beam/actions/runs/9447544394 https://github.com/apache/beam/actions/runs/9447549306

apache / beam