airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.5k stars 3.99k forks source link

Can't build python modules locally. #4148

Closed cgardens closed 1 year ago

cgardens commented 3 years ago

Expected Behavior

Current Behavior

> Task :airbyte-integrations:bases:base-normalization:unitTest FAILED
[python] .venv/bin/python -m pytest -s unit_tests
         ============================= test session starts ==============================
         platform darwin -- Python 3.7.9, pytest-6.1.2, py-1.10.0, pluggy-0.13.1
         rootdir: /Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization
         collected 0 items / 4 errors

         ==================================== ERRORS ====================================
         _______ ERROR collecting unit_tests/test_destination_name_transformer.py _______
         ImportError while importing test module '/Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization/unit_tests/test_destination_name_transformer.py'.
         Hint: make sure your test modules/packages have valid Python names.
         Traceback:
         .venv/lib/python3.7/importlib/__init__.py:151: in import_module
             return _bootstrap._gcd_import(name[level:], package, level)
         unit_tests/test_destination_name_transformer.py:29: in <module>
             from normalization.destination_type import DestinationType
         normalization/__init__.py:2: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/__init__.py:1: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/transform.py:30: in <module>
             import yaml
         E   ModuleNotFoundError: No module named 'yaml'
         _____________ ERROR collecting unit_tests/test_stream_processor.py _____________
         ImportError while importing test module '/Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization/unit_tests/test_stream_processor.py'.
         Hint: make sure your test modules/packages have valid Python names.
         Traceback:
         .venv/lib/python3.7/importlib/__init__.py:151: in import_module
             return _bootstrap._gcd_import(name[level:], package, level)
         unit_tests/test_stream_processor.py:30: in <module>
             from airbyte_protocol.models.airbyte_protocol import DestinationSyncMode, SyncMode
         E   ModuleNotFoundError: No module named 'airbyte_protocol'
         ___________ ERROR collecting unit_tests/test_table_name_registry.py ____________
         ImportError while importing test module '/Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization/unit_tests/test_table_name_registry.py'.
         Hint: make sure your test modules/packages have valid Python names.
         Traceback:
         .venv/lib/python3.7/importlib/__init__.py:151: in import_module
             return _bootstrap._gcd_import(name[level:], package, level)
         unit_tests/test_table_name_registry.py:32: in <module>
             from normalization.transform_catalog.catalog_processor import CatalogProcessor
         normalization/__init__.py:2: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/__init__.py:1: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/transform.py:30: in <module>
             import yaml
         E   ModuleNotFoundError: No module named 'yaml'
         _____________ ERROR collecting unit_tests/test_transform_config.py _____________
         ImportError while importing test module '/Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization/unit_tests/test_transform_config.py'.
         Hint: make sure your test modules/packages have valid Python names.
         Traceback:
         .venv/lib/python3.7/importlib/__init__.py:151: in import_module
             return _bootstrap._gcd_import(name[level:], package, level)
         unit_tests/test_transform_config.py:30: in <module>
             from normalization.transform_catalog.transform import extract_schema
         normalization/__init__.py:2: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/__init__.py:1: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/transform.py:30: in <module>
             import yaml
         E   ModuleNotFoundError: No module named 'yaml'
         =========================== short test summary info ============================
         ERROR unit_tests/test_destination_name_transformer.py
         ERROR unit_tests/test_stream_processor.py
         ERROR unit_tests/test_table_name_registry.py
         ERROR unit_tests/test_transform_config.py
         !!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!
         ============================== 4 errors in 0.52s ===============================
marcosmarxm commented 3 years ago

@cgardens did you activate the pyenv? image

cgardens commented 3 years ago

yeah. python is working.

The following eventually worked.

./gradlew :cleanPythonVenv
./gradlew clean
git clean -xfd
rm -rf .gradle

the bad thing here is it means clean and cleanPythonVenv aren't actually enough to get the python env into a blank slate state.