Closed Michael-J-Ward closed 3 weeks ago
Lastly, because our tests use relative imports for test-helpers, we need to make the test directory a proper package and we can run the entire test suite
mv python-tests python_tests
touch python_tests/__init__.py
pytest python_tests/
❯ pytest python_tests/
========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.11.9, pytest-8.3.2, pluggy-1.5.0
rootdir: /home/mike/workspace/tmp/datafusion-python
configfile: pyproject.toml
collected 338 items
python_tests/test_aggregation.py ......................... [ 7%]
python_tests/test_catalog.py . [ 7%]
python_tests/test_config.py ... [ 8%]
python_tests/test_context.py .................................... [ 19%]
python_tests/test_dataframe.py .............................................................................................. [ 47%]
python_tests/test_expr.py ....... [ 49%]
python_tests/test_functions.py ..................................................................................................................... [ 83%]
python_tests/test_imports.py ..... [ 85%]
python_tests/test_indexing.py .. [ 85%]
python_tests/test_input.py . [ 86%]
python_tests/test_sql.py ......................XxXXXxxxx........ [ 97%]
python_tests/test_store.py . [ 97%]
python_tests/test_substrait.py ... [ 98%]
python_tests/test_udaf.py s.. [ 99%]
python_tests/test_wrapper_coverage.py . [100%]
========================================================================== 328 passed, 1 skipped, 5 xfailed, 4 xpassed in 0.70s ==========================================================================
Goal: Document how to run the unit-tests against a release candidate
Previously, I had simply checked out the repo at the release tag, built
datafusion-python
withmaturin develop
, and ran the tests.This morning, I tried running the unit-tests strictly against the released candidate of
datafusion-python
hosted ontest.pypi.org
.The original attempt failed in a surprising way, so I'll share what I tried and found.
Please don't hesitate to correct me or suggest a better way to do something.
Setting up the attempt to verify
Problem 1) The location of the tests causes it to look in the
datafusion
source code instead of the installeddatafusion
packageNotice, the error originates from our local repo code
python/datafusion/__init__.py:29
instead of the installed candidatevenv/lib/python3.11/site-packages/datafusion/__init__.py:29
This is why the setup starts with cloning a fresh repo instead of re-using your existing
datafusion-python
clone. Your local clone almost certainly has a./python/datafusion/_internal.abi3.so
created from a previousmaturin develop
run - this doesn't get cleaned up withcargo clean
, andmaturin
doesn't appear to have aclean
command that removes it.This was the source of my original "surprising" test failure. My
_internal.abi3.so
was from my work on #802, which made it appear as if the candidate-release had been compiled against the wrong datafusion version.Problem 2)
typing_extensions
is not installed as a dependencyAfter moving the tests repo, running the test uses the installed
datafusion
package (venv/lib/python3.11/site-packages/datafusion/__init__.py
), buttyping_extensions
is not installed as a dependency when we ranpip install --extra-index-url https://test.pypi.org/simple/ datafusion==40.0.0
.So we install the dependency, re-run the test and now it's green.