Closed chethanuk closed 3 years ago
cc: @gucciwang @jaoanan1126 Once this is merged I will work on pytest migration [to simplify the tests]
any updates on this? I want to use pydeequ on my project.
thank you
Working on this! Doesn't seem to be passing tests at the moment, will reach out!
If required let's merge #40 first, since it will add all workflows
Or Locally install poetry and Install dependencies then https://github.com/awslabs/python-deequ/pull/40/files#diff-f279a155e67df88ab450450a567b747e2db4e3db42eeb2b241a86693eb2a214cR133
My local output:
> poetry version
pydeequ 0.1.8
> poetry run coverage run -m pytest --reruns 5 --reruns-delay 30
====================================================== test session starts ======================================================
platform darwin -- Python 3.8.7, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/chethanuk/Work/Github/deequ/python-deequ
plugins: flake8-1.0.7, rerunfailures-9.1.1, cov-2.12.0
collected 156 items
tests/test_analyzers.py .........................sssssssssssssssssssss [ 29%]
tests/test_anomaly_detection.py .s...ss [ 33%]
tests/test_checks.py .......ssssssssssssssssssssssssssssssssss................................ [ 80%]
tests/test_pandas_utils.py ...... [ 84%]
tests/test_profiles.py ... [ 86%]
tests/test_pydeequ.py . [ 87%]
tests/test_repository.py ....ss.... [ 93%]
tests/test_scala_utils.py .. [ 94%]
tests/test_suggestions.py ........ [100%]
============================================================= warnings summary =============================================================
tests/test_pandas_utils.py::TestPandasUtils::test_p2s_analyzer
tests/test_pandas_utils.py::TestPandasUtils::test_p2s_profiles
tests/test_pandas_utils.py::TestPandasUtils::test_p2s_suggestion
tests/test_pandas_utils.py::TestPandasUtils::test_p2s_verification
tests/test_pandas_utils.py::TestPandasUtils::test_s2p_analyzers
tests/test_pandas_utils.py::TestPandasUtils::test_s2p_verification
/Users/chethanuk/Work/Github/deequ/python-deequ/pydeequ/pandas_utils.py:26: UserWarning: WARNING:
You passed in a Pandas DF, so we will be using our experimental utility to convert it to a PySpark DF.
warnings.warn(
-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================== 96 passed, 60 skipped, 6 warnings in 115.31s (0:01:55) ==========================================
> poetry run coverage report
Name Stmts Miss Cover
-------------------------------------------------------------------------------------------------------
.venv/lib/python3.8/site-packages/_pytest/__init__.py 5 2 60%
.venv/lib/python3.8/site-packages/_pytest/_argcomplete.py 37 23 38%
.venv/lib/python3.8/site-packages/_pytest/_code/__init__.py 10 0 100%
.venv/lib/python3.8/site-packages/_pytest/_code/code.py 699 380 46%
> spark-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.0.2
/_/
Using Scala version 2.12.10, OpenJDK 64-Bit Server VM, 1.8.0_282
Branch HEAD
Compiled by user centos on 2021-02-16T04:53:13Z
Revision 648457905c4ea7d00e3d88048c63f360045f0714
Url https://gitbox.apache.org/repos/asf/spark.git
Would love for support to pyspark 3.0 and 3.1, any idea on when this would be available?
Ping!. @gucciwang
Issue #, if available: May be https://github.com/awslabs/deequ/issues/310 #4
Description of changes: Adding support to pyspark 3.0 Generally faced few properly because of current setup.py files so Migrating from setup to Poetry which is one of the best python dependency management and packaging
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.