Closed AJM10565 closed 7 months ago
laughingman7743
commented
8 months ago Thank you, that's a good improvement, could you add a test for the case of a value error? https://github.com/laughingman7743/PyAthena/blob/master/tests/pyathena/pandas/test_util.py
AJM10565
commented
7 months ago I added some unit tests and also another use case I encountered with a similar issue.
laughingman7743
commented
7 months ago The test you added looks good. Thank you. If you fix the lint error, I will merge this PR.
$ make test
hatch run chk
cmd [1] | ruff check .
cmd [2] | ruff format --check .
43 files already formatted
cmd [3] | mypy .
Success: no issues found in 41 source files
hatch run test
============================================================================================================================================================================================================================== test session starts ===============================================================================================================================================================================================================================
platform darwin -- Python 3.11.8, pytest-8.0.0, pluggy-1.4.0
rootdir: /Users/foobar/github/PyAthena
configfile: pyproject.toml
plugins: cov-4.1.0, dependency-0.6.0, xdist-3.5.0, anyio-4.2.0
8 workers [502 items]
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................... [ 91%]
...................................s........ [100%]
================================================================================================================================================================================================================================ warnings summary ================================================================================================================================================================================================================================
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex[pandas_cursor2-1000000]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex[pandas_cursor0-None]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex[pandas_cursor1-1000]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex_as_pandas[pandas_cursor0-None]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex_as_pandas[pandas_cursor1-1000]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex_as_pandas[pandas_cursor2-1000000]
/Users/foobar/github/PyAthena/pyathena/pandas/result_set.py:214: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[datetime.time(0, 0)]' has dtype incompatible with datetime64[ns], please explicitly cast to a compatible dtype first.
df.loc[:, times] = df.loc[:, times].apply(lambda r: r.dt.time)
tests/pyathena/sqlalchemy/test_base.py: 65 warnings
/Users/foobar/github/PyAthena/tests/pyathena/conftest.py:96: SADeprecationWarning: The dbapi() classmethod on dialect classes has been renamed to import_dbapi(). Implement an import_dbapi() classmethod directly on class <class 'pyathena.sqlalchemy.rest.AthenaRestDialect'> to remove this warning; the old .dbapi() classmethod may be maintained for backwards compatibility.
return sqlalchemy.engine.create_engine(
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------- coverage: platform darwin, python 3.11.8-final-0 ----------
Name Stmts Miss Cover
---------------------------------------------------------
pyathena/__init__.py 43 9 79%
pyathena/arrow/__init__.py 0 0 100%
pyathena/arrow/async_cursor.py 42 0 100%
pyathena/arrow/converter.py 32 2 94%
pyathena/arrow/cursor.py 90 1 99%
pyathena/arrow/result_set.py 148 14 91%
pyathena/arrow/util.py 45 3 93%
pyathena/async_cursor.py 54 1 98%
pyathena/common.py 291 47 84%
pyathena/connection.py 127 32 75%
pyathena/converter.py 81 9 89%
pyathena/cursor.py 72 1 99%
pyathena/error.py 21 0 100%
pyathena/fastparquet/__init__.py 0 0 100%
pyathena/fastparquet/util.py 44 3 93%
pyathena/filesystem/__init__.py 0 0 100%
pyathena/filesystem/s3.py 279 74 73%
pyathena/filesystem/s3_object.py 34 0 100%
pyathena/formatter.py 99 5 95%
pyathena/model.py 490 13 97%
pyathena/pandas/__init__.py 3 0 100%
pyathena/pandas/async_cursor.py 44 0 100%
pyathena/pandas/converter.py 23 0 100%
pyathena/pandas/cursor.py 97 1 99%
pyathena/pandas/result_set.py 230 26 89%
pyathena/pandas/util.py 160 6 96%
pyathena/result_set.py 523 97 81%
pyathena/spark/__init__.py 0 0 100%
pyathena/spark/async_cursor.py 34 5 85%
pyathena/spark/common.py 188 48 74%
pyathena/spark/cursor.py 32 2 94%
pyathena/sqlalchemy/__init__.py 0 0 100%
pyathena/sqlalchemy/arrow.py 15 15 0%
pyathena/sqlalchemy/base.py 526 71 87%
pyathena/sqlalchemy/pandas.py 19 19 0%
pyathena/sqlalchemy/requirements.py 98 98 0%
pyathena/sqlalchemy/rest.py 4 0 100%
pyathena/sqlalchemy/types.py 31 9 71%
pyathena/sqlalchemy/util.py 3 1 67%
pyathena/util.py 31 1 97%
---------------------------------------------------------
TOTAL 4053 613 85%
Coverage HTML written to dir htmlcov
============================================================================================================================================================================================================ 501 passed, 1 skipped, 71 warnings in 329.07s (0:05:29) =============================================================================================================================================================================================================when I run
make test hatch run chk cmd [1] | ruff check . cmd [2] | ruff format --check . 43 files already formatted cmd [3] | mypy . Success: no issues found in 41 source files hatch run test ImportError while loading conftest '/Users/allanmiller/Source/PyAthena/tests/pyathena/conftest.py'. tests/__init__.py:38: in <module> ENV = Env() tests/__init__.py:15: in __init__ assert self.region_name, "Required environment variableAWS_DEFAULT_REGIONnot found." E AssertionError: Required environment variableAWS_DEFAULT_REGIONnot found. make: *** [test] Error 4
So I'm not certain if I missed something. Please let me know if I did.
laughingman7743
commented
7 months ago For more information on running the test, please check the following https://laughingman7743.github.io/PyAthena/testing.html Some environment variables are required; an AWS account is also required.
laughingman7743
commented
7 months ago $ make test
hatch run chk
cmd [1] | ruff check .
cmd [2] | ruff format --check .
43 files already formatted
cmd [3] | mypy .
Success: no issues found in 41 source files
hatch run test
============================================================================================================================================================================================================================== test session starts ===============================================================================================================================================================================================================================
platform darwin -- Python 3.11.8, pytest-8.0.0, pluggy-1.4.0
rootdir: /Users/foobar/github/PyAthena
configfile: pyproject.toml
plugins: cov-4.1.0, dependency-0.6.0, xdist-3.5.0, anyio-4.2.0
8 workers [502 items]
...........................................................................................................................................................................................................................................................................................................................................................................................................................................................s.............. [ 91%]
............................................ [100%]
================================================================================================================================================================================================================================ warnings summary ================================================================================================================================================================================================================================
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex[pandas_cursor0-None]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex_as_pandas[pandas_cursor0-None]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex[pandas_cursor1-1000]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex_as_pandas[pandas_cursor1-1000]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex[pandas_cursor2-1000000]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex_as_pandas[pandas_cursor2-1000000]
/Users/foobar/github/PyAthena/pyathena/pandas/result_set.py:214: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[datetime.time(0, 0)]' has dtype incompatible with datetime64[ns], please explicitly cast to a compatible dtype first.
df.loc[:, times] = df.loc[:, times].apply(lambda r: r.dt.time)
tests/pyathena/sqlalchemy/test_base.py: 65 warnings
/Users/foobar/github/PyAthena/tests/pyathena/conftest.py:96: SADeprecationWarning: The dbapi() classmethod on dialect classes has been renamed to import_dbapi(). Implement an import_dbapi() classmethod directly on class <class 'pyathena.sqlalchemy.rest.AthenaRestDialect'> to remove this warning; the old .dbapi() classmethod may be maintained for backwards compatibility.
return sqlalchemy.engine.create_engine(
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------- coverage: platform darwin, python 3.11.8-final-0 ----------
Name Stmts Miss Cover
---------------------------------------------------------
pyathena/__init__.py 43 9 79%
pyathena/arrow/__init__.py 0 0 100%
pyathena/arrow/async_cursor.py 42 0 100%
pyathena/arrow/converter.py 32 2 94%
pyathena/arrow/cursor.py 90 1 99%
pyathena/arrow/result_set.py 148 14 91%
pyathena/arrow/util.py 45 3 93%
pyathena/async_cursor.py 54 1 98%
pyathena/common.py 291 47 84%
pyathena/connection.py 127 32 75%
pyathena/converter.py 81 9 89%
pyathena/cursor.py 72 1 99%
pyathena/error.py 21 0 100%
pyathena/fastparquet/__init__.py 0 0 100%
pyathena/fastparquet/util.py 44 3 93%
pyathena/filesystem/__init__.py 0 0 100%
pyathena/filesystem/s3.py 279 74 73%
pyathena/filesystem/s3_object.py 34 0 100%
pyathena/formatter.py 99 5 95%
pyathena/model.py 490 13 97%
pyathena/pandas/__init__.py 3 0 100%
pyathena/pandas/async_cursor.py 44 0 100%
pyathena/pandas/converter.py 23 0 100%
pyathena/pandas/cursor.py 97 1 99%
pyathena/pandas/result_set.py 230 26 89%
pyathena/pandas/util.py 160 6 96%
pyathena/result_set.py 523 97 81%
pyathena/spark/__init__.py 0 0 100%
pyathena/spark/async_cursor.py 34 5 85%
pyathena/spark/common.py 188 48 74%
pyathena/spark/cursor.py 32 2 94%
pyathena/sqlalchemy/__init__.py 0 0 100%
pyathena/sqlalchemy/arrow.py 15 15 0%
pyathena/sqlalchemy/base.py 526 71 87%
pyathena/sqlalchemy/pandas.py 19 19 0%
pyathena/sqlalchemy/requirements.py 98 98 0%
pyathena/sqlalchemy/rest.py 4 0 100%
pyathena/sqlalchemy/types.py 31 9 71%
pyathena/sqlalchemy/util.py 3 1 67%
pyathena/util.py 31 1 97%
---------------------------------------------------------
TOTAL 4053 613 85%
Coverage HTML written to dir htmlcov
============================================================================================================================================================================================================ 501 passed, 1 skipped, 71 warnings in 328.65s (0:05:28) =============================================================================================================================================================================================================
Hopefully this is a decent stopgap measure to the issue I raised.