search

ploomber / jupysql

Better SQL in Jupyter. 📊
https://jupysql.ploomber.io
Apache License 2.0
706 stars 75 forks source link

remove duplicated integration tests #827

Closed edublancas closed 1 year ago

edublancas commented 1 year ago

we have some duplicate integration tests, instead we should only write them once and parameterized them, just like we are already doing with many of our integration tests:

image

https://github.com/search?q=repo%3Aploomber%2Fjupysql%20test_query_count&type=code

bryannho commented 1 year ago

To clarify this issue, I should remove the duplicate test_query_count tests from test_mssql.py and test_oracle.py. Then I should modify test_query_count in test_generic_db_operations.py to include MSSQL and Oracle via parameterization?

edublancas commented 1 year ago

yes.

but those are some that I found. you might encounter other duplicated tests, so take look at the testing files. see if you spot anything duplicated

bryannho commented 1 year ago

I have attempted to run the integration tests using pytest src/tests/integration/test_generic_db_operations.py with my changes however am running into connection errors with all DB's. For example, clickhouse, oracle, and mssql return errors:

FAILED src/tests/integration/test_generic_db_operations.py::test_run_query[ip_with_clickhouse--LIMIT 3] - requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=N...

FAILED src/tests/integration/test_generic_db_operations.py::test_run_query[ip_with_oracle--FETCH FIRST 3 ROWS ONLY] - ModuleNotFoundError: No module named 'oracledb'

FAILED src/tests/integration/test_generic_db_operations.py::test_run_query[ip_with_MSSQL-TOP 3-] - ImportError: dlopen(/Users/bryanho/opt/anaconda3/envs/jupysql/lib/python3.10/sit...

It seems I am missing some setup but I cannot find a guide to running integration tests specifically. Is there anywhere else I can refer to troubleshoot this issue? Must I spin up an instance of each DB individually to ensure tests run correctly?

edublancas commented 1 year ago

we have an integration testing guide here: https://jupysql.ploomber.io/en/latest/community/developer-guide.html#integration-tests there's a command there that shows you how to run tests for specific dbs

looks like you're missing some dependencies, so you need to install them. the integration tests automatically spin up a database and shut it down (except for snowflake and redshift). the guide might be inaccurate since we haven't updated it so if you find any inaccuracies, open a PR.

bryannho commented 1 year ago

we have an integration testing guide here: https://jupysql.ploomber.io/en/latest/community/developer-guide.html#integration-tests there's a command there that shows you how to run tests for specific dbs

Thank you, I was able to solve most of the dependency issues with the guide. Will continue working

bryannho commented 1 year ago

After following the developer guide I have fixed a few issues however I am still not able to get integration tests for Oracle or MSSQL working. To run the tests, I ensure that Docker is running and execute pytest src/tests/integration -k oracle or pytest src/tests/integration -k mssql where I run into these errors:

Oracle Tests: sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:oracle.oracledb

MSSQL Tests: ImportError: dlopen(/Users/my_user/opt/anaconda3/envs/jupysql/lib/python3.10/site-packages/pyodbc.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace (_SQLAllocHandle)

Thus far, I have tried:

  1. Installing all dependencies with: pip install -e '.[integration]'
  2. Installing a new version of Docker Desktop
  3. For Oracle: Installing Colima and running colima start --cpu 4 --memory 4 --disk 30 --arch x86_64
  4. For Oracle: Experimenting with different versions of SQLAlchemy, although this breaks other tests
  5. For Oracle: Adding this code to conftest.py in accordance with this guide: import sys import oracledb oracledb.version = "8.3.0" sys.modules["cx_Oracle"] = oracledb
  6. For MSSQL: Reinstalling mssql and pyodbc with: brew tap microsoft/mssql-release https://github.com/Microsoft/homebrew-mssql-release brew update brew upgrade HOMEBREW_ACCEPT_EULA=Y brew install msodbcsql18 mssql-tools18 pip3 install --pre --no-binary :all: pyodbc --force-reinstall

The issue may be related to Apple Silicon, as I am running everything on Apple M1 Macbook Pro. Please let me know if these are known issues. I am aware that running integration tests locally is tricky, so I'll continue looking for workarounds.

edublancas commented 1 year ago

at this point, it might be better to just run them in the CI, so you don't have to find workarounds