search

Snowflake-Labs / schemachange

A Database Change Management tool for Snowflake
Apache License 2.0
517 stars 228 forks source link

Update of Numpy to 2.0 has broken schemachange #256

Closed cuddebtj closed 4 months ago

cuddebtj commented 5 months ago

Describe the bug A ValueError occurs when attempting to run SchemaChange, this did not occur last Friday (6/14) but is occurring this Monday (6/17).

To Reproduce Steps to reproduce the behavior:

  1. pip install schemachange
  2. run schemachange

Expected behavior Application of a versioned migration and repeatable files.

Screenshots 2024-06-17T15:57:56.5457992Z Traceback (most recent call last): 2024-06-17T15:57:56.5458535Z File "/opt/hostedtoolcache/Python/3.10.14/x64/bin/schemachange", line 5, in 2024-06-17T15:57:56.5458837Z from schemachange.cli import main 2024-06-17T15:57:56.5459520Z File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/schemachange/cli.py", line 15, in 2024-06-17T15:57:56.5459791Z import snowflake.connector 2024-06-17T15:57:56.5460319Z File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/snowflake/connector/init.py", line 19, in 2024-06-17T15:57:56.5460671Z from .connection import SnowflakeConnection 2024-06-17T15:57:56.5461162Z File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/snowflake/connector/connection.py", line 35, in 2024-06-17T15:57:56.5461731Z from . import errors, proxy 2024-06-17T15:57:56.5462137Z File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/snowflake/connector/errors.py", line 15, in 2024-06-17T15:57:56.5462514Z from .compat import BASE_EXCEPTION_CLASS 2024-06-17T15:57:56.5462948Z File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/snowflake/connector/compat.py", line 19, in 2024-06-17T15:57:56.5463285Z from . import constants 2024-06-17T15:57:56.5463719Z File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/snowflake/connector/constants.py", line 12, in 2024-06-17T15:57:56.5463989Z from .options import pyarrow as pa 2024-06-17T15:57:56.5464397Z File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/snowflake/connector/options.py", line 135, in 2024-06-17T15:57:56.5464708Z pandas, pyarrow, installed_pandas = _import_or_missing_pandas_option() 2024-06-17T15:57:56.5465295Z File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/snowflake/connector/options.py", line 77, in _import_or_missing_pandas_option 2024-06-17T15:57:56.5465683Z pandas = importlib.import_module("pandas") 2024-06-17T15:57:56.5465951Z File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/importlib/init.py", line 126, in import_module 2024-06-17T15:57:56.5515033Z return _bootstrap._gcd_import(name[level:], package, level) 2024-06-17T15:57:56.5516339Z File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/pandas/init.py", line 22, in 2024-06-17T15:57:56.5516827Z from pandas.compat import is_numpy_dev as _is_numpy_dev # pyright: ignore # noqa:F401 2024-06-17T15:57:56.5517385Z File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/pandas/compat/init.py", line 18, in 2024-06-17T15:57:56.5518015Z from pandas.compat.numpy import ( 2024-06-17T15:57:56.5518522Z File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/pandas/compat/numpy/init.py", line 4, in 2024-06-17T15:57:56.5518894Z from pandas.util.version import Version 2024-06-17T15:57:56.5519393Z File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/pandas/util/init.py", line 2, in 2024-06-17T15:57:56.5519762Z from pandas.util._decorators import ( # noqa:F401 2024-06-17T15:57:56.5520269Z File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/pandas/util/_decorators.py", line 14, in 2024-06-17T15:57:56.5520620Z from pandas._libs.properties import cache_readonly 2024-06-17T15:57:56.5521222Z File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/pandas/_libs/init.py", line 13, in 2024-06-17T15:57:56.5521536Z from pandas._libs.interval import Interval 2024-06-17T15:57:56.5521835Z File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval 2024-06-17T15:57:56.5522229Z ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

Schemachange (please complete the following information):

eddie-brenes commented 5 months ago

I have the same issue but the current workaround that I have is to run this after the schemachange install: pip install --force-reinstall numpy==1.26.0

cuddebtj commented 5 months ago

I have the same issue but the current workaround that I have is to run this after the schemachange install: pip install --force-reinstall numpy==1.26.0

I am doing that as well, but feel this is more of a band aid and anyone looking to use schemachange will run into the same issue until it is resolved.

nmcclintock commented 5 months ago

I have the same issue but the current workaround that I have is to run this after the schemachange install: pip install --force-reinstall numpy==1.26.0

After seeing this, I was able to use the same technique as a workaround. Thank you for mentioning this.

sfc-gh-dszmolka commented 5 months ago

this looks to be numpy issue https://github.com/numpy/numpy/issues/26710 and adding numpy<2 to your requirements file might provide a relief until this is sorted

Serendipitique commented 5 months ago

installing schemachange with -c option with numpy==1.26.4 in the constraints.txt also works. also, next release will probably remove the pandas dependency (https://github.com/Snowflake-Labs/schemachange/issues/229) which I think removes numpy from the dependencies here.

recruz02 commented 5 months ago

For what it's worth, my team is also running into this issue. We are currently unable to resolve it, due to the nature in which we are attempting to run.

We are running via Gitlab, their docker-in-docker service. We are running a pre-compiled docker image (google/cloud-sdk:slim), from there we are (pipx) installing schemachange. After installing schemachange, we tried force installing a lowered version of numpy, but to no avail.

We will continue to try different solutions, but this is our version of the problem. We have temporarily disabled use of schemachange for our project until this package gets updated appropriately.

BoAustin commented 5 months ago

For those of us who have installed schemachange with pipx be aware pipx creates venvs which is going to isolate the installed packages, so you need to "inject" the package change:

$ pipx inject --force schemachange "numpy<2"
  injected package numpy into venv schemachange
done! ✨ 🌟 ✨

schemachange should now work but this is a bandaid.

recruz02 commented 5 months ago

For those of us who have installed schemachange with pipx be aware pipx creates venvs which is going to isolate the installed packages, so you need to "inject" the package change:

$ pipx inject --force schemachange "numpy<2"
  injected package numpy into venv schemachange
done! ✨ 🌟 ✨

schemachange should now work but this is a bandaid.

Wow, thanks for this pro-tip. I'll give this a try -- but yes, this is a band-aid! The schemachange package needs to be updated appropriately