Breaking change with numpy 2.0.0

[isobel-taylor]

We've identified an issue in our data pipeline caused by the recent release of version 2.0.0 of numpy. The error is thrown by pandas:

2024-06-17T00:07:40.703462Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.703817Z [info     ]   File "/app/src/projects/ea_meltano/mpa_export/.meltano/loaders/target-snowflake/venv/bin/target-snowflake", line 5, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.704162Z [info     ]     from target_snowflake import main cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.704395Z [info     ]   File "/app/src/projects/ea_meltano/mpa_export/.meltano/loaders/target-snowflake/venv/lib/python3.11/site-packages/target_snowflake/__init__.py", line 18, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.704583Z [info     ]     from target_snowflake.file_formats import parquet cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.704823Z [info     ]   File "/app/src/projects/ea_meltano/mpa_export/.meltano/loaders/target-snowflake/venv/lib/python3.11/site-packages/target_snowflake/file_formats/parquet.py", line 3, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.705053Z [info     ]     import pandas              cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.709128Z [info     ]   File "/app/src/projects/ea_meltano/mpa_export/.meltano/loaders/target-snowflake/venv/lib/python3.11/site-packages/pandas/__init__.py", line 22, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.709460Z [info     ]     from pandas.compat import is_numpy_dev as _is_numpy_dev  # pyright: ignore # noqa:F401 cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.709691Z [info     ]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.709920Z [info     ]   File "/app/src/projects/ea_meltano/mpa_export/.meltano/loaders/target-snowflake/venv/lib/python3.11/site-packages/pandas/compat/__init__.py", line 25, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.710221Z [info     ]     from pandas.compat.numpy import ( cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.710407Z [info     ]   File "/app/src/projects/ea_meltano/mpa_export/.meltano/loaders/target-snowflake/venv/lib/python3.11/site-packages/pandas/compat/numpy/__init__.py", line 4, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.710584Z [info     ]     from pandas.util.version import Version cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.710760Z [info     ]   File "/app/src/projects/ea_meltano/mpa_export/.meltano/loaders/target-snowflake/venv/lib/python3.11/site-packages/pandas/util/__init__.py", line 2, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.710948Z [info     ]     from pandas.util._decorators import (  # noqa:F401 cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.711135Z [info     ]   File "/app/src/projects/ea_meltano/mpa_export/.meltano/loaders/target-snowflake/venv/lib/python3.11/site-packages/pandas/util/_decorators.py", line 14, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.711330Z [info     ]     from pandas._libs.properties import cache_readonly cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.711512Z [info     ]   File "/app/src/projects/ea_meltano/mpa_export/.meltano/loaders/target-snowflake/venv/lib/python3.11/site-packages/pandas/_libs/__init__.py", line 13, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.711709Z [info     ]     from pandas._libs.interval import Interval cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.711941Z [info     ]   File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2024-06-17T00:07:40.712244Z [info     ] ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake

Looking online, it seems that this version introduced a breaking change. We've remedied this by forcing numpy to version 1.26.4. We tried upgrading the outer venv to 2.0.0, but this did not fix the issue - it seems like this is a true incompatibility with version 2.0.0.

[edgarrmondragon]

I just released meltanolabs-target-snowflake 0.7.1 which pins numpy to <2 until all issues are fixed upstream :)

Thanks for logging this issue!

[isobel-taylor]

Awesome, thank you for jumping on this so fast!