dask / fastparquet

python implementation of the parquet columnar file format.
Apache License 2.0
787 stars 178 forks source link

Numpy version confliction #943

Closed 988158 closed 2 weeks ago

988158 commented 2 weeks ago

Describe the issue: I got an error when I'm using fastqarquet in my project:

File "C:\Users\apps\core\ScenarioDataProvider.py", line 63, in loadDataset    datasetName, dataset = self.__loadDataset(datasetName, selection)
File "C:\Users\apps\core\ScenarioDataProvider.py", line 40, in __loadDataset    scenarioDataset = pd.read_parquet(io.BytesIO(blob_data))
File "C:\Users\apps\.venv\Lib\site-packages\pandas\io\parquet.py", line 667, in read_parquet    return impl.read(
File "C:\Users\apps\.venv\Lib\site-packages\pandas\io\parquet.py", line 402, in read    parquet_file = self.api.ParquetFile(path, **parquet_kwargs)
File "C:\Users\apps\.venv\Lib\site-packages\fastparquet\api.py", line 135, in __init__    self._parse_header(fn, verify) 
File "C:\Users\apps\.venv\Lib\site-packages\fastparquet\api.py", line 236, in _parse_header    self._set_attrs()  
File "C:\Users\apps\.venv\Lib\site-packages\fastparquet\api.py", line 250, in _set_attrs    self._dtypes()  
File "C:\Users\apps\.venv\Lib\site-packages\fastparquet\api.py", line 996, in _dtypes    dtype[col] = np.float_()                 
File "C:\Users\apps\.venv\Lib\site-packages\numpy\__init__.py", line 414, in __getattr__    raise AttributeError(AttributeError: np.float_ was removed in the NumPy 2.0 release. Use np.float64 instead.

I am using fastparquet version 2024.5.0 and numpy version 2.1.2.

I tried to downgrade the numpy version to 1.26.4, but it broke other parts of the code.

I temporarily fixed this issue by manually going into my local fastparquet folder in my virutal environment and replace the np.float_ with np.float64.

But I am not sure if there's a better way of solving this issue

Thanks in advance!


Environment:

martindurant commented 2 weeks ago

Recent CI run succeeded with numpy 2.1.2 (see logs). Can you test with the main branch version of fastparquet? Perhaps it just needs a release.

martindurant commented 2 weeks ago

I think was fixed in 36ed695c41882b82aa239d4b21d44506f6c65260

988158 commented 2 weeks ago

Hey Martin, thank you so much for replying! The issue is gone after trying the latest version.