search

osgeonepal / geotile

GeoTile is the python library for tiling the geographic raster data (eg. Tiff etc)
https://geotile.readthedocs.io
MIT License
74 stars 9 forks source link

MemoryError: Unable to allocate ... GiB for an array with shape (num_features, 256, 256, 4) and data type `float64` #42

Closed iamtekson closed 11 months ago

iamtekson commented 11 months ago 
---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
[c:\Users\tek.kshetri\OneDrive](file:///C:/Users/tek.kshetri/OneDrive) - University of Calgary (1)\sand_mining\code\02. generate_tiles\imagery_&_mask_tiles_near_ratu.ipynb Cell 11 line 2
      [1](vscode-notebook-cell:/c%3A/Users/tek.kshetri/OneDrive%20-%20University%20of%20Calgary%20%281%29/sand_mining/code/02.%20generate_tiles/imagery_%26_mask_tiles_near_ratu.ipynb#X20sZmlsZQ%3D%3D?line=0) # tile_data = tile_data.astype(np.uint16)
----> [2](vscode-notebook-cell:/c%3A/Users/tek.kshetri/OneDrive%20-%20University%20of%20Calgary%20%281%29/sand_mining/code/02.%20generate_tiles/imagery_%26_mask_tiles_near_ratu.ipynb#X20sZmlsZQ%3D%3D?line=1) gt.save_numpy(f'{OUTPUT_TILES}/planet_near_ratu.npy')
      [3](vscode-notebook-cell:/c%3A/Users/tek.kshetri/OneDrive%20-%20University%20of%20Calgary%20%281%29/sand_mining/code/02.%20generate_tiles/imagery_%26_mask_tiles_near_ratu.ipynb#X20sZmlsZQ%3D%3D?line=2) # np.save(f'{OUTPUT_TILES}/planet_near_ratu.npy', tile_data)

File [c:\Users\tek.kshetri\.conda\envs\gee\lib\site-packages\geotile\GeoTile.py:573](file:///C:/Users/tek.kshetri/.conda/envs/gee/lib/site-packages/geotile/GeoTile.py:573), in GeoTile.save_numpy(self, file_name, dtype)
    570     self.tile_data = np.array(self.tile_data)
    572 # save the numpy file
--> 573 np.save(file_name, self.tile_data.astype(dtype))

MemoryError: Unable to allocate 73.9 GiB for an array with shape (37842, 256, 256, 4) and data type float64
iamtekson commented 11 months ago

If you encountered this error, make sure to mention the datatype as a parameter as below,

gt.save_numpy('filename.npy', dtype = np.float32)