search

ReneHeim / proj_on_uav

This repository contains the code to reproduce the article (https://doi.org/10.1007/s11119-024-10133-0). The presented method allows to extract oblique and nadir viewing angles from UAV-mounted, multispectral snapshot cameras.
https://reneheim.github.io/
Apache License 2.0
6 stars 3 forks source link

numpy.core._exceptions.MemoryError： need 6.03 TiB ? #1

Open MajorTom-LT opened 1 week ago

MajorTom-LT commented 1 week ago

When I am running code; Memory overflow detected; After removing parallel computing; Still experiencing memory explosion during travel， :

Break DEM into pixel: 760.102015 seconds 0%| | 0/1 [00:00<?, ?it/s]Finding camera position for first ortho in first chunk: 0.022835200000031364 seconds Getting SAA and Sun Elevation from ortho EXIF data: 0.18809069999997519 seconds 0%| | 0/1 [04:31<?, ?it/s] Traceback (most recent call last): File "d:/Git_Depository/Research_Files/BRDF/workplace/proj_on_uav/main_public/01_main_extract_vza_aza_reflectance.py", line 212, in build_database(i)
File "d:/Git_Depository/Research_Files/BRDF/workplace/proj_on_uav/main_public/01_main_extract_vza_aza_reflectance.py", line 164, in build_database df_allbands = reduce(my_reduce, bands.values()) # iterates functions for all 11 bands File "D:\Work_Software\AnaConda\envs\micasense\lib\site-packages\pandas\core\reshape\merge.py", line 162, in merge return op.get_result(copy=copy) File "D:\Work_Software\AnaConda\envs\micasense\lib\site-packages\pandas\core\reshape\merge.py", line 809, in get_result join_index, left_indexer, right_indexer = self._get_join_info() File "D:\Work_Software\AnaConda\envs\micasense\lib\site-packages\pandas\core\reshape\merge.py", line 1065, in _get_join_info (left_indexer, right_indexer) = self._get_join_indexers() File "D:\Work_Software\AnaConda\envs\micasense\lib\site-packages\pandas\core\reshape\merge.py", line 1038, in _get_join_indexers return get_join_indexers( File "D:\Work_Software\AnaConda\envs\micasense\lib\site-packages\pandas\core\reshape\merge.py", line 1690, in get_join_indexers return join_func(lkey, rkey, count, **kwargs) # type: ignore[operator] File "pandas_libs\join.pyx", line 176, in pandas._libs.join.full_outer_join numpy.core._exceptions.MemoryError: Unable to allocate 6.03 TiB for an array with shape (828737122500,) and data type int64

mak8427 commented 1 week ago

Hello, Currently I am working on it. This code is currently being refactored to be memory stable and optimized, so expect changes. Here be Dragons.

ReneHeim commented 1 week ago

Hi @MajorTom-LT , are you running it on the example data or on data that you collected yourself? Thanks for opening the issue!