Memory issue with Satellite imagery on Windows

[seamuslo]

After installing gdal to work with the new additions to the code (which was surprisingly difficult), I am now able to run the code with its new additions.

However, when I first boot up the prototype, I can these messages: an't load requested DLL: C:\Program Files\GDAL\gdalplugins\gdal_BAG.dll 193: %1 is not a valid Win32 application.

ERROR 1: Can't load requested DLL: C:\Program Files\GDAL\gdalplugins\gdal_BAG.dll 193: %1 is not a valid Win32 application.

ERROR 1: Can't load requested DLL: C:\Program Files\GDAL\gdalplugins\gdal_FITS.dll 193: %1 is not a valid Win32 application.

ERROR 1: Can't load requested DLL: C:\Program Files\GDAL\gdalplugins\gdal_FITS.dll 193: %1 is not a valid Win32 application.

ERROR 1: Can't load requested DLL: C:\Program Files\GDAL\gdalplugins\gdal_GMT.dll 193: %1 is not a valid Win32 application.

ERROR 1: Can't load requested DLL: C:\Program Files\GDAL\gdalplugins\gdal_GMT.dll 193: %1 is not a valid Win32 application.

ERROR 1: Can't load requested DLL: C:\Program Files\GDAL\gdalplugins\gdal_HDF4.dll 193: %1 is not a valid Win32 application.

ERROR 1: Can't load requested DLL: C:\Program Files\GDAL\gdalplugins\gdal_HDF4.dll 193: %1 is not a valid Win32 application.

ERROR 1: Can't load requested DLL: C:\Program Files\GDAL\gdalplugins\gdal_HDF4Image.dll 193: %1 is not a valid Win32 application.

ERROR 1: Can't load requested DLL: C:\Program Files\GDAL\gdalplugins\gdal_HDF4Image.dll 193: %1 is not a valid Win32 application.

ERROR 1: Can't load requested DLL: C:\Program Files\GDAL\gdalplugins\gdal_HDF5.dll 193: %1 is not a valid Win32 application.

ERROR 1: Can't load requested DLL: C:\Program Files\GDAL\gdalplugins\gdal_HDF5.dll 193: %1 is not a valid Win32 application.

ERROR 1: Can't load requested DLL: C:\Program Files\GDAL\gdalplugins\gdal_HDF5Image.dll 193: %1 is not a valid Win32 application.

ERROR 1: Can't load requested DLL: C:\Program Files\GDAL\gdalplugins\gdal_HDF5Image.dll 193: %1 is not a valid Win32 application.

ERROR 1: Can't load requested DLL: C:\Program Files\GDAL\gdalplugins\gdal_KEA.dll 193: %1 is not a valid Win32 application.

ERROR 1: Can't load requested DLL: C:\Program Files\GDAL\gdalplugins\gdal_KEA.dll 193: %1 is not a valid Win32 application.

ERROR 1: Can't load requested DLL: C:\Program Files\GDAL\gdalplugins\gdal_netCDF.dll 193: %1 is not a valid Win32 application.

ERROR 1: Can't load requested DLL: C:\Program Files\GDAL\gdalplugins\gdal_netCDF.dll 193: %1 is not a valid Win32 application.

Then, when I attempt to use the drop down menu to view the visual composite imagery in the Rio De Janeiro case, I get this error: Exception in Tkinter callback Traceback (most recent call last): File "C:\Users\slomb\Python\Python38-32\lib\tkinter__init.py", line 1883, in call return self.func(args) File "C:\Users\slomb\Python\Python38-32\lib\tkinter__init.py", line 3943, in call self.callback(self.__value, args) File "c:/Users/slomb/OneDrive/Desktop/SD_UI/SD_UI_v1_5.py", line 774, in command=lambda e: self.replace_map_image(self.subframe_map) File "c:/Users/slomb/OneDrive/Desktop/SD_UI/SD_UI_v1_5.py", line 838, in replace_map_image self.MAP = MapWindow.Map(mapframe, File "c:\Users\slomb\OneDrive\Desktop\SD_UI\MapWindow_v4.py", line 95, in init self.draw_background(imagename) File "c:\Users\slomb\OneDrive\Desktop\SD_UI\MapWindow_v4.py", line 269, in draw_background image_array8 = np.nan_to_num(image_array8) File "<array_function__ internals>", line 5, in nan_to_num File "C:\Users\slomb\OneDrive\Desktop\SD_UI.venv\lib\site-packages\numpy\lib\type_check.py", line 461, in nan_to_num x = _nx.array(x, subok=True, copy=copy) MemoryError: Unable to allocate 47.6 MiB for an array with shape (1088, 1912, 3) and data type float64

I get a very similar error when I attempt to load the nightlights: Exception in Tkinter callback Traceback (most recent call last): File "C:\Users\slomb\Python\Python38-32\lib\tkinter__init.py", line 1883, in call return self.func(args) File "C:\Users\slomb\Python\Python38-32\lib\tkinter__init.py", line 3943, in call self.callback(self.__value, args) File "c:/Users/slomb/OneDrive/Desktop/SD_UI/SD_UI_v1_5.py", line 774, in command=lambda e: self.replace_map_image(self.subframe_map) File "c:/Users/slomb/OneDrive/Desktop/SD_UI/SD_UI_v1_5.py", line 838, in replace_map_image self.MAP = MapWindow.Map(mapframe, File "c:\Users\slomb\OneDrive\Desktop\SD_UI\MapWindow_v4.py", line 95, in init__ self.draw_background(imagename) File "c:\Users\slomb\OneDrive\Desktop\SD_UI\MapWindow_v4.py", line 250, in draw_background image_array = np.array(gdal.Open(imagename).ReadAsArray()) File "C:\Users\slomb\OneDrive\Desktop\SD_UI.venv\lib\site-packages\osgeo\gdal.py", line 2328, in ReadAsArray return gdalnumeric.DatasetReadAsArray(self, xoff, yoff, xsize, ysize, buf_obj, File "C:\Users\slomb\OneDrive\Desktop\SD_UI.venv\lib\site-packages\osgeo\gdal_array.py", line 297, in DatasetReadAsArray return BandReadAsArray(ds.GetRasterBand(1), xoff, yoff, win_xsize, win_ysize, File "C:\Users\slomb\OneDrive\Desktop\SD_UI.venv\lib\site-packages\osgeo\gdal_array.py", line 383, in BandReadAsArray buf_obj = numpy.empty([buf_ysize, buf_xsize], dtype=typecode) MemoryError: Unable to allocate 396. MiB for an array with shape (5439, 9555) and data type float64

[jackreid]

Regarding the first error, I saw this which might help you? https://stackoverflow.com/questions/47173459/error-windowserror-error-193-1-is-not-a-valid-win32-application-with-gdal . That one in general will be harder for me to figure out since it is on a different OS.

For the other issue, I am aware that this can be a thing, but have only heard of it for issues measured in 100's of GiB, not MiB. How much memory (RAM) does your computer have? There is a suggestion for windows about halfway down this page that might help: https://stackoverflow.com/questions/57507832/unable-to-allocate-array-with-shape-and-data-type

If those fail, on the geotiffs branch, I have added ./Date/Rio de Janeiro/Images/Landsat_Composite_lowres.tif. It's a much lower resolution image (500m/px). Try replacing the Landsat_Composite_unit8.tif with that one and seeing if you get the same error. That will at least help us narrow it down.

[seamuslo]

I spent A LOT of time trying to get this to work today and unfortunately I wasn't able to make any meaningful progress.

• I took a look at your first stack overflow link and installed OSGeo4W 32bit as it suggested, but I was unable to find the settings.py file to specify the path that was referred to in the answer. I wasn't able to get these error messages to go away, although they don't prevent it from running, so I'm not sure its related to the memory issue.

• I also took a look at your second stack overflow link. I messed around with the settings and still wasn't able to get the image composites to load without a memory error, although now I sometimes get a different error that does not reference a specific file size like the previous errors did: "

• I also tried using the much smaller geotiff in the branch you mentioned and still got a memory error like the one above.

I spent pretty much my whole day today trying to figure this out using your suggestions as well as other Stack Overflow postings I saw to no avail. I'm going to move on to other stuff for the rest of the day, but I will give this another shot Monday.

[seamuslo]

After looking into this for a few more hours today, I was able to get it to work by uninstalling python (which had been the 32-bit version) reinstalling python (the new 64 bit version) and re-cloning the directory. I can now do the satellite imagery with no memory errors (though it takes ~5 full seconds to load each image)

[jackreid]

Oh nifty. Glad that that was resolved. Yeah 5 seconds is longer than it takes on mind but is within the same order of magnitude which means it is probably just a processing this. We will want to keep an eye on resolutions / filesizes of the imagery that we add to make sure that it doesn't slow things down too much. Sorry this took so much effort to resolve, but now we can add Indonesia imagery if you would like.