search

CCI-Tools / cate

ESA CCI Toolbox (Cate)
MIT License
50 stars 15 forks source link

Laptop crashed when opening local data source #274

Closed HelenClifton closed 7 years ago

HelenClifton commented 7 years ago

Expected behaviour

I opened a local copy of data source esacci.CLOUD.mon.L3C.CLD_PRODUCTS.AVHRR.NOAA-17.AVHRR_NOAA.1-0.r1. I expected it to open.

Actual behavior

After a while all the applications on my laptop froze and then my laptop crashed and I had to manually shut down. I think it could be a problem with the input data source, but in any event, cate should be able to catch this type of error.

Steps to reproduce the problem

  1. Open Windows Task Manager, select Processes.
  2. Open CLI, type: cate ds copy esacci.CLOUD.mon.L3C.CLD_PRODUCTS.AVHRR.multi-platform.AVHRR_AM.2-0.r1 -n cld_2007 -t 2007-01-01,2007-12-31
  3. Wait for local copy to load.
  4. Type: cate res open cloud local.cld_2007
  5. Watch Windows Task Manager and observe python.exe process using more and more physical memory.
  6. Eventually machine crashes.

Note: This problem does NOT occur when the data source is opened with time constraints of 2007-01-01 to 2007-06-28. It is only when the end date is set to later than that, e.g. 2007-06-30, that the problem happens. So it looks like it could be a problem with the data itself, but cate should be able to handle bad data without causing machine to crash. It could also be a problem with the way cate is reading the remote data and applying the temporal subsetting. Needs to be investigated.

Same behaviour is seen when the cate GUI is used.

Specifications

cate 0.8.0-rc6 Internet Explorer 11 Windows 7 Professional

mzuehlke commented 7 years ago

This issue has the same reason as #279. The re-indexing of the datasets to align them requires a lot of memory and CPU times. It is dependent on the selected time range (but not visible to the user) whether the individual netCDF file have the same latitude and longitude values or not.

mzuehlke commented 7 years ago

This is an issue of the data set: https://github.com/CCI-Tools/cate-core/wiki/Problems-with-ODP-datasets