search

pytroll / satpy

Python package for earth-observing satellite data processing
http://satpy.readthedocs.org/en/latest/
GNU General Public License v3.0
1.05k stars 289 forks source link

Scenes vanish from MultiScene after resampling #1273

Open gerritholl opened 4 years ago

gerritholl commented 4 years ago

Describe the bug

When I .resample(...) a fresh MultiScene before I have accessed the scenes from the source MultiScene, accessing the scenes from one MultiScene makes them disappear from the other.

To Reproduce

# Your code here
import satpy
import satpy.utils
satpy.utils.debug_on()
import glob

ms2 = satpy.MultiScene.from_files(
        glob.glob("/path/to/ABI/C14/OR_ABI-L1b-RadF-M6C14_G16_s202010401*_e*_c*.nc"),
        reader="abi_l1b")
ms2.load(["C14"])
ms3 = ms2.resample("northamerica")

print(len(ms2.scenes), len(ms3.scenes))

Expected behavior

I expect that len(ms2.scenes) and len(ms3.scenes) are equal.

Actual results

The final result is

6 0

i.e. the scenes that were present in ms2 have vanished from ms3. The full output:

Full output ``` [DEBUG: 2020-07-24 10:15:07 : satpy.readers] Reading ['/home/gholl/checkouts/satpy/satpy/etc/readers/abi_l1b.yaml'] [DEBUG: 2020-07-24 10:15:07 : satpy.multiscene] Forcing iteration of generator-like object of Scenes [DEBUG: 2020-07-24 10:15:07 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/satpy/satpy/etc/' [DEBUG: 2020-07-24 10:15:07 : satpy.readers] Reading ['/home/gholl/checkouts/satpy/satpy/etc/readers/abi_l1b.yaml'] [DEBUG: 2020-07-24 10:15:07 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['/data/gholl/cache/fogtools/abi/2020/04/13/01/C14/OR_ABI-L1b-RadF-M6C14_G16_s20201040100169_e20201040109477_c20201040109537.nc'] [DEBUG: 2020-07-24 10:15:07 : satpy.composites] Looking for composites config file abi.yaml [DEBUG: 2020-07-24 10:15:07 : satpy.composites] Looking for composites config file visir.yaml /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() [DEBUG: 2020-07-24 10:15:07 : satpy.readers.abi_l1b] Reading in get_dataset C14. [DEBUG: 2020-07-24 10:15:07 : satpy.readers.abi_l1b] Calibrating to brightness temperatures [DEBUG: 2020-07-24 10:15:07 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/satpy/satpy/etc/' [DEBUG: 2020-07-24 10:15:07 : satpy.readers] Reading ['/home/gholl/checkouts/satpy/satpy/etc/readers/abi_l1b.yaml'] [DEBUG: 2020-07-24 10:15:08 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['/data/gholl/cache/fogtools/abi/2020/04/13/01/C14/OR_ABI-L1b-RadF-M6C14_G16_s20201040110169_e20201040119477_c20201040119549.nc'] [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file abi.yaml [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file visir.yaml /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Reading in get_dataset C14. [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Calibrating to brightness temperatures [DEBUG: 2020-07-24 10:15:08 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/satpy/satpy/etc/' [DEBUG: 2020-07-24 10:15:08 : satpy.readers] Reading ['/home/gholl/checkouts/satpy/satpy/etc/readers/abi_l1b.yaml'] [DEBUG: 2020-07-24 10:15:08 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['/data/gholl/cache/fogtools/abi/2020/04/13/01/C14/OR_ABI-L1b-RadF-M6C14_G16_s20201040120169_e20201040129477_c20201040129562.nc'] [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file abi.yaml [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file visir.yaml /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Reading in get_dataset C14. [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Calibrating to brightness temperatures [DEBUG: 2020-07-24 10:15:08 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/satpy/satpy/etc/' [DEBUG: 2020-07-24 10:15:08 : satpy.readers] Reading ['/home/gholl/checkouts/satpy/satpy/etc/readers/abi_l1b.yaml'] [DEBUG: 2020-07-24 10:15:08 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['/data/gholl/cache/fogtools/abi/2020/04/13/01/C14/OR_ABI-L1b-RadF-M6C14_G16_s20201040130169_e20201040139477_c20201040139544.nc'] [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file abi.yaml [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file visir.yaml /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Reading in get_dataset C14. [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Calibrating to brightness temperatures [DEBUG: 2020-07-24 10:15:08 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/satpy/satpy/etc/' [DEBUG: 2020-07-24 10:15:08 : satpy.readers] Reading ['/home/gholl/checkouts/satpy/satpy/etc/readers/abi_l1b.yaml'] [DEBUG: 2020-07-24 10:15:08 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['/data/gholl/cache/fogtools/abi/2020/04/13/01/C14/OR_ABI-L1b-RadF-M6C14_G16_s20201040140169_e20201040149477_c20201040149556.nc'] [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file abi.yaml [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file visir.yaml /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Reading in get_dataset C14. [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Calibrating to brightness temperatures [DEBUG: 2020-07-24 10:15:08 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/satpy/satpy/etc/' [DEBUG: 2020-07-24 10:15:08 : satpy.readers] Reading ['/home/gholl/checkouts/satpy/satpy/etc/readers/abi_l1b.yaml'] [DEBUG: 2020-07-24 10:15:08 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['/data/gholl/cache/fogtools/abi/2020/04/13/01/C14/OR_ABI-L1b-RadF-M6C14_G16_s20201040150169_e20201040159477_c20201040159493.nc'] [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file abi.yaml [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file visir.yaml /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Reading in get_dataset C14. [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Calibrating to brightness temperatures [DEBUG: 2020-07-24 10:15:08 : satpy.multiscene] Forcing iteration of generator-like object of Scenes 6 0 ```

Environment Info:

Additional context

Interestingly, despite ms3.scenes being an empty list, I can access ms3.first_scene, but only before I acces ms3.scenes. If I change the final lines to:

ms3.first_scene["C14"]
print(len(ms2.scenes), len(ms3.scenes))

then the final lines of the output become:

[DEBUG: 2020-07-24 10:23:00 : satpy.readers.abi_l1b] Reading in get_dataset C14.
[DEBUG: 2020-07-24 10:23:00 : satpy.readers.abi_l1b] Calibrating to brightness temperatures
[DEBUG: 2020-07-24 10:23:00 : satpy.multiscene] Forcing iteration of generator-like object of Scenes
5 1

but if I swap the final two lines around such that they are

print(len(ms2.scenes), len(ms3.scenes))
ms3.first_scene["C14"]

then I get StopIteration:

6 0
Traceback (most recent call last):
  File "mwe75.py", line 12, in <module>
    ms3.first_scene["C14"]
  File "/home/gholl/checkouts/satpy/satpy/multiscene.py", line 174, in first_scene
    return self._scene_gen.first
  File "/home/gholl/checkouts/satpy/satpy/multiscene.py", line 113, in first
    return next(iter(self))
StopIteration

If I swap sc2 and sc3:

print(len(ms3.scenes), len(ms2.scenes))

then the final output line remains

6 0

i.e. in this case it's ms2 that is empty rather than ms3.

If I access ms2.scenes before the resampling, the result is as expected:

print(len(ms2.scenes))
ms3 = ms2.resample("northamerica")
print(len(ms2.scenes), len(ms3.scenes))

results in

6 6
djhoese commented 4 years ago

Short answer/excuse is that the MultiScene is using generators internally to avoid creating a ton of Scene objects. Trying to list and use these generators in different places is bound to have unexpected results. That said, I think there might be a simple fix if a new pointer/copy to the main generator object is provided to the resampled MultiScene (I tried to design it so this would work).

gerritholl commented 1 month ago

Trying to list and use these generators in different places is bound to have unexpected results

Maybe it should be private (._scenes) then?

djhoese commented 1 month ago

I think I had considered that but decided against it in the short term because it is the only way to easily access some of this stuff. A simple solution might be to rename it to express the generator-ness of the attribute. Another would be to define better interfaces with more sophisticated iterable handling. We could probably rely on dask Delayed objects more now that we're better at serialization and dask.distributed compatibility.