Open gerritholl opened 4 years ago
Short answer/excuse is that the MultiScene is using generators internally to avoid creating a ton of Scene objects. Trying to list and use these generators in different places is bound to have unexpected results. That said, I think there might be a simple fix if a new pointer/copy to the main generator object is provided to the resampled MultiScene (I tried to design it so this would work).
Trying to list and use these generators in different places is bound to have unexpected results
Maybe it should be private (._scenes
) then?
I think I had considered that but decided against it in the short term because it is the only way to easily access some of this stuff. A simple solution might be to rename it to express the generator-ness of the attribute. Another would be to define better interfaces with more sophisticated iterable handling. We could probably rely on dask Delayed objects more now that we're better at serialization and dask.distributed compatibility.
Describe the bug
When I
.resample(...)
a freshMultiScene
before I have accessed the scenes from the sourceMultiScene
, accessing the scenes from one MultiScene makes them disappear from the other.To Reproduce
Expected behavior
I expect that
len(ms2.scenes)
andlen(ms3.scenes)
are equal.Actual results
The final result is
i.e. the scenes that were present in
ms2
have vanished fromms3
. The full output:Full output
``` [DEBUG: 2020-07-24 10:15:07 : satpy.readers] Reading ['/home/gholl/checkouts/satpy/satpy/etc/readers/abi_l1b.yaml'] [DEBUG: 2020-07-24 10:15:07 : satpy.multiscene] Forcing iteration of generator-like object of Scenes [DEBUG: 2020-07-24 10:15:07 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/satpy/satpy/etc/' [DEBUG: 2020-07-24 10:15:07 : satpy.readers] Reading ['/home/gholl/checkouts/satpy/satpy/etc/readers/abi_l1b.yaml'] [DEBUG: 2020-07-24 10:15:07 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['/data/gholl/cache/fogtools/abi/2020/04/13/01/C14/OR_ABI-L1b-RadF-M6C14_G16_s20201040100169_e20201040109477_c20201040109537.nc'] [DEBUG: 2020-07-24 10:15:07 : satpy.composites] Looking for composites config file abi.yaml [DEBUG: 2020-07-24 10:15:07 : satpy.composites] Looking for composites config file visir.yaml /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() [DEBUG: 2020-07-24 10:15:07 : satpy.readers.abi_l1b] Reading in get_dataset C14. [DEBUG: 2020-07-24 10:15:07 : satpy.readers.abi_l1b] Calibrating to brightness temperatures [DEBUG: 2020-07-24 10:15:07 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/satpy/satpy/etc/' [DEBUG: 2020-07-24 10:15:07 : satpy.readers] Reading ['/home/gholl/checkouts/satpy/satpy/etc/readers/abi_l1b.yaml'] [DEBUG: 2020-07-24 10:15:08 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['/data/gholl/cache/fogtools/abi/2020/04/13/01/C14/OR_ABI-L1b-RadF-M6C14_G16_s20201040110169_e20201040119477_c20201040119549.nc'] [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file abi.yaml [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file visir.yaml /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Reading in get_dataset C14. [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Calibrating to brightness temperatures [DEBUG: 2020-07-24 10:15:08 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/satpy/satpy/etc/' [DEBUG: 2020-07-24 10:15:08 : satpy.readers] Reading ['/home/gholl/checkouts/satpy/satpy/etc/readers/abi_l1b.yaml'] [DEBUG: 2020-07-24 10:15:08 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['/data/gholl/cache/fogtools/abi/2020/04/13/01/C14/OR_ABI-L1b-RadF-M6C14_G16_s20201040120169_e20201040129477_c20201040129562.nc'] [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file abi.yaml [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file visir.yaml /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Reading in get_dataset C14. [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Calibrating to brightness temperatures [DEBUG: 2020-07-24 10:15:08 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/satpy/satpy/etc/' [DEBUG: 2020-07-24 10:15:08 : satpy.readers] Reading ['/home/gholl/checkouts/satpy/satpy/etc/readers/abi_l1b.yaml'] [DEBUG: 2020-07-24 10:15:08 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['/data/gholl/cache/fogtools/abi/2020/04/13/01/C14/OR_ABI-L1b-RadF-M6C14_G16_s20201040130169_e20201040139477_c20201040139544.nc'] [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file abi.yaml [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file visir.yaml /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Reading in get_dataset C14. [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Calibrating to brightness temperatures [DEBUG: 2020-07-24 10:15:08 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/satpy/satpy/etc/' [DEBUG: 2020-07-24 10:15:08 : satpy.readers] Reading ['/home/gholl/checkouts/satpy/satpy/etc/readers/abi_l1b.yaml'] [DEBUG: 2020-07-24 10:15:08 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['/data/gholl/cache/fogtools/abi/2020/04/13/01/C14/OR_ABI-L1b-RadF-M6C14_G16_s20201040140169_e20201040149477_c20201040149556.nc'] [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file abi.yaml [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file visir.yaml /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Reading in get_dataset C14. [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Calibrating to brightness temperatures [DEBUG: 2020-07-24 10:15:08 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/satpy/satpy/etc/' [DEBUG: 2020-07-24 10:15:08 : satpy.readers] Reading ['/home/gholl/checkouts/satpy/satpy/etc/readers/abi_l1b.yaml'] [DEBUG: 2020-07-24 10:15:08 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['/data/gholl/cache/fogtools/abi/2020/04/13/01/C14/OR_ABI-L1b-RadF-M6C14_G16_s20201040150169_e20201040159477_c20201040159493.nc'] [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file abi.yaml [DEBUG: 2020-07-24 10:15:08 : satpy.composites] Looking for composites config file visir.yaml /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Reading in get_dataset C14. [DEBUG: 2020-07-24 10:15:08 : satpy.readers.abi_l1b] Calibrating to brightness temperatures [DEBUG: 2020-07-24 10:15:08 : satpy.multiscene] Forcing iteration of generator-like object of Scenes 6 0 ```Environment Info:
Additional context
Interestingly, despite
ms3.scenes
being an empty list, I can accessms3.first_scene
, but only before I accesms3.scenes
. If I change the final lines to:then the final lines of the output become:
but if I swap the final two lines around such that they are
then I get
StopIteration
:If I swap
sc2
andsc3
:then the final output line remains
i.e. in this case it's ms2 that is empty rather than ms3.
If I access
ms2.scenes
before the resampling, the result is as expected:results in