Is your feature request related to a problem? Please describe.
The proposal is to change the usage of streams in the following way.
When stream=True the returned object would be a Fieldlist (for GRIB data):
ds = from_source("url", "http://..../my_data.grib", stream=True)
for f in ds:
# f is now a Field
# at this point ds consumed the stream
Iterating in batches would be a generic option (not only stream specific):
ds1 = from_source("file", "my_local_data.grib")
ds2 = from_source("url", "http://..../my_data.grib", stream=True)
for f in ds1.batched(2):
# f is now a Fieldlist with 2 Fields
for f in ds2.batched(2):
# f is now a Fieldlist with 2 Fields
group_by would behave in a similar way.
ds1 = from_source("file", "my_local_data.grib")
ds2 = from_source("url", "http://..../my_data.grib", stream=True)
for f in ds1.group_by("level"):
# f is now a Fieldlist
for f in ds2.group_by("level"):
# f is now a Fieldlist
Please note that using group_by for non-stream data will be based on the metadata from the full dataset. However, for the stream it would be simply built by consuming GRIB messages from the stream until the values of the metadata keys specified in group_by change.
We could read the whole stream into memory with the read_all option:
ds = from_source("url", "http://..../my_data.grib", stream=True, read_all=True)
# ds is now a Fieldlist in memory, so all these work
len(ds)
r = ds.sel(param="t")
for f in ds:
# f is now a Field
for f in ds.batched(2):
# f is now a Fieldlist with 2 Fields
Is your feature request related to a problem? Please describe.
The proposal is to change the usage of streams in the following way.
When
stream=True
the returned object would be a Fieldlist (for GRIB data):Iterating in batches would be a generic option (not only stream specific):
group_by
would behave in a similar way.Please note that using
group_by
for non-stream data will be based on the metadata from the full dataset. However, for the stream it would be simply built by consuming GRIB messages from the stream until the values of the metadata keys specified ingroup_by
change.We could read the whole stream into memory with the
read_all
option: