Feature/refactor streaming - Githubissues

ecmwf / earthkit-data

A format-agnostic Python interface for geospatial data

Apache License 2.0

53 stars 12 forks source link

Feature/refactor streaming #371

Closed sandorkertesz closed 4 months ago

sandorkertesz commented 5 months ago

Fixes #364

This PR:

addsbatched() and group_by() for stream and fieldlist like objects
adds theread_all option for from_source() when stream=True
removes the batch_size and group_by options from from_source (This is a breaking change!)

Examples

ds = from_source("url", "http://..../my_data.grib", stream=True)

for f in ds:
     # f is a field

# at this point ds consumed the stream

ds1 = from_source("file", "my_local_data.grib")
ds2 = from_source("url", "http://..../my_data.grib", stream=True)

for f in ds1.batched(2):
     # f is now a Fieldlist with 2 Fields

for f in ds2.batched(2):
     # f is now a Fieldlist with 2 Fields

ds1 = from_source("file", "my_local_data.grib")
ds2 = from_source("url", "http://..../my_data.grib", stream=True)

for f in ds1.group_by("level"):
     # f is a Fieldlist

for f in ds2.group_by("level"):
     # f is a Fieldlist

ds = from_source("url", "http://..../my_data.grib", stream=True, read_all=True)

# ds is a Fieldlist in memory, so all these work
len(ds)
r = ds.sel(param="t")

for f in ds:
     # f isa Field

for f in ds.batched(2):
     # f is a Fieldlist with 2 Fields

Documentation

Notebooks

https://earthkit-data--371.org.readthedocs.build/en/371/examples/data_from_stream.html https://earthkit-data--371.org.readthedocs.build/en/371/examples/url_stream.html https://earthkit-data--371.org.readthedocs.build/en/371/examples/fdb.html

codecov-commenter commented 5 months ago

Welcome to Codecov :tada:

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered :open_umbrella: