search

bird-house / bird-house.github.io

Project pages for Birdhouse http://bird-house.github.io
Apache License 2.0
4 stars 1 forks source link

MAGIC spec #18

Closed huard closed 6 years ago

huard commented 6 years ago

Send MAGIC spec to David and Ben. Including IRIS/Dask support. Provide contact to Bill Little, Met Office.

cehbrecht commented 6 years ago

Here are infos I got from Bill (@bjlittle) about Iris and Dask:

Yes, it’s true that since Iris 2.0 we now utilise the benefit of Dask for deferred loading, out-of-core lazy evaluation and distributed processing.

I’d recommend going through the Dask documentation https://dask.pydata.org/en/latest/, other than that we did put together a notebook as an introduction to dask.

We also added a section to the iris user guide on real and lazy data.

Is this the kind of information that you’re looking for?

cehbrecht commented 6 years ago

@bekozi, @huard: There is also a SciPy conference in July in Texas where Bill will attend: https://scipy2018.scipy.org/

huard commented 6 years ago

Thanks, I've started looking it Iris and will continue to investigate. There is a lot to like.

tomLandry commented 6 years ago

I have thrown the link to my colleagues. Just tell if investigation is required, I can start separate work on HPC technologies and libraries.

huard commented 6 years ago

I think the killer feature Iris provides is an abstraction for ensembles ("data cube"). It has capabilities to go through a list of files and cluster them in the smallest number of logical "cubes". I suggest we consider this to improve our search capability. At the moment search returns a list of individual files. I think we should start thinking about a different search interface that returns "cubes" instead of files.

What I mean is that currently we return a list of items, and then it is the user job to aggregate them. It might be easier to let the platform aggregate, and then let the user filter out the unwanted content. Something to think about.

bekozi commented 6 years ago

I think the killer feature Iris provides is an abstraction for ensembles ("data cube"). It has capabilities to go through a list of files and cluster them in the smallest number of logical "cubes".

That is nice!

huard commented 6 years ago

Are there any action item related to this ?

bekozi commented 6 years ago

I'm still not sure what this spec is yet. Is it a document or reference implementation? It sounds like it may just be Iris or a part of it. Did I miss a link?

huard commented 6 years ago

Don't worry, I'm confused too, and I opened the ticket ! @cehbrecht, was the idea to provide Ben with an example of dask integration or you had something else in mind?

cehbrecht commented 6 years ago

Well, I just wanted to make the introduction to Bill who is working on IRIS, and version 2.0 of IRIS is using dask. The rest is up to you :)

The upcoming esmvaltool 2.0 version is also using IRIS: https://esmvaltool.readthedocs.io/en/version2_development/

bekozi commented 6 years ago

Well, I just wanted to make the introduction to Bill who is working on IRIS, and version 2.0 of IRIS is using dask.

Got it. Hi, @bjlittle! Thanks for the reference materials.

The rest is up to you :)

That doesn't sounds like magic, sounds like work. :smile:

bjlittle commented 6 years ago

Hi Guys, will any of you be at SciPy 2018, Austin, Texas this month? I'll be attending, so if you want to meet-up I'm happy to do so and talk Iris, if that helps :-)

tomLandry commented 6 years ago

No, my conference travels will bring me at IGARSS at the end of July, then FOSS4G at the end of August.

I'll do one more call to colleague for SciPy. My organisation, CRIM, would have enough show at the conference for sure.

Tom

De : Bill Little notifications@github.com Envoyé : 3 juillet 2018 05:55:36 À : bird-house/bird-house.github.io Cc : Tom Landry; Comment Objet : Re: [bird-house/bird-house.github.io] MAGIC spec (#18)

Hi Guys, will any of you be at SciPy 2018, Austin, Texas this month? I'll be attending, so if you want to meet-up I'm happy to do so and talk Iris, if that helps :-)

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/bird-house/bird-house.github.io/issues/18#issuecomment-402086000, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMw9edSbvp9P92MB3s6amgkt4ZhXAWEHks5uCz-YgaJpZM4UQQCf.