Fido Scraper Client - Githubissues

Helioviewer-Project / python-api

Python wrapper around the Helioviewer API.

https://hvpy.readthedocs.io/en/latest/

BSD 2-Clause "Simplified" License

6 stars 5 forks source link

Fido Scraper Client #75

Open nabobalis opened 1 year ago

nabobalis commented 1 year ago

We can create a Fido (https://docs.sunpy.org/en/stable/generated/api/sunpy.net.Fido.html?highlight=fido) client that would let people search and fetch the full images that are stored in JPEG2000 or PNG formats.

Implementing the Client

The client would use the URL scraper to find files on the remote server. The three components that are needed to make the client are:

Attribute Base URL: https://api.helioviewer.org/jp2/
Attribute Pattern: This is a template that is used to extract the metadata from urls matched by base url.

For example, https://api.helioviewer.org/jp2/AIA/335/2012/01/29/ this URL can be used to make a request, breaking it down we have this template:

Untitled

Define the method register_value() which registers attributes supported by the client. The attrs returned by this method are used to match the client search attrs.

exitflynn commented 1 year ago

hey @nabobalis! do we still want this? I would love to contribute!

nabobalis commented 1 year ago

I think so but I want @dgarciabriseno to chime in here.

If the plan is to create an API that does this, then in theory we won't need this client as it stands. Instead of a Fido scrapper client, it would become an Fido API client.

dgarciabriseno commented 1 year ago

@nabobalis @exitflynn This topic just came up. I do plan to implement an API for querying and downloading images.

nabobalis commented 1 year ago

@nabobalis @exitflynn This topic just came up. I do plan to implement an API for querying and downloading images.

In that case, I think this issue should hold until that API exists, or would you want this client to be created before that?

dgarciabriseno commented 1 year ago

We can make the FIDO scraper client, I expect it would be easier to implement since the base classes for implementing it are already in place.

I am working on a search API. It'll offer a bit more than just pointing to images to download. We have some use cases for being able to search images by more than just date and time and also for returning some FITS header data instead of just pointing directly to the images.

So I'm thinking we can have both since they will serve different use cases.