Example for downloading specific data - [merged]

[nabobalis]

_Merges realexample -> master

A basic example that just shows how someone can filter and download JSOC data (without the download being run) based on some primekey logic.

[nabobalis]

In GitLab by @markcheung on Sep 20, 2021, 16:53

Thanks @nabobalis

Suggest showing how to read in a sequence of images using Map. Is the sunpy MO to use lists of Maps, or do they get consolidated into an ndcube object?

[nabobalis]

I can add a step to download and create a sequence of maps.

Right now it will be basically a list of maps. NDcube support is in the future for sunpy.

[nabobalis]

In GitLab by @wtbarnes on Sep 21, 2021, 07:02

Obviously this pertains to AIA because it is querying AIA data, but why does this example belong here? There is no aiapy code being used here. Why isn't this being PR'd to drms?

Even better: could this kind of query be enabled within Fido? I think prompting people to use drms directly, unless absolutely necessary, is not ideal.

[nabobalis]

In GitLab by @markcheung on Sep 21, 2021, 07:27

The aiapy pckage serves to help users of AIA data. There’s no example in aiapy on how to download a sequence of AIA images and do something with them. Given most AIA uses don’t look at just single frames, I thought it important to include. Hence I asked Nabil for such an example.

It’s not transparent to me how VSO queries work and how to filter by criteria. Jsoc is the official data provider and many users are familiar with at least some jsoc queries, per the SDO analysis guide by DeRosa and Slater.

So we need an example of how to download from JSOC using drms queries in the aiapy gallery example.

[nabobalis]

added 1 commit

• b718448a - Example for downloading specific data

Compare with previous version

[nabobalis]

In GitLab by @markcheung on Sep 21, 2021, 11:04

Commented on examples/download_specific_data.py line 127

This is how I would naively do it too. Is there a more efficient way than a for loop?

[nabobalis]

Unless the aia calibration functions take in a list of maps, I would assume not. Let me check the API documentation.

[nabobalis]

added 1 commit

• 1c8599b8 - Example for downloading specific data

Compare with previous version

[nabobalis]

I can't see a way to make this more straightforward, maybe @wtbarnes has some ideas?

[nabobalis]

In GitLab by @wtbarnes on Sep 21, 2021, 16:48

Commented on examples/download_specific_data.py line 127

As it currently stands, register, normalize_exposure, and update_pointing just accept a single map. I've added a few minor suggestions below, but I think a for loop here is fine. It's not really an operation that can be vectorized.

[nabobalis]

In GitLab by @wtbarnes on Sep 21, 2021, 16:51

Commented on examples/download_specific_data.py line 128

You could create the maps all at once and then iterate over the Map objects rather than doing this in a loop. I doubt its any more sufficient since this is what is happening in the map factory anyway, but it is a bit cleaner.

e.g.

level_1_maps = smap.Map(img_files)
for m in level_1_maps:
    ...

[nabobalis]

In GitLab by @wtbarnes on Sep 21, 2021, 16:58

Commented on examples/download_specific_data.py line 146

I would query the pointing table first outside of the loop for the relevant time range. Otherwise you're making a call to the JSOC every single time.

from aiapy.calibrate.util import get_pointing_table
pt = get_pointing_table(level_1_maps[0].date-3*u.h, level_1_maps[-1].date+3*u.h)

You can then pass it as a kwarg to update_pointing: https://aiapy.readthedocs.io/en/stable/api/aiapy.calibrate.update_pointing.html#aiapy.calibrate.update_pointing

[nabobalis]

In GitLab by @wtbarnes on Sep 21, 2021, 17:00

Commented on examples/download_specific_data.py line 127

Is there a reason the degradation correction isn't being applied here? I'm guessing because it doesn't matter for a 211 flare observation?

[nabobalis]

I did not know that was possible, I can add it.

[nabobalis]

In GitLab by @markcheung on Sep 21, 2021, 17:02

Commented on examples/download_specific_data.py line 127

I don't think there's a reason why degradation shouldn't be applied. Worth adding.

[nabobalis]

In GitLab by @wtbarnes on Sep 21, 2021, 17:09

That is fair. I hadn't thought about the use of aiapy functions on a sequence and this is a nice illustration of that.

Fido can be used to query the JSOC directly via the JSOCClient: https://docs.sunpy.org/en/stable/api/sunpy.net.jsoc.JSOCClient.html. Is there really no way to do this exposure time filtering except through drms?

[nabobalis]

I am not aware of any way to do that via Fido. I don't think we have a system in place to take those kind of attrs and turn them into the correct query string.

[nabobalis]

changed this line in version 4 of the diff

[nabobalis]

changed this line in version 4 of the diff

[nabobalis]

added 1 commit

• e7aadb88 - fix review comments

Compare with previous version

[nabobalis]

Fixed.

[nabobalis]

Fixed.

[nabobalis]

I added the correction for degradation.

[nabobalis]

In GitLab by @wtbarnes on Sep 22, 2021, 05:12

Commented on examples/download_specific_data.py line 138

Same as the pointing update. Query the correction table outside of the loop,

ct = aiapy.calibrate.util.get_correction_table()

and then pass it to the function to avoid redundant calls to the JSOC,

map_degradation = correct_degradation(map_registered, correction_table=ct)

[nabobalis]

In GitLab by @wtbarnes on Sep 22, 2021, 05:14

Commented on examples/download_specific_data.py line 149

In this example, are we telling users that, in order for a map to be level 1.5, it has to be normalized by the exposure time and corrected for degradation?

[nabobalis]

In GitLab by @wtbarnes on Sep 22, 2021, 05:19

Commented on examples/download_specific_data.py line 97

# To do this, we will have to do a second query to the JSOC,
# this time using the query string syntax the

[nabobalis]

In GitLab by @wtbarnes on Sep 22, 2021, 05:26

Commented on examples/download_specific_data.py line 127

You should be able to avoid querying the spikes files by only asking for the image segment I think? When you call query, you specify this via the seg kwarg: https://docs.sunpy.org/projects/drms/en/stable/api/drms.client.Client.html#drms.client.Client.query. When you call export, you need to append '{image}' to the end of updated_qstr. Why this inconsistency exists, I do not know.

[nabobalis]

In GitLab by @wtbarnes on Sep 22, 2021, 05:29

Interesting. Yes, I guess we have no entry point to the '? ... ?' logic that is being used here. I wonder why not...

[nabobalis]

In GitLab by @markcheung on Sep 22, 2021, 10:22

Commented on examples/download_specific_data.py line 127

BTW for AIA files, the keywords are already in the fits files residing at JSOC. So we don't even need to do an export. We can just directly pass the urls to Map.

[nabobalis]

Fixed.

[nabobalis]

Fixed.

[nabobalis]

Fixed.

[nabobalis]

Ah that was what I was missing.

Do we want to suggest to users to bypass the export stage and use the urls directly?

[nabobalis]

changed this line in version 5 of the diff

[nabobalis]

changed this line in version 5 of the diff

[nabobalis]

changed this line in version 5 of the diff

[nabobalis]

added 1 commit

• 54376887 - more fixes

Compare with previous version

[nabobalis]

I have corrected all of this now.

[nabobalis]

In GitLab by @markcheung on Sep 22, 2021, 12:35

Caution though: I don't know if you provided urls directed to Map, whether you can still choose where to cache the files locally. So you could choose to explicitly download (without export), or let Map do that automatically. Mark

[nabobalis]

In my case, it downloaded it once but I on repeat it doesn't seem to.

I however can not find the files.

Edit: I found them and there is a "bug" here in sunpy. It downloads them but since they have the same last name, it overrides it, so it just plots the same map for each timestep.

So either I work around this or I go back to the export which gives us nice filenames?

[nabobalis]

In GitLab by @markcheung on Sep 22, 2021, 15:36

I guess go back to export for now?

[nabobalis]

In GitLab by @markcheung on Sep 22, 2021, 15:38

Actually, you can still use sunpy.util.net.download_file and not need to export, right? Would this allow you to control for the filenames?

[nabobalis]

I could do that or use a downloader directly. I forget if that util.download_file will be removed in future versions of sunpy.

[nabobalis]

In GitLab by @markcheung on Sep 22, 2021, 15:43

OK I think we like aiapy to have as few dependencies as possible. What about this one? https://docs.astropy.org/en/stable/api/astropy.utils.data.download_file.html#astropy.utils.data.download_file

[nabobalis]

We could copy that function into aiapy's util and use that directly instead.

[nabobalis]

In GitLab by @markcheung on Sep 22, 2021, 16:02

I think aiapy requires astropy anyway so it’s fine to use it from there.

[nabobalis]

The astropy util is nice and it works but it stores it with random file names in the depths of the users system. /Users/nabil/.astropy/cache/download/url/8312e38e46e0c52a5beb309ef08a1542/contents in my case.

I am ok with this and it uses the url to to workout out the path. So it would not download each time. It isn't as pretty as the export request and normal download.

[nabobalis]

added 1 commit

• 32dbe204 - update downloader so avoid filename issue in map

Compare with previous version

[nabobalis]

We could make open an issue upstream to sunpy/drms but I do wonder if the effort to implement that kind of feature would be worthwhile in the long run.

[nabobalis]

In GitLab by @wtbarnes on Sep 23, 2021, 05:48

Commented on examples/download_specific_data.py line 12

and instead we will use the `drms <https://docs.sunpy.org/projects/drms/en/stable/>`__ Python library directly.