Add a new FIDO client that downloads the AIA synoptic Dataset

[GillySpace27]

Describe the feature

Downloading and working with large sets of AIA 4k x 4k images can be time-consuming and expensive. I learned of the existence of another set of AIA images that are hosted on the jsoc website but they aren't able to be seen by FIDO, which feels like a choice someone made. Nevertheless, they are here: https://jsoc1.stanford.edu/data/aia/synoptic/

Their description:

“This directory contains FITS files at a 2-minute cadence which have been produced by processing 4Kx4K AIA level 1 images to level 1.5, then reducing them to 1024x1024 2.4 arc-sec pixels by summing each non NaN valued pixel in the 4x4 pixel cell and dividing by the number of non NaN valued pixels. The pixel value units are DN per original 0.6 arc-sec pixel. The files are stored in a year, month, day, hour, and filename directory hierarchy. These images are produced approximately 7 days after T_OBS since they are derived from definitive level 1 data. The image_times and image_times.json provide the time for the most recent (current final) synoptic file.”

The ability to query and download from this dataset is self-evident, as it does basically all of the prep work to look at the images you need, along with being both spatially binned and time integrated. Just a fantastic dataset (unless you're looking at the most fine-scale or dynamic features that are detectable by the instrument.

Proposed solution

Anyways, I've already written a couple of other clients for FIDO now, so I went ahead and implemented a version of a Client that seems to work. I'm sure we'll have to go back and forth but I'm going to set up a pull request draft to get the process going. I also wouldn't be surprised if there are other approaches than my implementation, which might be more consistent with the other Clients.

[wtbarnes]

If this data is available via the JSOC, it should be accessible through drms and then by extension the JSOCClient that already is supported in Fido. Is there some reason that the existing client is not adequate for querying this dataset?

[GillySpace27]

Well, I guess I see the product on the status page: http://jsoc.stanford.edu/data/jsoc_proc_status.htm but I can't use that as the series in an attrs.jsoc.Series call, so I'm not sure what else to do. I can't find it mentioned anywhere else either. Please let me know what you think I should do.

sunpy.net.jsoc.attrs.Series

The JSOC Series to Download.

          Attribute Name           Client             Full Name                                                Description                                   
---------------------------------- ------ ---------------------------------- --------------------------------------------------------------------------------
aia_flatfield                      JSOC   aia.flatfield                      AIA flatfield                                                                   
aia_lev1                           JSOC   aia.lev1                           AIA Level 1                                                                     
aia_lev1_euv_12s                   JSOC   aia.lev1_euv_12s                   AIA Level 1, 12 second cadence                                                  
aia_lev1_uv_24s                    JSOC   aia.lev1_uv_24s                    AIA Level 1, 24 second cadence                                                  
aia_lev1_vis_1h                    JSOC   aia.lev1_vis_1h                    AIA Level 1, 3600 second cadence                                                
aia_master_pointing3h              JSOC   aia.master_pointing3h              Master Pointing Parameters                                                      
aia_response                       JSOC   aia.response                       AIA instrument response table                                                   
aia_temperature_summary_300s       JSOC   aia.temperature_summary_300s       Temperature Statistics from AIA Housekeeping - Thermal Packet                   
hmi_b_135s                         JSOC   hmi.b_135s                         Full-disk Milne-Eddington inversion with the azimuth disambiguation informati...
hmi_b_5760s                        JSOC   hmi.b_5760s                        96-min full-disk Milne-Eddington inversion with the azimuth disambiguation in..

[wtbarnes]

Hmm. Doing a naive search on the Lookdata page doesn't reveal anything either. The directory structure also doesn't follow the usual SUMS pattern so maybe these are not ingested into drms. @nabobalis do you know if these data are available as a series?

[nabobalis]

Let me ask the JSOC about that dataset.

[GillySpace27]

Cool, and if you decide you want a web-scraping client for the data, I’ve got one running in my for that mostly works, too.

Glad to help however I can, though. 😊

Gilly

From: Nabil Freij @.> Date: Monday, October 14, 2024 at 10:22 AM To: sunpy/sunpy @.> Cc: Gilbert, Gilly @.>, Author @.> Subject: Re: [sunpy/sunpy] Add a new FIDO client that downloads the AIA synoptic Dataset (Issue #7828) [EXTERNAL EMAIL]

Let me ask the JSOC about that dataset.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/sunpy/sunpy/issues/7828*issuecomment-2411714383__;Iw!!D-JDmu3Lc2wo0Jiybg!dWz8dKITblanf-u7qInAdfwOSHFXBhb4_4W9pzPFjLgSpBRYSd9JxxvQbVMi_44Kfv9oiAD7IBrQixB1UzhB$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AC3B6UKDLXX3ILM6I3LEQCLZ3PVTDAVCNFSM6AAAAABP3BLJV2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJRG4YTIMZYGM__;!!D-JDmu3Lc2wo0Jiybg!dWz8dKITblanf-u7qInAdfwOSHFXBhb4_4W9pzPFjLgSpBRYSd9JxxvQbVMi_44Kfv9oiAD7IBrQi0WWqsfm$. You are receiving this because you authored the thread.Message ID: @.***>

[nabobalis]

I asked and I was told that the original expectation was that people would access this via the web.

So I think a client would be ok.

[GillySpace27]

Yay! I’ll make a Pull Request about it today. 😊

From: Nabil Freij @.> Date: Monday, October 14, 2024 at 11:44 AM To: sunpy/sunpy @.> Cc: Gilbert, Gilly @.>, Author @.> Subject: Re: [sunpy/sunpy] Add a new FIDO client that downloads the AIA synoptic Dataset (Issue #7828) [EXTERNAL EMAIL]

I asked and I was told that the original expectation was that people would access this via the web.

So I think a client would be ok.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/sunpy/sunpy/issues/7828*issuecomment-2411879544__;Iw!!D-JDmu3Lc2wo0Jiybg!ZT4Atip9EYixvm8gLHs7F1U_vsHiX4LMIHtZ3ZHneWkSHEZ73T6uqaw0OIJUmdZtE2sE4UQoHWdwhIDDraKO$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AC3B6ULDSUI7YMKL4AA36PDZ3P7ITAVCNFSM6AAAAABP3BLJV2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJRHA3TSNJUGQ__;!!D-JDmu3Lc2wo0Jiybg!ZT4Atip9EYixvm8gLHs7F1U_vsHiX4LMIHtZ3ZHneWkSHEZ73T6uqaw0OIJUmdZtE2sE4UQoHWdwhKmNo5MX$. You are receiving this because you authored the thread.Message ID: @.***>

[GillySpace27]

I'm going to open a PR about this new client but I found something that was breaking my code that is deeper than I should probably touch on my own.

In sunpy.net.attr.py line 471 or so, there is a comparison that looks like this: return self.min <= other <= self.max

which throws exceptions unless I change it to this: return self.min.value <= int(other) <= self.max.value

I'm almost completely certain this is the wrong way to handle this but as a hotfix it does unbreak the code.

Anyway, I'm not sure if I should raise this as a separate issue (probably?), or if theres a way to link to that spot in the code from here, so please let me know what you'd like me to do about it. Thanks!

[nabobalis]

Without more context, it is hard for me to know if that is a good thing to change. Can you open the PR with the hotfix and we can go from there?