Feature request: add HTTPS support for HEASARC archive access

USRA-STI / gdt-fermi

Gamma-ray Data Tools - Fermi mission components

Apache License 2.0

2 stars 4 forks source link

Feature request: add HTTPS support for HEASARC archive access #31

Closed joshuarwood closed 1 month ago

joshuarwood commented 6 months ago

FTP access to HEASARC is currently unreliable due to frequent outages at the HEASARC FTP servers. HTTPS access currently appears more reliable. We should add a method to access archival HEASARC data through the HTTPS servers. I can start working on this in May 2024.

derekocallaghan commented 6 months ago

Hi @joshuarwood, the gdt-core PR https://github.com/USRA-STI/gdt-core/pull/16 contains proposed changes for accessing FITS files via the fsspec capability provided by astropy. I've been using this locally for accessing Fermi files via HTTPS without prior downloading. The cache capability also means that they're only downloaded once to a local cache location. I thought this might be useful for your changes.

joshuarwood commented 5 months ago

I see the usefulness of your PR since getting HTTPS files without downloading is a neat feature. However, the goal of my request is much broader than downloading an HTTPS file from a known url. Essentially I'd like an interface similar to TriggerFtp/ContinousFtp where a user can say "I want X files from trigger Y" and it will go find the URLs and collect the files from the HTTPS interface to HEASARC. This way users won't have to explicitly lookup urls. Currently we use the FTP servers for this since they have nice ways for describing which files are in a given directory but the downside is the server outages lasting for several hours each week.

BillCleveland-USRA commented 1 month ago

Closed with merge #38