lofar-astron / RMextract

extract TEC, vTEC, Earthmagnetic field and Rotation Measures from GPS and WMM data for radio interferometry observations
GNU General Public License v3.0
31 stars 22 forks source link

Allow for local path to IONEX data #46

Closed AlecThomson closed 4 months ago

AlecThomson commented 1 year ago

Hi @maaijke,

It'd be really useful to be able to handle local (or other exotic server) versions of IONEX data. This is often required on HPCs where internet access is locked down - especially for FTP connections! Thankfully, urllib can already handle this with e.g. server=file://path/to/the/data.

Along side this, it'd be handy to be able to pass a formatter for the file naming convention used. This is currently somewhat hard-coded in the get_urllib_IONEXfile function. My suggestion would be to have an optional formatter argument that is called to appropriately format the URL strings, with a lookup table of known formatters e.g.

KNOWN_FORMATTERS = {
    "ftp.aiub.unibe.ch": 
        lambda server, year, dayofyear, prefix, yy: 
            f"{server}/CODE/{year:4d}/{prefix.upper()}{dayofyear:03d}0.{yy:02d}I.Z",
    "cddis.gsfc.nasa.gov": 
        lambda server, year, dayofyear, prefix, yy: 
            f"{server}/gnss/products/ionex/{year:4d}/{dayofyear:03d}/{prefix}{dayofyear:03d}0.{yy:02d}i.Z",
    "igsiono.uwm.edu.pl":
        lambda server, year, dayofyear, prefix, yy: 
            f"{server}/data/ilt/{year:4d}/igrg{dayofyear:03d}0.{yy:02d}i",
}

@tjgalvin and myself have been playing with implementing these changes and along the way we also found that, due to the speed of a local cache, some race conditions can occur. We've put in some checks to catch these.

Would you be open to a PR that brings in these changes? All being well, the current behaviour should be completely preserved.

AlecThomson commented 5 months ago

Hi @maaijke - would you be interest this at all?

maaijke commented 5 months ago

Hi Alec, I think something went wrong with my earlier reply, sorr for that. YEs I am interested, especially in the formatter! Local IONEX files is already supported, but if you have a better way of doing that I am happy to have a look at that

AlecThomson commented 5 months ago

Ok cool - I'd be curious to see what is currently supported! I'll open a draft PR so you can inspect the modifications I've been using locally. I'm keen to see what you think :)

AlecThomson commented 4 months ago

Hey @maaijke - thanks for the merge! Just wondering, will you push a new release of RMextract along with this?