Finals2000A download fails, server unavailable

[FinduschkaLi]

Since two days I get the error: cannot download ftp://ftp.iers.org/products/eop/rapid/standard/finals2000A.all because <urlopen error ftp error: error_perm('530 connect failed: Operation timed out. No response from server.')>

When checking on the website, there seems indeed to be some server error, because the website also does not show the required information (of course accessed through regular browser)

Does anyone have this issue aswell and is there a way to provide a backup download source for the finals2000A?

Running latest version of skyfield on python 3.8 on a ubuntu server.

[aelanman]

I'm having the same issue. astropy's url for the EOP data still works, so you can get it here (look at astropy.utils.iers.IERS_A_URL) ftps://anonymous:mail%40astropy.org@gdc.cddis.eosdis.nasa.gov/pub/products/iers/finals2000A.all

skyfield should probably consider adding a backup url.

[FinduschkaLi]

Here is the message sent by IERS:

Due to urgent maintenance work, the servers datacenter.iers.org, data.iers.org and ftp.iers.org are currently unavailable. We will inform you as soon as the maintenance work has been carried out.

Thank you for that idea. Here is what I implemented as a fallback for my own code in case the original loader throws an exception.

def downloadNASA():
    print("Downloading from NASA...", end ="")
    ftps = FTP_TLS('gdc.cddis.eosdis.nasa.gov')
    ftps.login('anonymous','mail@astropy.org')
    ftps.prot_p()
    ftps.cwd("/pub/products/iers/")
    #ftps.retrlines('LIST')
    filename = 'finals2000A.all'
    homepath = './'
    local_filename = os.path.join(homepath, filename)
    lf = open(local_filename, "wb")
    ftps.retrbinary('RETR %s' % filename, lf.write)
    ftps.close()
    print("done.")

Probably this ticket should be moved to a feature request.

[brandon-rhodes]

Thanks for the information you have both provided! Hopefully other Skyfield users who are affected can use the URLs and example code you've shared. I'm not sure I want to start adding backup URLs to Skyfield yet, since it could lead to two runs of the program leading to different results without the user realizing why, if Skyfield was automatically switching URLs based on availability (and thus perhaps getting different or old versions of a file).

[aendie]

I spoke with someone at IERS in Frankfurt who assured me that the ftp.iers.org server should be the primary choice for downloading EOP data. He also reminded me that although the USNO server (maia.usno.navy.mil) is up and running, the last time it was down it took 2.5 years to bring it back online. So I agree that Skyfield should stick only to the official download URL.

However, to cater for the geeky types that, like me, prefer to have another option that "always" works, I implemented a solution (in SFalmanac and Skyalmanac) that first checks if the IERS server is working (because Skyfield 1.42 just fails like ... end of story):

I first check if we have an Internet connection at all (by checking if either of the two servers is reachable) ... because if you are stuck on a yacht in the middle of an ocean it would be far more prudent to continue using the current EOP data than fussing about it being over 30 days old, of course.

I borrowed heavily on the Skyfield iokit.py code to implement an alternative connection to the USNO server, which uses https (not ftp) ... and it works nevertheless. Sorry, no certificate checks. So here is my code snippet in case it helps anyone. Initially init_sf(spad) is called with a folder (in "spad") in which to store the downloaded file. config.useIERS is True if the user wants to use the EOP data (as opposed to the bulit-in UT1-tables). config.ageIERS is the maximum age in days (as text) tolerated. I use this on Windows 10 and Ubuntu 20.04 LTS - I have no access to MacOS. Good luck!

import os
import errno
import socket
import sys          # required for .stdout.write()
from urllib.request import urlopen
from skyfield import VERSION
from skyfield.api import Loader

def isConnected():
    try:
        # connect to the host -- tells us if the host is actually reachable
        sock = socket.create_connection(("www.iers.org", 80))
        if sock is not None: sock.close
        return True
    except OSError:
        pass
    # try alternate source if above server is down ...
    try:
        # connect to the host -- tells us if the host is actually reachable
        sock = socket.create_connection(("maia.usno.navy.mil", 80))
        if sock is not None: sock.close
        return True
    except OSError:
        pass
    return False    # if neither is reachable

# NOTE: the IERS server is unavailable (due to maintenance work in the first 3 weeks, at least, of April 2022)
#       however, although the USNO server currently works, it was previously down for 2.5 years!
#       So it is still best to try using the IERS server as first oprion, and USNO as second.

def testIERSserver(filename):
    url = "ftp://ftp.iers.org/products/eop/rapid/standard/" + filename
    try:
        connection2 = urlopen(url)
    except Exception as e:
        e2 = IOError('cannot download {0} because {1}'.format(url, e))
        e2.__cause__ = None
#        raise e2
        return False
    return True     # server works

def downloadUSNO(path, filename):
    # NOTE: the following 'print' statement does not print immediately in Linux!
    #print("Downloading EOP data from USNO...", end ="")
    sys.stdout.write("Downloading EOP data from USNO...")
    sys.stdout.flush()
    filepath = os.path.join(path, filename)
    url = "https://maia.usno.navy.mil/ser7/" + filename
    connection = urlopen(url)
    blocksize = 128*1024

    # Claim our own unique download filename.

    tempbase = tempname = path + filename + '.download'
    flags = getattr(os, 'O_BINARY', 0) | os.O_CREAT | os.O_EXCL | os.O_RDWR
    i = 1
    while True:
        try:
            fd = os.open(tempname, flags, 0o666)
        except OSError as e:  # "FileExistsError" is not supported by Python 2
            if e.errno != errno.EEXIST:
                raise
            i += 1
            tempname = '{0}{1}'.format(tempbase, i)
        else:
            break

    # Download to the temporary filename.

    with os.fdopen(fd, 'wb') as w:
        try:
            length = 0
            while True:
                data = connection.read(blocksize)
                if not data:
                    break
                w.write(data)
                length += len(data)
            w.flush()
        except Exception as e:
            raise IOError('error getting {0} - {1}'.format(url, e))

    # Rename the temporary file to the destination name.

    if os.path.exists(filepath):
        os.remove(filepath)
    try:
        os.rename(tempname, filepath)
    except Exception as e:
        raise IOError('error renaming {0} to {1} - {2}'.format(tempname, filepath, e))

    #print("done.")
    sys.stdout.write("done.\n")
    sys.stdout.flush()

def init_sf(spad):
    load = Loader(spad)         # spad = folder to store the downloaded files
    EOPdf  = "finals2000A.all"  # Earth Orientation Parameters data file
    dfIERS = spad + EOPdf

    if config.useIERS:
        if compareVersion(VERSION, "1.31") >= 0:
            if os.path.isfile(dfIERS):
                if load.days_old(EOPdf) > float(config.ageIERS):
                    if isConnected():
                        if testIERSserver(EOPdf): load.download(EOPdf)
                        else: downloadUSNO(spad,EOPdf)
                    else: print("NOTE: no Internet connection... using existing '{}'".format(EOPdf))
                ts = load.timescale(builtin=False)  # timescale object
            else:
                if isConnected():
                    if testIERSserver(EOPdf): load.download(EOPdf)
                    else: downloadUSNO(spad,EOPdf)
                    ts = load.timescale(builtin=False)  # timescale object
                else:
                    print("NOTE: no Internet connection... using built-in UT1-tables")
                    ts = load.timescale()   # timescale object with built-in UT1-tables
        else:
            ts = load.timescale()   # timescale object with built-in UT1-tables
    else:
        ts = load.timescale()   # timescale object with built-in UT1-tables

    and so on - - - -

EDIT: A minor update to downloadUSNO() is required. Using the print("Downloading EOP data from USNO...", end ="") statement works perfectly in Windows however in Linux it does not print Downloading EOP data from USNO... when download starts ... it only prints Downloading EOP data from USNO...done when download has finished. The bugfix is simple: using

sys.stdout.write("Downloading EOP data from USNO...")
sys.stdout.flush()

works perfectly in Windows and in Linux. The above code has been updated.