nkphysics / autoNICER

A program that automates the data retrieval and data reduction of observational data gathered by NASA's NICER x-ray mission
Apache License 2.0
5 stars 0 forks source link

Investigate possibilities to migrate away from wget #60

Open nkphysics opened 3 months ago

nkphysics commented 3 months ago

I know and generally like to follow the motto "If it ain't broke don't fix it", but I've been continually reminded of the benefits of minimizing dependencies. The heasoft dependency clearly cannot be dropped for obvious reasons, but autoNICER still relies on wget (not the wget python lib) to retrieve datasets from the heasarc.

See the first part of the pull reduce function:

def pull_reduce(self):
        """
        Downloads the NICER data
        Puts the retrieved data through a standardized data reduction scheme
        """
        downCommand = (
            "wget -q -nH --no-check-certificate --cut-dirs=5 -r -l0 -c -N -np -R "
            + "'index*'"
            + " -erobots=off --retr-symlinks https://heasarc.gsfc.nasa.gov/FTP/nicer/data/obs/"
        )

This potentially could be improved upon with just using a conventional get request with the requests lib, it also could potentially lead to converting all of this to be async, which could certainly improve the code.