to find the darkest regions in the night sky
git clone https://github.com/void4/lowestflux.git
python -m pip install -r requirements.txt
In this case I used Gaia Data Release 1 (DR1), because even though DR2 and (E)DR3 are more complete, they require more than a terabyte of data, too much for me to download, store and process.
The DR1 files are around 200 Gigabytes, so may take a while
Download VisualWget:
https://sites.google.com/site/visualwget/a-download-manager-gui-based-on-wget-for-windows
and set this as the source URL:
http://cdn.gea.esac.esa.int/Gaia/gdr1/gaia_source/csv/
then enable the Advanced->Recursive Retrieval->--recursive and Advanced->Recursive Accept/Reject->--no-parent flag
as visually documented here: https://stackoverflow.com/questions/23446635/how-to-download-http-directory-with-all-files-and-sub-directories-as-they-appear/24247715#24247715
should be (not tested)
wget --recursive --no-parent --no-host-directories http://cdn.gea.esac.esa.int/Gaia/gdr1/gaia_source/csv/
Then copy the downloaded GaiaSource_000-xxx-yyy.csv.gz files into this repository folder, so they are at the same level as the main.py file.
python main.py
This first generates an index ("sourcemeta.json") of the minimum and maximum right ascension and declination values, so that when a region (given by min/max ra/dec itself) is queried, it only loads a subset of the files (going through all files every time, filtering for stars of that region would take too long).
Given that the DR1 data is not expected to change, and so you do not have to generate it yourself, I have included my generated sourcemeta.json in this repository. If you do want to recreate it yourself, set recreate = True on the first run only.
Once it has loaded or generated/extended that file, it opens the files (possibly) containing stars in that queried region, and filters them down to the ones that are definitely contained within it, generating: