Place the 3d reddening map(s) somewhere accessible

[willclarkson]

EDIT 2021-04-09 Here are some additional maps, depending on which reddening combination you prefer. The FITS header information provides values for most of the keywords that were used to construct these maps.

• Defaults (L+19 scaled to mwdust, using Planck for gaps, sampled out to 17.5 kpc - this is probably the one you want): http://www-personal.umd.umich.edu/~wiclarks/rubin/merged_ebv3d_nside64_defaults.fits.gz

• Ignoring Lallement et al. everywhere except where mwdust predicts 0 at large distances, in which case Planck is used (i.e. using dmaxL19=-1): http://www-personal.umd.umich.edu/~wiclarks/rubin/merged_ebv3d_nside64_noL19.fits.gz

• Treating both L+19 and mwdust as "correct", and smoothly transitioning from L19 to mwdust over a 500pc distance from the maximum distance for which L19 is defined (i.e. using bridgeL19 = True and bridgeWidthL19=500): http://www-personal.umd.umich.edu/~wiclarks/rubin/merged_ebv3d_nside64_bridge.fits.gz

• Defaults, sampled at NSIDE=128 (note this already seems to be at greater spatial resolution than L+19 but still requires averaging over Bovy et al.).: http://www-personal.umd.umich.edu/~wiclarks/rubin/merged_ebv3d_nside128_defaults.fits.gz

Here is the previous default, similar to the default above but sampled out to 15 kpc and with a slightly lower pixFillFac used: http://www-personal.umd.umich.edu/~wiclarks/rubin/merged_ebv3d_nside64.fits.gz

These maps are generated by calls to compareExtinctions.loopSightlines() to sample at each healpix, then the resulting fits files are merged into a single skymap with compareExtinctions.mergeMaps(). The maps can then be read and queried using methods found in readExtinction.py.

[Thalos12]

I was wondering whether we should use git-lfs to handle the map files. The limit for a single file is 2GB and the total size of the repository should be lower than 10Gb, but the maps are small and we are far from both limits. We might also write a script that the user has to run after cloning the repository that downloads the maps with git-lfs, extracts them and removes compressed files.

[willclarkson]

Hi @Thalos12 - I think that's a great idea. I am not very familiar with git-lfs, do you know how to set this up?

[Thalos12]

I am reading about it and it appears to be simple to use. I will try it first on my repository and if it works I will apply it here.

[Thalos12]

Hi @willclarkson! It appears that I cannot push git-LFS objects to a fork of a public repository unless the original repository already is using git LFS to track large files. I believe that now I could either push directly to a branch on your repository, but I think I have to be added as a contributor to do that, or write below the steps required to setup git-LFS. Let me know how you want to proceed!

[willclarkson]

Hi @Thalos12 - I have invited you as a collaborator, it should work now.

[Thalos12]

Hi @willclarkson, I have pushed a new branch (gitlfs) with the git-LFS tracked files in a subfolder called extmaps. I tried to clone it and everything works after git checkout gitlfs.

There are two thing I wish to discuss with you.

1. All the files are downloaded immediately when cloning, but there are ways to prevent this and allow an user to download only what he needs. This requires the user to configure git filters, and while it is not difficult, we have to write some instructions for that. Moreover, since the maps take around 4GB, this might not be a problem at all, with the exception of very slow internet connections.
2. Now that the maps are in a subfolder, we might want to change the default paths in the code. I already did it locally, but I wanted to ask for your opinion before pushing anything. The alternative would be to either have the user copy the files in the correct location (the python folder) or leave symlinks in the python folder pointing to the files in extmaps. Let me know if there is anything you do not agree with or want to change!

[Thalos12]

I pushed some changes to the gitlfs branch, also with the updates to the default maps. In compareExtinction, the pathJoined argument of mergeMaps is now checked and an exception is raised if it already exists, to prevent overwriting the maps. Also, all function arguments with as default value the path to a default map are now set to None and the path to the default map is built later only if the value of the function argument is None. Now there is also a README explaining how to download the repository and the maps.