Auto install files

[gusbeane]

It would be nice if the necessary tables were downloaded and stored automatically instead of having to gather them by hand

[joshspeagle]

Agreed -- it would be nice to automatically install these files in a particular directory (say, data/DATAFILES).

Some of the inputs can be a bit large, so the easiest option would probably be to make some simple utility functions to download the desired models. I'll think about how best to do this and push something in the next ~week or so, along with archiving the current set of models and data tables.

[gusbeane]

Only possible issue with placing the files in data/DATAFILES is integration with pip

[joshspeagle]

Well, I'm actually hesitant to have pip auto-install a lot of these, since the files can be a bit large. That's why I was thinking it'd be better just to have some simple convenience functions that users can call to do so, which would be outlined prominently in the documentation.

[gusbeane]

I just mean that if brutus is pip installed there is no "default" directory into which to make the data/DATAFILES structure, not for pip to install the files into the system libraries. (Or, in other words, brutus.__path__ is definitely not the place to autoinstall the files) E.g., one option would be to place the files in ${HOME}/.brutus, though I'm not sure this is the best solution since it hides large files in the user's filesystem

The user can also specify which directory, but then that directory has to be managed/provided at every runtime, which is not elegant but probably the best solution.

[joshspeagle]

Yea, this is always a pain. I'll keep thinking about whether there's a good workaround...

[catherinezucker]

The dustmaps package by Greg has a good interface for this I think, where you can set the download directory etc and fetch individual data products via a python interface: https://dustmaps.readthedocs.io/en/latest/. Might be useful to possibly port some code from there. We can also put these files on the Dataverse if we want to replicate some of these functions.

On Sun, May 24, 2020 at 5:44 PM Josh Speagle notifications@github.com wrote:

Yea, this is always a pain. I'll keep thinking about whether there's a good workaround...

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/joshspeagle/brutus/issues/44#issuecomment-633305320, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEL5OSJFYU3WHYOQUFVPPVDRTGIKZANCNFSM4NI6VK5Q .

[joshspeagle]

Yes, I had hoped to emulate some of this functionality. I'm currently creating a Harvard Dataverse with all the public files just to enable this.

[joshspeagle]

Obviously this isn't as fancy as Greg's dustmap functions, but I think this should be good enough for now. I also did check that this will output the wget verbosity in the terminal, even if it doesn't pipe through to the notebook.