stan-dev / posteriordb

Database with posteriors of interest for Bayesian inference
161 stars 26 forks source link

add github database for Python #215

Closed ahartikainen closed 3 years ago

ahartikainen commented 3 years ago

This PR adds PosteriorDatabaseGithub class that downloads files from GitHub when needed. The class is similar with the PosteriorDatabase, but has .refresh_github method to manually update the filelist.

Also the init takes repo and ref info. Maybe these could also be class variables (let's see).

The default database location is POSTERIOR_DB_DIR and if it is not defined, it uses $HOME/.posteriordb/posterior_database.

It uses GITHUB_PAT, but if it is not defined, it is limited to 60 calls/h. Currently, I have not optimized the number of API calls, but I think users can at least get a few posteriors without the token. There is probably room for improvement here.

MansMeg commented 3 years ago

This is excellent! Could you add an example of this functionality in the Python README?

ahartikainen commented 3 years ago

Sure. I also need to add tests.

ahartikainen commented 3 years ago

There is some error with branch name and R code

https://api.github.com/repos/MansMeg/posteriordb/contents/posterior_database?ref=gh_db

->

https://api.github.com/repos/MansMeg/posteriordb/contents/posterior_database?ref=feature/gh_db

ahartikainen commented 3 years ago

There should possibility to "overwrite" the current files with github versions. It should check the sha value and determine if it needs to overwrite the file or not (depending on overwrite flags/keyword).

ahartikainen commented 3 years ago

Ok, we still need to check that sha1 value calculated also works for files over 1MB.

ahartikainen commented 3 years ago

Ok, I think this starts to be ready for review.

I added a simple progress bar when downloading all models. There are also now overwrite handles that do what they should

MansMeg commented 3 years ago

Great! Im done then for now. Can I merge now?

ahartikainen commented 3 years ago

Yes