Conserve Git LFS bandwidth

[philipp-classen]

Currently, we exceed our Git LFS limits relatively quickly. Some ideas to reduce the amount of downloaded data:

• By default, download only the most recent data (configure lfs.fetchinclude / lfs.fetchexclude, see https://github.com/git-lfs/git-lfs/issues/2717). New data per month takes around 250M, while the whole data set is currently around 6,7G (and will grow with each month).
• Compress the csv files (the expected compression ratio with xz is 20%-25%)

[philipp-classen]

After some experiments, I come to the conclusion that LFS creates more problems in our case then it solves. lfs.fetchexclude does not integrate well and looking at the state of the art of switching the Git LFS backend from Github to S3 is not promising (support is experimental at best, and the few solutions that are still active developed seem to require running your own servers).

My recommendation is to move all data files out of Git and put them on a public mirror. The data should be compressed and scripts should be provided to download it (maybe rsync based to detect existing data) and extract it locally. The current PR workflow could be preserved by updating the download script (adding a new month), so we can run tests for the PR before it has an effect.

---

For documentation, this was my example .lfsconfig:

 [lfs]
    fetchexclude = "whotracksme/data/assets/2017-*/**/*,whotracksme/data/assets/2018-*/**/*,whotracksme/data/assets/2019-*/**/*,whotracksme/data/assets/2020-*/**/*"