Dealing with large data files

[sashahafner]

This came up in a discussion with @Stoeckler. GitHub gives some warnings for big files (50 MB?) and at some size prevents pushing (100 MB?). And then it takes some effort to purge files from the commit history (see the guidance doc on this). Some discussions or articles online suggest that data should be excluded via .gitignore in general (e.g., https://drivendata.github.io/cookiecutter-data-science/#data-is-immutable). That makes sense if they are public data, but where we created and possibly edit data, it does not.

Here are some possible imperfect solutions that can be used in combination:

• Keep data in a subdirectory outside of your repo
• Back up or share data with some other tool or service
• Refer to data files outside of a repo using relative paths in any scripts
• Include data in repo but have Git ignore it by specifying directory in .gitignore

I had thought it would be problematic to let Git/GitHub Desktop put repos within a OneDrive folder, because of possible sycing issues. But I see some discussion about doing just this online: https://stackoverflow.com/questions/25735148/git-repository-in-onedrive-synced-folder. So that could be one decent and simple solution:

1. Use something like OneDrive/GitHub-repos directory for all repos
2. Include all data files/directories in these repos
3. But exclude them from git in the .gitignore file

@Stoeckler that might be worth trying.

[sashahafner]

Let's create some new guidance documents related to this:

• on .gitignore
• using OneDrive w Git

[sashahafner]

Now there is this: https://github.com/AU-BCE-EE/GitHub-guidance/blob/main/ignoring-files.md

[sashahafner]

And this

https://github.com/AU-BCE-EE/GitHub-guidance/blob/main/big-files.md

But I need some help with the text in these. @Stoeckler once you sort out OneDrive (if you use it), could you add a few sentences in the big file doc?