Don't store data in the repo

Stefan-Endres / DWPM-Mixture-Model

Phase seperation calculation using the DWPM mixture rule.

7 stars 3 forks source link

Don't store data in the repo #3

Closed alchemyst closed 8 years ago

alchemyst commented 9 years ago

It is by and large not a good idea to store data in the repository. It's likely to change often, leading to messy commits. It can also get large, leading to large commits. If multiple people run the code, the data can also change differently leading to messy merges.

The pattern that I have found to work well is this:

Have only code and a very small amount of static data in the repository itself (so stuff like the gas constant is not going to change, for instance).
Have a config file which you read the location of the data from (see for instance how it is handled in pulpsim)
Store the data on Dropbox if it takes long to generate or is to be used by everyone who wants to run the code, or locally if the program produces output which is fast to produce

Stefan-Endres commented 9 years ago

I some point I'll do a total revamp of the data handling including implementing Panda over csvDict as per issue #1. The current .csv files are temporary to test the code.

alchemyst commented 9 years ago

Not all these issues are equally urgent. I just add them so that they have a place to be, don't feel compelled to act on all of them immediately.

On 08 Oct2015, at 17:47 , Stefan-Endres notifications@github.com wrote:

I some point I'll do a total revamp of the data handling including implementing Panda over csvDict as per issue #1. The current .csv files are temporary to test the code.

— Reply to this email directly or view it on GitHub.

__ ,------------http://www.up.ac.za/chemeng/csandrock-------------. | N-CH3 | | ,---.,----. Carl Sandrock / \ | | / ,--'| ---< MEng (Control), PrEng + N---C C=O | | \ ------ ) University of Pretoria + " " | | | ~-------~ carl.sandrock@up.ac.za +++ CH C N-CH3 | | \ / \ / | | Brought to you by the miracle of Caffeine 3HC-N C=O | --------------------------------------------------------------

alchemyst commented 8 years ago

I've now started running the code and I see that the code modifies the CSV files in the repo. I am kind of OK with the idea of storing static component information in the repo, but storing program outputs is a very bad idea, as we will start developing contradictory versions of the file every time we run the code! I really think the products of the code should not be stored in the repo.

Stefan-Endres commented 8 years ago

Config files implemented with 1c0f3fa. Add personal project dropbox directory ~/Dropbox/DWPM Mixutre Model Project/Data/ as in the config example.cfg file to run code.