Testing possible formats for the open-data repository (meta)

[evaristoc]

The current testing repo: https://github.com/evaristoc/open-data

• [x] Creating a folder inside the repository for an already existing repository SEE: https://github.com/freeCodeCamp/open-data/issues/8 => NONRECOMMENDED
• [x] Creating a folder using the git submodule command SEE: https://github.com/freeCodeCamp/open-data/issues/9
• [ ] Evaluating external repositories for datasets, creating folders where how to accessing the data would be explained and a initial script would be provided SEE: https://github.com/freeCodeCamp/open-data/issues/19

Expected Results:

• A repository that it is easy to manage and to maintain ==> clear and simple procedures and templates
• Rapid availability to the user, without too many steps
• As much as possible contributor authorship respected
• Not creating burden to the contributors in term of memory, etc.
• Centralization of the dataset but descentralization of the contributions
• Clear visibility of freeCodeCamp as the main reservoir of open datasets (linked to centralized approach and the use of Github open-data repository)
• Clear but flexible policy for data citation and publication

---

Task 1:

Create at least a first demo with the selected options

[evaristoc]

@QuincyLarson and @erictleung please check above.

Issues for each test point would be opened. The result should be a concise structure showing the best repository format possible based on either one or an appropriate combination of options, taking the limitations and possibilities of Github and other resources.

[evaristoc]

@pdurbin the invitation is also extended to you?

[QuincyLarson]

@evaristoc Awesome - this looks like a sane path forward to make sure GitHub is likely to work as a hosting platform for all these data going forward.

[evaristoc]

@QuincyLarson : I am using datadotworld as an example) and git-submodule. I think that the tasks for anyone in charge of managing this repository could be:

• to accept/reject some contributor's Github repositories as submodules of an specific dataset subfolder when those repositories were related to the dataset in question; when works around the datasets are not in Github, try to find ways to reference them
• to evaluate and provide information about the datasets (format, metadata, etc)
• when the data is large and/or relevant, then to manage an external fCC storage and provide enough information about how to get the datasets from there
• to encourage the use of the datasets and projects around them
• to specify standards about how to create projects or work on datasets in order to facilitate replication

[evaristoc]

@QuincyLarson

I let you know my idea of how the open-data repo should be organised:

• Should include official and other datasets when about fCC
• Every dataset (or related ones) would open a folder when there will be documentation describing the project that motivated the dataset(s) as well as describing the dataset(s) it(them)self
• If datasets are small enough, keep the dataset in Github
• In the dataset(s) folder, create subfolders as submodules from contributors working on the dataset(s)
• If big and/or relevant to fCC, look for an external data storage that facilitate storage, easy access as well as additional exposure; submodules will be also considered when related to the dataset

So far I have been enjoyed datadotworld data storage. However it is important to understand your ideas to see if this is the most convenient place for the project.

Be aware that although I like data.world and the current structure, all this is a proposal that I expect to evaluate with you. We can find out eventually that data.world might not be the best external storage, or you can suggest that no external storage will be used, for example.

[QuincyLarson]

@evaristoc Your proposal seems sound. I would recommend having GitHub be the singe authoritative place where we store the data, and for individual files that exceed what GitHub is capable of storing, we link to their trackers on BitTorrent (like we did with open datasets in the past).

[evaristoc]

@QuincyLarson

I will start working on this then. I will share my views with your personally about BitTorrent in Gitter.

[evaristoc]

@QuincyLarson shared:

if you think there's a better place to host it other than BitTorrent, that is:

• free
• allows for unauthenticated access