CERN

[ghost]

http://opendata.cern.ch

CERN is, since the end of 2014, serving some fraction of the colossal amount of captured data about particle collision in LHC (with detectors like CMS, ATLAS, ALICE) - summing up to 60,000,000 GB.

Through the help of a small Python crawler, I've compiled an index of all CMS-detector primary datasets (all .root files totaling cca. 27,4TB). Also index of indexes. Other detector indexes of datasets + derivative datasets to come :)

• [ ] CMS
  • [ ] Primary datasets from 2010 runs (27,4TB)
  • [x] Scrap data from CERN's OpenData (via cmspull.py)
  • [x] Compile index of all primary dataset files (.root)
  • [ ] Somehow get those 28TB into IPFS (maybe in cooperation w/ CERN? - than all steps are unnecessary)
• [ ] ATLAS
• [ ] ALICE
• [ ] LHCb

To use all that data, a special environment is required - normally CERN's OpenData is recommending the use of their CernVM, which is basically Scientific Linux + ROOT, a data analysis framework (therefore the .root files). Without ROOT, this historical milestones cannot be used as computable data directly - so the tool must also be, as the collision data, preserved/archived. There's also a mirror right here at Github.

Oh, and thanks for the amazing project!

[davidar]

Wow, that's a lot of data! Unless you know someone with that much storage available, we might have to do this in cooperation with CERN? @jbenet Thoughts?

[jbenet]

i think we're not ready to ingest much CERN data. that's the biggest data cache there is. the metadata is possible now, but still big. here's what i propose:

• [ ] think through the CERN data and metadata and how best to dedup it-- we can write a custom archiver to generate merkledags that optimize the layout to benefit deduping their data
• [ ] either get a large machine with lots of disks somewhere in the backbone, or
• [ ] write ipfs-cluster (a tool to organize a set of machines to back a pinset together. using some form of consensus, likely etcd to get started).
• [ ] ingest the 28TB
• [ ] ship filecoin
• [ ] get this to happen to distributed storage https://blockchain.info/charts/hash-rate?timespan=all&showDataPoints=false&daysAverageString=1&show_header=true&scale=0&address=
• [ ] ingest the 60PB

[Kubuxu]

Crawler and index hashes are gone, any hope on restoring them?

[davidar]

Hmm, seems I forgot to pin this to one of the storage nodes. We really need a better system for mirroring archives stuff - I'll make this a priority for the new year.