search

ipfs-inactive / archives

[ARCHIVED] Repo to coordinate archival efforts with IPFS
https://awesome.ipfs.io/datasets
183 stars 24 forks source link

Sprint: Data.gov (aka 300 TB Challenge) #87

Open flyingzumwalt opened 7 years ago

flyingzumwalt commented 7 years ago

Dates: 16-27 January 2017 Sprint Milestone: https://github.com/ipfs/archives/milestone/1 Waffle Board: https://waffle.io/ipfs/archives

Participants from IPFS Team:

Collaborators

Advisors

Description

During this sprint, we will work with collaborating institutions to load all of data.gov (350 TB of datasets) into IPFS, publish the hashes on the DHT, and replicate the data to nodes at participating institutions.

Main Issues & Boards for Tracking this Work

Sprint Milestone: https://github.com/ipfs/archives/milestone/1 Waffle Board: https://waffle.io/ipfs/archives

Main Issues for Tracking this work:

Objectives

Top level objectives

What will be Downloaded

The data.gov website is a portal for searching through all the open data published by US federal agencies. It currently lists over 190,000 datasets. The goal is to download those datasets, back them up, and eventually publish them on IPFS, and replicate them across multiple institutions.

How is this different from the Internet Archive's EOT Harvest of data.gov?

In short,the End of Term Presidential Harvest will capture the data.gov website but is not likely to capture the datasets that it links to. We aim to capture and replicate the datasets.

From the Federal Depository Library Program website:

The Library of Congress, California Digital Library, University of North Texas Libraries, Internet Archive, George Washington University Libraries, Stanford University Libraries, and the U.S. Government Publishing Office (GPO) are leading a collaborative project to harvest and preserve public U.S. Government websites at the conclusion of the current Presidential administration ending on January 20, 2017.

This End of Term Presidential Harvest is focused on crawling websites (ie. grabbing HTML & static files, following links), including data.gov. It is not focused on downloading whole datasets, which often have to be retrieved using tools/processes beyond the capabilities of regular web crawlers.

You can see the ongoing EOT Harvest work, and nominate sites for harvest, at the End of Term Presidential Harvest 2016 Website

Obstacles

Notes

Isolating from the Main IPFS Network: We might do this on a separate "private" IPFS network to ensure stability while we load-test the system (we want to make sure it's all working smoothly before we flood the main public IPFS network with provide statements and additions to the DHT.

Possible Areas of Focus for Engineering Efforts

Areas that we might focus on in this sprint (needs prioritization):

flyingzumwalt commented 7 years ago

@jbenet suggested here:

Some things to do that would help, before tue:

  • review existing filestore stuff
  • review ipfs-pack draft proposal -- https://github.com/ipfs/notes/issues/205
  • make a short list of the "big bugs" and "big optimizations" relevant for this sprint. (i can think of a couple -- file attrs, bitswap supporting paths (kills so many RTTs) -- , but we'll want a good list to have in mind)
  • refine the concrete use case and UX we're shooting for
  • create the test workloads + scripts we'll work against:
  • review ipfs-s3 datastore options -- https://github.com/ipfs/notes/issues/214
flyingzumwalt commented 7 years ago

Sprint Prep Action Items

@jbenet:

Together:

@flyingzumwalt:

kevina commented 7 years ago

This sounds like it involves me. Is there a reason I am not mentioned?

flyingzumwalt commented 7 years ago

@kevina this sprint arose very quickly based on sudden interest outside our org. I'm putting together the docs as quickly as I can so we can coordinate. I suspect that @jbenet and @whyrusleeping will pull you onto the sprint if you're available.

flyingzumwalt commented 7 years ago

Sprint Planning: data.gov (aka 300 TB Challenge)

Date: 2017-17-01

Lead: @flyingzumwalt

Notetaker: @flyingzumwalt

Participants

Notes

Useful Links & Issues

Big Optimizations

TODO: dig up diagram @whyrusleeping created

Test Suite

See ipfs/archives#102

Currently not scaling well. Don't have good metrics, graphs or reports about performance -- where/when/how performance dipped under certain circumstances.

We need to know more than "Does it scale?". We need to know "how does it scale?" So we can identify the domain of problems, etc.

Filestore

See filestore Stories & Epics

The current implementation mixes porcelain UX concerns with the underlying iplementation/plumbing. This makes the interfaces confusing & complicated. It also makes the underlying plumbing more complicated and less robust than it should be. Best approach: take the pieces of the code that we need and package it as an experimental feature with simple, straightforward interfaces.

@jbenet & @whyrusleeping need to sit down and figure out how they want to proceed with this. @flyingzumwalt will try to capture that info in the filestore Stories & Epics Main things that need to be specified:

ipfs-pack

The case for ipfs-pack

Currently the way people use go-ipfs is with ipfs add which creates a duplicate copy of the added data on the machine. With filestore we aim to build indexes of pointers to data/blocks in-place. This solves performance concerns, but creates a brittle situation -- if you move the file, ipfs won't be able to serve it any more. ipfs-pack aims to address this by building manifest files that hold the indexes that match ipfs hashes to the content. If you store those manifest files alongside the cotnent they point to, it becomes a portable dataset.

Extending that idea, if you create little .ipfs repositories next to the manifest files, it becomes possible to

Why implement ipfs-pack now?

kevina commented 7 years ago

I listened in from around 1:00 to 1:30.

I like the idea of ipfs-pack, but I see some potential problems. i have not had time to review the spec so it will be premature to bring them up.

I too would like to be present for the meeting on the filestore core so I can give feedback before we try to implement anything, there are some tricky aspects regarding multiple files with the same hash that need to be addressed for this to be considered a stable format. Most likely, the existing code can be adopted.

flyingzumwalt commented 7 years ago

@mejackreed so @whyrusleeping can do more realistic load testing in #126 can you please run du -a {path-to-data} on the datasets you've downloaded so far?

mejackreed commented 7 years ago

currently 2053258584 /data/master/pairtree_root/

Kubuxu commented 7 years ago

@mejackreed we are also interested in whole output of this command, it will allows us to know the distribution of filesizes, directories and so on.

mejackreed commented 7 years ago

@Kubuxu sorry for the delay. Here you go: https://drive.google.com/file/d/0BzWuWHFTTIPERnpJSGJzYUFkYTA/view?usp=sharing

Kubuxu commented 7 years ago

compressed version: https://ipfs.io/ipfs/QmW82hMetgeM1K4dTFL5w9aDsZ3LYekYkeMgHfY6wtED5c

flyingzumwalt commented 7 years ago

Report from data.gov Sprint

The IPFS team have reached the end of our data.gov Sprint. Due to constraints on our very busy Q1 Roadmap, we were only able to allocate a single sprint 16-27 January 2017 (2 weeks) to work on this full-time. While we didn't reach all of the objectives, we have done our best to clear the path for our collaborators to finish the experiment. In the coming weeks, @flyingzumwalt will continue to participate in the project and the IPFS maintainers will provide information & advice when possible.

Within the IPFS team, we were excited to have the opportunity to help. This situation gets at one of the key reasons why we're building IPFS -- we want everyone to be able to hold and serve copies of the data they care about rather than relying on centralized services.

What we Accomplished

A number of collaborators have stepped up on very short notice to replicate these datasets. We're happy to tell them that the software is ready for you to use. Here's what @whyrusleeping, @kubuxu, @kevina, @jbenet and @flyingzumwalt have done:

Next Steps

Next Steps with the data.gov Datasets

Specifically regarding the data.gov Datasets, next steps include:

@flyingzumwalt will remain the main point of contact on the ipfs side coordinating this work.

Next Steps for the IPFS Code Base

Relevant follow-up work on the IPFS code bases involve:

Here's a breakdown of each:

ipfs-cluster: IPFS Nodes Coordinating to Hold datasets

The DataRescue effort has triggered multiple requests for tools that allow IPFS nodes to coordinate with each other in order to hold valuable content. We were already working on this functionality, under the name ipfs-cluster. The captain for ipfs-cluster is @hsanjuan. That code base will be moving forward throughout the quarter.

Some relevant discussions related to ipfs-cluster:

More Testing and Optimization to Come

We wish we could have done more testing and optimizing before the collaborators started replicating the actual data.gov datasets, but that work will have to wait for a few more weeks. We have two sprints scheduled later in the quarter that will be specifically focused on Improving our Testing & CI Infrastructure and Building a Proper Test Lab for Distributed Networks. We're confident that those tests will allow us to achieve major improvements in speed, stability, configurability, and security.

Deduplication of Datasets

In the aftermath of this high-speed effort to download datasets, people are now asking how to deduplicate datasets. This becomes especially relevant when we consider distributing and archiving datasets as they change over time -- if parts of the datasets stay the same between versions, we want to avoid storing & replicating them multiple times.

This has spurred interest in the different chunking algorithms IPFS supports. In particular, people are taking interest in rabin fingerprinting. Here are some Github issues where the discussion is happening:

Anything Else?

If we've missed anything important from our list of Next Steps, please let us know so we don't lose track of it.