search

ipfs / kubo

An IPFS implementation in Go
https://docs.ipfs.tech/how-to/command-line-quick-start/
Other
15.84k stars 2.97k forks source link

[WIP] Filestore Implementation #2634

Closed kevina closed 7 years ago

kevina commented 8 years ago

Closes Issue #875, avoid duplicating files added to ipfs

NOT READY FOR MERGE

Rebased #2600 on master.

Quicklinks: Code, README,

TODO to get this merged:

Note The filestore is very basic right now, but it is functional. I will likely continue to improve and submit new pull requests for the enhanced functionally but right now I fell it is important a basic implementation gets in so that it will get used, it can be labeled as an experimental feature and disabled by default, but available for those that want to use it. I consider the code production ready.

Resolves #875

whoizit commented 7 years ago

i need 'ipfs add --no-copy' by default btw.

kevina commented 7 years ago

@atommixz

i need 'ipfs add --no-copy' by default btw.

note, that it is ipfs filestore add now, and there is no way to make it "by default".

kevina commented 7 years ago

I picked out all the non-filestore specific bits I could. Here is an updated diff:

# (git diff --stat next..kevina/filestore  -- . ':!*filestore*' ':!*t026*.sh')
$ git diff --stat 31fcd1855..e7db3e4a1  -- . ':!*filestore*' ':!*t026*.sh'
 cmd/ipfs/ipfs.go                   |  2 ++
 core/builder.go                    |  7 +++++++
 core/commands/add.go               | 23 ++++++++++++++++++++++-
 core/commands/root.go              |  1 +
 repo/config/config.go              |  1 +
 repo/config/datastore.go           | 11 +++++++++++
 repo/fsrepo/defaultds.go           | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 test/sharness/t0235-cli-request.sh |  2 +-
 8 files changed, 91 insertions(+), 2 deletions(-)
jefft0 commented 7 years ago

Filestore: Support Raw Blocks.

@kevina, should I re-add my files so that they use raw blocks? (Tell me when is a good time because it will take days to re-add.)

kevina commented 7 years ago

should I re-add my files so that they use raw blocks? (Tell me when is a good time because it will take days to re-add.

I would wait, the implementation is still experimental. Raw block support is not even merged into ipfs-go/master yet. And even when it is there is no rush as the old format will continue to be supported for a good while (at least I think so, @whyrusleeping correct?).

kevina commented 7 years ago

@jefft0, I should add that one of the advantages of using the filestore is that you can have both formats (the original and with raw leaves) at the same time with little space wastage.

rht commented 7 years ago

This sure is a feat, pulled singlehandedly on filestore, :clap: :clap:, @kevina! If this PR is carefully splitted into 10 PR's consisting of 10 commits each (if 50% have already been merged), this should be doable within a week (or two the longest).

whyrusleeping commented 7 years ago

:raised_hands: :fireworks: #3629 has been merged, basic filestore functionality is in master. Go try it out :)