search

ipfs-shipyard / git-remote-ipld

MIT License
147 stars 33 forks source link

Wrap lobjs so they can be deduped w/ other IPFS objects, fixes #12 #21

Closed sameer closed 11 months ago

sameer commented 5 years ago

This PR fixes #12 by splitting large objects into a header and raw data which are individually hashed then patched together for the lobj hash. I kept a fallback to just adding the entire object if the header can't be found.

I ran it against a repo, one 3.6MB lobj example is:

headerHash = QmddzD2eZgPcLHYku5DMFXgBXh8GPzPGr3SaBr4ioPuVDU
dataHash= QmTBDL8wuSbjF5eCzkwdPXA2sTiTNmwKtqUUkJTYWKWa4W
objectHash = QmURkbZP6hVdt4MAo9CBv4rJxNUA7x57UKgHtzxgpfY8Np

Not exactly sure how to add tests for this -- any guidance is appreciated.

Edit: tests seem to fail on macOS but I think it's unrelated.

sameer commented 5 years ago

So I added a lobj branch to the mock git repo with a file large enough to generate a lobj that would hit my changes on a push. However, IpnsHandler.Push() is not called because the testing never does an actual push. I will try to add a push test.

Edit: I misspoke, there is a push test, but the large object does not get sent because it's pushing refs/heads/master.

sameer commented 5 years ago

Discovered that the push tests were never actually run because there was a missing newline at the end of testCase(t, args, "push refs/heads/master:refs/heads/master\n", []string{}).

magik6k commented 5 years ago

Jenkins is broken in ipfs-shipyard for go projects, I'll try to work around it tomorrow if time permits

sameer commented 5 years ago

Looks like macOS is still failing:

[git-remote-ipld] Running shell script

+ go test -v ./...

+ tee output

/var/jenkins/workspace/pyard_git-remote-ipld_PR-21-BEUY3MIKEUQKTXCLQUO4RZM773OQDB3XCGN6CAIWBSB6ADFZOXKQ/src/github.com/ipfs shipyard/git-remote-ipld/cmd/git-remote-ipld/ipld.go:15:2: cannot find package "github.com/ipfs-shipyard/git-remote-ipld/core" in any of:

    /var/jenkins/tools/org.jenkinsci.plugins.golang.GolangInstallation/1.11/src/github.com/ipfs-shipyard/git-remote-ipld/core (from $GOROOT)

    /var/jenkins/workspace/pyard_git-remote-ipld_PR-21-BEUY3MIKEUQKTXCLQUO4RZM773OQDB3XCGN6CAIWBSB6ADFZOXKQ/src/github.com/ipfs-shipyard/git-remote-ipld/core (from $GOPATH)