unable to run DHT tests

[noot]

Platform: ubuntu 22.04

Installed testground as per the testground readme.

Then, inside test-plans/dht, I'm running testground build composition -f compositions/find-provs-both-local.toml -w (as per here) but getting an error in the daemon:

Step 21/29 : RUN cd ${PLAN_DIR}     && go env -w GOPROXY="${GO_PROXY}"     && CGO_ENABLED=${CgoEnabled} GOOS=linux go build -o ${PLAN_DIR}/testplan.bin ${BUILD_TAGS} ${TESTPLAN_EXEC_PKG}
 ---> Running in 8d81c3bec1b4
 ---> Running in 948b594050aa
test/find_providers.go:13:2: missing go.sum entry for module providing package github.com/ipfs/go-cid (imported by github.com/libp2p/test-plans/dht/test); to add:
    go get github.com/libp2p/test-plans/dht/test
test/common.go:23:2: missing go.sum entry for module providing package github.com/ipfs/go-datastore (imported by github.com/libp2p/test-plans/dht/test); to add:
    go get github.com/libp2p/test-plans/dht/test
/go/pkg/mod/github.com/ipfs/go-ds-leveldb@v0.4.1/datastore.go:8:2: missing go.sum entry for module providing package github.com/ipfs/go-datastore/query (imported by github.com/ipfs/go-ds-leveldb); to add:
    go get github.com/ipfs/go-ds-leveldb@v0.4.1
test/common.go:24:2: missing go.sum entry for module providing package github.com/ipfs/go-datastore/sync (imported by github.com/libp2p/test-plans/dht/test); to add:
    go get github.com/libp2p/test-plans/dht/test
// errors omitted

Running go get github.com/libp2p/test-plans/dht/test doesn't change the errors unfortunately. I also tried with compositions/find-provs.toml but have the same issue.

Since I need to update the deps for some DHT testing, I went ahead and did that (see here) but now I get a different error on build:

Step 21/29 : RUN cd ${PLAN_DIR}     && go env -w GOPROXY="${GO_PROXY}"     && CGO_ENABLED=${CgoEnabled} GOOS=linux go build -o ${PLAN_DIR}/testplan.bin ${BUILD_TAGS} ${TESTPLAN_EXEC_PKG}
 ---> Running in 9d17566f6389
/go/pkg/mod/github.com/prometheus/client_golang@v1.13.0/prometheus/desc.go:22:2: missing go.sum entry needed to verify package github.com/cespare/xxhash/v2 (imported by github.com/prometheus/client_golang/prometheus) is provided by exactly one module; to add:
    go get github.com/prometheus/client_golang/prometheus@v1.13.0
/go/pkg/mod/github.com/containerd/cgroups@v1.0.4/systemd.go:25:2: missing go.sum entry needed to verify package github.com/coreos/go-systemd/v22/dbus (imported by github.com/containerd/cgroups) is provided by exactly one module; to add:
    go get github.com/containerd/cgroups@v1.0.4
test/find_providers.go:13:2: //go:build comment without // +build comment
/go/pkg/mod/github.com/klauspost/compress@v1.15.10/zstd/blockdec.go:17:2: //go:build comment without // +build comment
/go/pkg/mod/github.com/libp2p/go-libp2p@v0.23.2/p2p/transport/quic/tracer.go:10:2: //go:build comment without // +build comment
test/common.go:32:2: //go:build comment without // +build comment
/go/pkg/mod/github.com/libp2p/go-libp2p@v0.23.2/defaults.go:10:2: //go:build comment without // +build comment
/go/pkg/mod/github.com/libp2p/go-libp2p@v0.23.2/defaults.go:12:2: //go:build comment without // +build comment
/go/pkg/mod/github.com/libp2p/go-libp2p@v0.23.2/p2p/transport/tcp/tcp.go:15:2: //go:build comment without // +build comment
/go/pkg/mod/github.com/libp2p/go-libp2p@v0.23.2/p2p/protocol/holepunch/holepuncher.go:14:2: //go:build comment without // +build comment
/go/pkg/mod/github.com/libp2p/go-libp2p@v0.23.2/defaults.go:16:2: //go:build comment without // +build comment
/go/pkg/mod/github.com/libp2p/go-libp2p@v0.23.2/p2p/muxer/yamux/conn.go:8:2: //go:build comment without // +build comment
/go/pkg/mod/github.com/libp2p/go-libp2p@v0.23.2/p2p/transport/quic/conn.go:12:2: //go:build comment without // +build comment
/go/pkg/mod/github.com/lucas-clemente/quic-go@v0.29.1/internal/qerr/error_codes.go:6:2: //go:build comment without // +build comment
/go/pkg/mod/github.com/libp2p/go-libp2p@v0.23.2/core/peer/peer.go:12:2: //go:build comment without // +build comment

Some quick googling tells me this is due to some modules using go version <1.17, but not sure exactly how this can be fixed. Will continue trying to fix this, any pointers in the meantime would be much appreciated!

[laurentsenta]

Hi @noot thanks for reaching out, I'm not familiar with this test, but it seems that this wasn't updated in two years, it's using old packages (testground go-sdk is 0.1, we're at 0.3).

This might be a lot of work, what are you trying to achieve?

It looks like it was already broken in 2020: https://github.com/libp2p/test-plans/issues?q=is%3Aissue+is%3Aopen+dht

If you want to use this test: start from a minimal working version and add back features & update packages. I took some time to simplify the composition file, hopefully this is a starting point you can build on top:

Use testground build composition -f compositions/find-provs-both-local.toml --wait to build the test (don't use -w, which writes artifacts to your composition file, this will create noise)

• commit 1: only build balsam, it works: https://github.com/libp2p/test-plans/commit/25d2dba01f7dae67a313655c0d38fda276da6cdb
• commit 2, adds cypress, it breaks: https://github.com/libp2p/test-plans/commit/e3bab7191d0e7d00de54aadf665a9b131e281345

This is where you probably want to fix the go build, and update go packages (calling go get THIS-REPO is probably NOT going to help).

You can debug the build locally, cd in the dht folder and call: go build -tags balsam (this is how the first group is built, the one with selector = balsam) and go build -tags cypress (second group).

[noot]

@laurentsenta Thanks for the pointers! The balsam build worked for me, I'll continue looking into that.

I'm planning to use testground to test that Provide/GetProvIders functionality still works as expected with a privacy-improved DHT, as well as potentially gather some metrics.

[noot]

@laurentsenta good news, I managed to update all the deps and fix the build for testground build composition -f compositions/find-provs-both-local.toml --wait by adding the following under [builders."docker:go"] in manifest.toml:

[builders."docker:go"]
go_version = "1.19"
build_base_image = "golang:1.19-buster"

The updated plan is here. https://github.com/ChainSafe/test-plans/tree/noot/dht/dht It works with both balsam and cypress.

[mxinden]

@noot thanks for the work! Would you mind creating a pull request? //CC @laurentsenta

[noot]

@mxinden done! #66 :)

[p-shahi]

Moving dht tests out of the repo see https://github.com/libp2p/test-plans/pull/129