Closed char-1ee closed 11 months ago
Bridge could not be created" bridge=br0
means that the node is not clean. Please re-run on a clean cloudlab node.
So can I understand it as I dirty the node in certain steps. Do you think it is possibly caused when I set up/config the node with my Cloudlab profile (before I run anything on the node)
mostl likely, you ran the experiment several times and the experiment failed (or was interrupted) at least once. I recommend to start with a clean node after each failure of the cluster.
Ok I see, lemme retry
sudo mkdir -m777 -p /tmp/ctrd-logs && sudo env "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin" /usr/local/bin/firecracker-containerd --config /etc/firecracker-containerd/config.toml 1>/tmp/ctrd-logs/fccd_orch_noupf_log.out 2>/tmp/ctrd-logs/fccd_orch_noupf_log.err &
sudo env "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin" go test vhive_test.go stats.go vhive.go functions.go -short -v -race -cover
go: downloading github.com/stretchr/testify v1.8.0
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading gopkg.in/yaml.v3 v3.0.1
time="2023-08-30T14:24:41.771837368Z" level=info msg="Orchestrator snapshots enabled: false"
time="2023-08-30T14:24:41.771977147Z" level=info msg="Orchestrator UPF enabled: false"
time="2023-08-30T14:24:41.772040682Z" level=info msg="Orchestrator lazy serving mode enabled: false"
time="2023-08-30T14:24:41.772084671Z" level=info msg="Orchestrator UPF metrics enabled: false"
time="2023-08-30T14:24:41.772134434Z" level=info msg="Drop cache: true"
time="2023-08-30T14:24:41.772166611Z" level=info msg="Bench dir: bench_results"
time="2023-08-30T14:24:41.772211646Z" level=info msg="Registering bridges for tap manager"
time="2023-08-30T14:24:41.774167792Z" level=info msg="Creating containerd client"
time="2023-08-30T14:24:41.775977911Z" level=info msg="Created containerd client"
time="2023-08-30T14:24:41.776070188Z" level=info msg="Creating firecracker client"
time="2023-08-30T14:24:41.776338308Z" level=info msg="Created firecracker client"
=== RUN TestSendToFunctionSerial
time="2023-08-30T14:24:41.776693198Z" level=info msg="New function added" fID=1 image="ghcr.io/ease-lab/helloworld:var_workload" isPinned=true servedTh=0
time="2023-08-30T14:24:42.982803460Z" level=panic msg="Failed to get/pull image: failed to stat snapshot sha256:5216338b40a7b96416b8b9858974bbe4acc3096ee60acbc4dfb1ee02aecceb10: snapshotter not loaded: devmapper: invalid argument"
--- FAIL: TestSendToFunctionSerial (1.21s)
panic: (*logrus.Entry) 0xc0001b4460 [recovered]
panic: (*logrus.Entry) 0xc0001b4460
goroutine 11 [running]:
testing.tRunner.func1.2({0x1a927c0, 0xc0001b4460})
/usr/local/go/src/testing/testing.go:1396 +0x372
testing.tRunner.func1()
/usr/local/go/src/testing/testing.go:1399 +0x5f0
panic({0x1a927c0, 0xc0001b4460})
/usr/local/go/src/runtime/panic.go:890 +0x262
github.com/sirupsen/logrus.(*Entry).log(0xc0001b4310, 0x0, {0xc0002ee160, 0xae})
/users/char/go/pkg/mod/github.com/sirupsen/logrus@v1.9.0/entry.go:260 +0xa46
github.com/sirupsen/logrus.(*Entry).Log(0xc0001b4310, 0x0, {0xc00051f5f0, 0x1, 0x1})
/users/char/go/pkg/mod/github.com/sirupsen/logrus@v1.9.0/entry.go:304 +0x8c
github.com/sirupsen/logrus.(*Logger).Log(0xc0000f4000, 0x0, {0xc00051f5f0, 0x1, 0x1})
/users/char/go/pkg/mod/github.com/sirupsen/logrus@v1.9.0/logger.go:204 +0x85
github.com/sirupsen/logrus.(*Logger).Panic(...)
/users/char/go/pkg/mod/github.com/sirupsen/logrus@v1.9.0/logger.go:253
github.com/sirupsen/logrus.Panic(...)
/users/char/go/pkg/mod/github.com/sirupsen/logrus@v1.9.0/exported.go:129
command-line-arguments.(*Function).AddInstance(0xc000390210)
/users/char/go/src/vHive/functions.go:361 +0x431
command-line-arguments.(*Function).Serve.func1()
/users/char/go/src/vHive/functions.go:257 +0x153
sync.(*Once).doSlow(0xc0003b4304, 0xc0000d3c38)
/usr/local/go/src/sync/once.go:74 +0x102
sync.(*Once).Do(0xc0003b4304, 0x1a9d22a?)
/usr/local/go/src/sync/once.go:65 +0x47
command-line-arguments.(*Function).Serve(0xc000390210, {0x1a9d22a?, 0x1?}, {0x1ac6058?, 0x28?}, {0x9d2aab?, 0xc000014630?}, {0x1a9eef6, 0x5})
/users/char/go/src/vHive/functions.go:251 +0x465
command-line-arguments.(*FuncPool).Serve(0x2149400?, {0x1c74118, 0xc00003e128}, {0x1a9d22a, 0x1}, {0x1ac6058, 0x28}, {0x1a9eef6, 0x5})
/users/char/go/src/vHive/functions.go:121 +0xea
command-line-arguments.TestSendToFunctionSerial(0x0?)
/users/char/go/src/vHive/vhive_test.go:106 +0x155
testing.tRunner(0xc0000f36c0, 0x1b01160)
/usr/local/go/src/testing/testing.go:1446 +0x217
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1493 +0x75e
FAIL command-line-arguments 1.249s
FAIL
make: *** [Makefile:47: test] Error 1
time="2023-08-30T14:24:42.982803460Z" level=panic msg="Failed to get/pull image: failed to stat snapshot
this is a weird error, can you clean up and try again?
image should be vhive-ubuntu-20
Build cmd should be go build -race -v -a ./...
in my case the first run of make test didnt work but after cleaning, running make test-man-bench, killing it after awhile and then cleaning and finally running make test again
resolved? otherwise please re-open
Env
Profile: OpenStack Image: emulab-ops/UBUNTU22-64-STD Cluster: Utah xl170 (3x)
Workflow
vHive/.github/workflows/integration_tests.yml
Steps
Download project code and prepare the environment
Install Golang and add into PATH
Install Python
Noticed that in the workflow, it uses actions/setup-go@v4 and actions/setup-python@v4. The known difference between manual installation and installation with Github Action is Github Action provides additional features like caching, version management, and compatibility across different platforms.
Upgrade Git and git-lfs
(Skipped) Check out code into the Go module directory
Build setup scripts
Pull binaries
Build
Run vHive end-to-end tests
Archive log artifacts (used always())
Cleaning