Rerun vHive integration test on clean node

[char-1ee]

Env

Profile: OpenStack Image: emulab-ops/UBUNTU22-64-STD Cluster: Utah xl170 (3x)

Workflow

vHive/.github/workflows/integration_tests.yml

Steps

Download project code and prepare the environment

git clone https://github.com/vhive-serverless/vHive.git
cd vHive
sudo apt install build-essential zlib1g-dev libncurses5-dev\
 libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev curl
export GOOS=linux
export GO111MODULE=on

Install Golang and add into PATH

export GO_VERSION=1.19
wget https://dl.google.com/go/go$GO_VERSION.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go$GO_VERSION.linux-amd64.tar.gz
echo "export PATH=\$PATH:/usr/local/go/bin" >> ~/.bashrc
echo "export GOPATH=\$HOME/go" >> ~/.bashrc
echo "export PATH=\$PATH:\$GOPATH/bin" >> ~/.bashrc
source ~/.bashrc
go --version

Install Python

wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tar.xz
tar -xvf Python-3.7.4.tar.xz
cd Python-3.7.4
./configure
python --version

Noticed that in the workflow, it uses actions/setup-go@v4 and actions/setup-python@v4. The known difference between manual installation and installation with Github Action is Github Action provides additional features like caching, version management, and compatibility across different platforms.

Upgrade Git and git-lfs

sudo add-apt-repository ppa:git-core/ppa -y
sudo apt update
sudo apt install git -y

curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
sudo apt-get install git-lfs
git-lfs install

(Skipped) Check out code into the Go module directory

uses: actions/checkout@v3

Build setup scripts

cd scripts 
go build -o setup_tool 
cd ..

Pull binaries

./scripts/setup_tool setup_firecracker_containerd

Build

go build -race -v -a ./..

Run vHive end-to-end tests

make test
# or make test-man-bench

Archive log artifacts (used always())

# Archive log artifacts
mkdir -p /tmp/ctrd-logs
cp -r ~/go/src/vHive/*.log /tmp/ctrd-logs/
cp -r ~/go/src/vHive/scripts/github_runner/*.log /tmp/ctrd-logs/
tar -czvf ctrd-logs.tar.gz -C /tmp ctrd-logs

# Upload the archived artifacts (optional)
curl -sLJO "https://github.com/actions/upload-artifact/releases/download/v3.0.0/upload-artifact-linux"
chmod +x upload-artifact-linux
./upload-artifact-linux ctrd-logs.tar.gz

Cleaning

./scripts/setup_tool clean_fcctr

[ustiugov]

Bridge could not be created" bridge=br0

means that the node is not clean. Please re-run on a clean cloudlab node.

[char-1ee]

So can I understand it as I dirty the node in certain steps. Do you think it is possibly caused when I set up/config the node with my Cloudlab profile (before I run anything on the node)

[ustiugov]

mostl likely, you ran the experiment several times and the experiment failed (or was interrupted) at least once. I recommend to start with a clean node after each failure of the cluster.

[char-1ee]

Ok I see, lemme retry

[char-1ee]

sudo mkdir -m777 -p /tmp/ctrd-logs && sudo env "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin" /usr/local/bin/firecracker-containerd --config /etc/firecracker-containerd/config.toml 1>/tmp/ctrd-logs/fccd_orch_noupf_log.out 2>/tmp/ctrd-logs/fccd_orch_noupf_log.err &
sudo env "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin" go test vhive_test.go stats.go vhive.go functions.go -short -v -race -cover
go: downloading github.com/stretchr/testify v1.8.0
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading gopkg.in/yaml.v3 v3.0.1
time="2023-08-30T14:24:41.771837368Z" level=info msg="Orchestrator snapshots enabled: false"
time="2023-08-30T14:24:41.771977147Z" level=info msg="Orchestrator UPF enabled: false"
time="2023-08-30T14:24:41.772040682Z" level=info msg="Orchestrator lazy serving mode enabled: false"
time="2023-08-30T14:24:41.772084671Z" level=info msg="Orchestrator UPF metrics enabled: false"
time="2023-08-30T14:24:41.772134434Z" level=info msg="Drop cache: true"
time="2023-08-30T14:24:41.772166611Z" level=info msg="Bench dir: bench_results"
time="2023-08-30T14:24:41.772211646Z" level=info msg="Registering bridges for tap manager"
time="2023-08-30T14:24:41.774167792Z" level=info msg="Creating containerd client"
time="2023-08-30T14:24:41.775977911Z" level=info msg="Created containerd client"
time="2023-08-30T14:24:41.776070188Z" level=info msg="Creating firecracker client"
time="2023-08-30T14:24:41.776338308Z" level=info msg="Created firecracker client"
=== RUN   TestSendToFunctionSerial
time="2023-08-30T14:24:41.776693198Z" level=info msg="New function added" fID=1 image="ghcr.io/ease-lab/helloworld:var_workload" isPinned=true servedTh=0
time="2023-08-30T14:24:42.982803460Z" level=panic msg="Failed to get/pull image: failed to stat snapshot sha256:5216338b40a7b96416b8b9858974bbe4acc3096ee60acbc4dfb1ee02aecceb10: snapshotter not loaded: devmapper: invalid argument"
--- FAIL: TestSendToFunctionSerial (1.21s)
panic: (*logrus.Entry) 0xc0001b4460 [recovered]
panic: (*logrus.Entry) 0xc0001b4460

goroutine 11 [running]:
testing.tRunner.func1.2({0x1a927c0, 0xc0001b4460})
    /usr/local/go/src/testing/testing.go:1396 +0x372
testing.tRunner.func1()
    /usr/local/go/src/testing/testing.go:1399 +0x5f0
panic({0x1a927c0, 0xc0001b4460})
    /usr/local/go/src/runtime/panic.go:890 +0x262
github.com/sirupsen/logrus.(*Entry).log(0xc0001b4310, 0x0, {0xc0002ee160, 0xae})
    /users/char/go/pkg/mod/github.com/sirupsen/logrus@v1.9.0/entry.go:260 +0xa46
github.com/sirupsen/logrus.(*Entry).Log(0xc0001b4310, 0x0, {0xc00051f5f0, 0x1, 0x1})
    /users/char/go/pkg/mod/github.com/sirupsen/logrus@v1.9.0/entry.go:304 +0x8c
github.com/sirupsen/logrus.(*Logger).Log(0xc0000f4000, 0x0, {0xc00051f5f0, 0x1, 0x1})
    /users/char/go/pkg/mod/github.com/sirupsen/logrus@v1.9.0/logger.go:204 +0x85
github.com/sirupsen/logrus.(*Logger).Panic(...)
    /users/char/go/pkg/mod/github.com/sirupsen/logrus@v1.9.0/logger.go:253
github.com/sirupsen/logrus.Panic(...)
                       /users/char/go/pkg/mod/github.com/sirupsen/logrus@v1.9.0/exported.go:129
command-line-arguments.(*Function).AddInstance(0xc000390210)
    /users/char/go/src/vHive/functions.go:361 +0x431
command-line-arguments.(*Function).Serve.func1()
        /users/char/go/src/vHive/functions.go:257 +0x153
sync.(*Once).doSlow(0xc0003b4304, 0xc0000d3c38)
    /usr/local/go/src/sync/once.go:74 +0x102
sync.(*Once).Do(0xc0003b4304, 0x1a9d22a?)
    /usr/local/go/src/sync/once.go:65 +0x47
command-line-arguments.(*Function).Serve(0xc000390210, {0x1a9d22a?, 0x1?}, {0x1ac6058?, 0x28?}, {0x9d2aab?, 0xc000014630?}, {0x1a9eef6, 0x5})
    /users/char/go/src/vHive/functions.go:251 +0x465
command-line-arguments.(*FuncPool).Serve(0x2149400?, {0x1c74118, 0xc00003e128}, {0x1a9d22a, 0x1}, {0x1ac6058, 0x28}, {0x1a9eef6, 0x5})
    /users/char/go/src/vHive/functions.go:121 +0xea
command-line-arguments.TestSendToFunctionSerial(0x0?)
    /users/char/go/src/vHive/vhive_test.go:106 +0x155
testing.tRunner(0xc0000f36c0, 0x1b01160)
    /usr/local/go/src/testing/testing.go:1446 +0x217
created by testing.(*T).Run
    /usr/local/go/src/testing/testing.go:1493 +0x75e
FAIL    command-line-arguments 1.249s
FAIL
make: *** [Makefile:47: test] Error 1

[ustiugov]

time="2023-08-30T14:24:42.982803460Z" level=panic msg="Failed to get/pull image: failed to stat snapshot

this is a weird error, can you clean up and try again?

[BenjaminChun]

image should be vhive-ubuntu-20 Build cmd should be go build -race -v -a ./...

in my case the first run of make test didnt work but after cleaning, running make test-man-bench, killing it after awhile and then cleaning and finally running make test again

[ustiugov]

resolved? otherwise please re-open