search

kata-containers / runtime

Kata Containers version 1.x runtime (for version 2.x see https://github.com/kata-containers/kata-containers).
https://katacontainers.io/
Apache License 2.0
2.1k stars 374 forks source link

Documentation: Running Kata CI offline #1359

Closed mcastelino closed 3 years ago

mcastelino commented 5 years ago

Running Kata CI offline

Given that the Kata CI can take time to run and flag errors it will save developers time to run the CI offline. This can be done easily using ccloudvm in a non destructive way. Here are some early instruction

Setup ccloudvm

For more info https://github.com/intel/ccloudvm

go get github.com/intel/ccloudvm/...
$ ccloudvm setup

Create a ccloud fedora VM

ccloudvm create fedora28 --cpus 8 --disk 40 --mem 8192 --name kata-ci

Log into the VM and set it up for CI

ccloudvm connect kata-ci
sudo gpasswd -a $USER kvm
sudo dnf -y update
sudo dnf install -y git gcc dnf-plugins-core patch || ( sudo apt-get update -y && sudo apt-get install -y gcc git patch)
sudo mkdir /etc/docker
curl -LOk https://dl.google.com/go/go1.11.1.linux-amd64.tar.gz
sudo tar -xf go1.11.1.linux-amd64.tar.gz -C /usr/local
sudo sync
rm -rf go1.11.1.linux-amd64.tar.gz
mkdir -p ~/go
echo export GOPATH=~/go >> .bashrc 
echo export GOROOT=\"/usr/local/go\" >> .bashrc
echo export PATH=\"\$PATH:\$GOROOT/bin\" >> .bashrc
echo export KUBECONFIG=/etc/kubernetes/admin.conf >> .bashrc
source .bashrc
go get -d github.com/kata-containers/tests
pushd go/src/github.com/kata-containers/tests/
export CI=true
.ci/setup.sh
sudo -E gpasswd -a $USER docker

Run the CI on your code

cd ~/go/src/github.com/kata-containers/runtime
export PATH=${GOPATH}/bin:/usr/local/go/bin:/usr/sbin:${PATH}
export CI=true
.ci/run.sh
mcastelino commented 5 years ago

/cc @chavafg @GabyCT @devimc please feel to correct/amend the issue directly.

GabyCT commented 5 years ago

@mcastelino , I will try to reproduce these steps

mcastelino commented 5 years ago

@GabyCT we also ensure we should be able to run the CI multiple times. So if we need to do any clean between runs we should document it.

grahamwhaley commented 5 years ago

@mcastelino - that could be tricky - a CI run can mess up your system (by that I mean, if a test fails it is possible for it to leave a container, docker or some other item in a state that is hard to clean up - think hung up container or runtime etc.) For the bare metal servers we have a set of scripts at https://github.com/kata-containers/tests/blob/master/.ci/lib.sh#L291 that try to clean the machine up, by removing all relevant items - but - you might not want to do that to your dev desktop for instance. So, just beware a little ;-)

GabyCT commented 5 years ago

@mcastelino , currently I am doing the setting and then I will run the tests to see possible failures and if we can run them multiply times, however, from the steps above, I will modify the following: After creating the ccloudvm fedora VM, I will add a note saying that sometimes you need to do

sudo gpasswd -a $USER kvm

I will add this as I got the error of failed to initialize KVM: Permission denied

Then for the setup of the CI, I will modify:

sudo dnf -y update

Now, we do not need to install manually docker as this will be install (with the version that we are supporting) by just simply running the .ci/setup.sh from the test repository (https://github.com/kata-containers/tests/blob/master/.ci/setup.sh#L126 and https://github.com/kata-containers/tests/blob/master/cmd/container-manager/manage_ctr_mgr.sh#L83) , so we can remove those steps. Also, I will do the export CI=true before running the .ci/setup.sh.

mcastelino commented 5 years ago

@mcastelino - that could be tricky - a CI run can mess up your system (by that I mean, if a test fails it is possible for it to leave a container, docker or some other item in a state that is hard to clean up - think hung up container or runtime etc.) For the bare metal servers we have a set of scripts at https://github.com/kata-containers/tests/blob/master/.ci/lib.sh#L291 that try to clean the machine up, by removing all relevant items - but - you might not want to do that to your dev desktop for instance. So, just beware a little ;-)

@grahamwhaley exactly the reason why this will run in a ccloudvm. So the user can run the CI non destructively.

GabyCT commented 5 years ago

@mcastelino , I can not run properly the docker tests, I am getting the following error

$ docker run -ti --runtime kata-runtime busybox sh
docker: Error response from daemon: OCI runtime create failed: Failed to check if grpc server is working: rpc error: code = Unavailable desc = transport is closing: unknown.

I am going to try to use an ubuntu image. @chavafg have you ever try to running the tests using ccloudvm?

jodh-intel commented 5 years ago

Looks like this recipe is ripe for converting into a one-liner like we have for the automated install:

All the user would need to do is specify the name of the particular Kata repo they want to test (say runtime):

$ bash -c "$(curl -fsSL $url) runtime"

We'd need to determine golang version without hard-coding though so the script should query https://github.com/kata-containers/runtime/blob/master/versions.yaml.

GabyCT commented 5 years ago

@mcastelino , I run the ccloudvm using ubuntu, all the functional tests passed, only one of docker integration failed, CRI-O all passed, however, kubernetes only one failed and shimv2 failed (I could not run them).

ganeshmaharaj commented 5 years ago

I just ran this on a fedora-28 machine and all the tests seem to pass and complete without issues. This is what i see at the end of the run, sharing here to make sure that is what is expected.

Install Openshift
Install Openshift Origin
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   648    0   648    0     0   1417      0 --:--:-- --:--:-- --:--:--  1417
100  181M  100  181M    0     0  2699k      0  0:01:08  0:01:08 --:--:-- 3085k
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.selinux'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.selinux'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.selinux'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.selinux'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.selinux'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.selinux'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.selinux'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.selinux'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.selinux'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.selinux'
Openshift installed successfully
Disable systemd-journald rate limit
Drop caches

I also saw a note that says that k8s is only tested on ubuntu. 

Install Kubernetes components
Currently this script only works for Ubuntu. Skipped Kubernetes Setup

it would be nice to test this on fedora and others distros too.

ganeshmaharaj commented 5 years ago

@GabyCT @jodh-intel @mcastelino what are the next steps here? i have been using these steps consistently to test things locally and they work fairly well. Should we call this an experimental try at your own risk doc?

jodh-intel commented 5 years ago

@ganeshmaharaj - sounds good to me. Once this is documented, my only concern is how we ensure it continues to work. We either have a separate CI test for offline mode that we run periodically and/or we add a big red warning as you're suggesting. Let's start with a doc (that conforms to https://github.com/kata-containers/documentation/blob/master/Documentation-Requirements.md) plus a nice juicy warning ;)