abandoned/outdated k8s manifests

aleks-mariusz commented 6 years ago

per the readme, i ran kubectl create -f ./k8s after git cloning the repo.. the result:

NAMESPACE                     NAME                                   READY     STATUS             RESTARTS   AGE       IP              NODE
default                       alertmanager-cbf7cf875-qrfzx           0/1       ImagePullBackOff   0          4m        10.32.0.12      k8lab1bs
default                       configs-d9474b5bd-bvf2p                0/1       ImagePullBackOff   0          4m        10.32.0.13      k8lab1bs
default                       configs-db-75c6b4df96-n46l6            1/1       Running            0          4m        10.36.0.10      k8lab3bs
default                       consul-744475fd58-mw6lz                1/1       Running            0          4m        10.44.0.9       k8lab2bs
default                       distributor-885c5c85c-csd98            0/1       ImagePullBackOff   0          4m        10.32.0.11      k8lab1bs
default                       dynamodb-868489b88c-tbrgz              1/1       Running            0          4m        10.44.0.10      k8lab2bs
default                       ingester-84fc6c7947-vb2dv              0/1       ImagePullBackOff   0          4m        10.32.0.16      k8lab1bs
default                       memcached-cf8545f6d-rqdvd              1/1       Running            0          4m        10.44.0.11      k8lab2bs
default                       nginx-55b9f744d5-mhjml                 0/1       CrashLoopBackOff   4          4m        10.44.0.12      k8lab2bs
default                       querier-6d484876c8-8v2fc               0/1       ErrImagePull       0          4m        10.32.0.14      k8lab1bs
default                       retrieval-6d849ccb59-slplc             1/1       Running            0          4m        10.36.0.12      k8lab3bs
default                       ruler-795f8c59c7-xh55m                 0/1       ImagePullBackOff   0          4m        10.44.0.13      k8lab2bs
default                       s3-79655d5455-9q5xl                    1/1       Running            0          4m        10.32.0.15      k8lab1bs
default                       table-manager-797c5df74b-snwc5         0/1       ImagePullBackOff   0          4m        10.36.0.13      k8lab3bs

seems most of these manifests weren't updated in a year or two, they have gone out of date resulting in a lot of errors even pulling the images, and one of them, the nginx container has the following error:

$ kubectl logs nginx-55b9f744d5-mhjml
2018/07/31 09:59:27 [emerg] 1#1: open() "/etc/nginx/nginx.pid" failed (30: Read-only file system)
nginx: [emerg] open() "/etc/nginx/nginx.pid" failed (30: Read-only file system)

seems these have been abandoned ? i'm using k8s v1.11.1 and pulling from the master branch of weaveworks/cortex

bboreham commented 6 years ago

Did you build the images?

aleks-mariusz commented 6 years ago

negative.. i was trying to get it up and running on a lab k8s cluster w/ the pre-built images (i presumed that's what those manifests use, as that's how most projects are set up)

aleks-mariusz commented 6 years ago

ok so i tried to build this myself and ran into this now:

time docker run --rm --tty -i \
        -v /home/username/k/cortex/.cache:/go/cache \
        -v /home/username/k/cortex/.pkg:/go/pkg \
        -v /home/username/k/cortex:/go/src/github.com/weaveworks/cortex \
        quay.io/weaveworks/cortex-build-image pkg/ingester/client/cortex.pb.go;
make: Entering directory '/go/src/github.com/weaveworks/cortex'
make: *** No rule to make target 'pkg/ingester/client/cortex.pb.go'.  Stop.
make: Leaving directory '/go/src/github.com/weaveworks/cortex'

real    0m0.422s
user    0m0.000s
sys     0m0.016s
make: *** [pkg/ingester/client/cortex.pb.go] Error 2

bboreham commented 6 years ago

The first thing the Makefile should do is build the build image.

Can you post the full output of make please? Try make clean first.

aleks-mariusz commented 6 years ago

I have uploaded the full output of a clean make session:

make.log

Also, per the convo in the weave user slack #general channel, i realize that the manifests rely on the latest tag which don't exist in the repo (which causes a lot of the images to fail to pull, the original point of opening this issue), so i came up with the following one-liner to update the manifests to set the tag for the latest master commit (what the actual latest tag should be):

sed -i -e 's/image: quay.io\/weaveworks\/cortex-\(.*\)/&:master-'$(\
  curl -L https://github.com/weaveworks/cortex/commits/master 2>/dev/null|\
  grep -A 3 'repository-content'|awk -F / '/Permalink/ {print substr($7,0,8)}'\
  )/ ./k8s/*yaml

The above would be ran from the top-level github-clone'd directory and would update all the manifests with cortex specific container images to add the tag that references the latest master commit (under the presumption that such a tagged image would exist at the docker container registry, quay.io, that weaveworks uses).

But if you guys can also help me resolve the above build issues, that would be great too! :-)

bboreham commented 6 years ago

The point where your build fails is:

...
Successfully built 46e9db8f3f77
docker tag quay.io/weaveworks/cortex-build-image quay.io/weaveworks/cortex-build-image:master-6f0118a
touch build-image/.uptodate
time docker run --rm --tty -i \
        -v /home/username/k/cortex/.cache:/go/cache \
        -v /home/username/k/cortex/.pkg:/go/pkg \
        -v /home/username/k/cortex:/go/src/github.com/weaveworks/cortex \
        quay.io/weaveworks/cortex-build-image pkg/ingester/client/cortex.pb.go;
make: Entering directory '/go/src/github.com/weaveworks/cortex'
make: *** No rule to make target 'pkg/ingester/client/cortex.pb.go'.  Stop.

when I run it, it executes the protobufs compiler:

...
Successfully built c62909fb3b02
Successfully tagged quay.io/weaveworks/cortex-build-image:latest
docker tag quay.io/weaveworks/cortex-build-image quay.io/weaveworks/cortex-build-image:master-6f0118a0
touch build-image/.uptodate
time docker run --rm --tty -i \
    -v /home/vagrant/src/github.com/weaveworks/cortex/.cache:/go/cache \
    -v /home/vagrant/src/github.com/weaveworks/cortex/.pkg:/go/pkg \
    -v /home/vagrant/src/github.com/weaveworks/cortex:/go/src/github.com/weaveworks/cortex \
    quay.io/weaveworks/cortex-build-image pkg/ring/ring.pb.go;
make: Entering directory '/go/src/github.com/weaveworks/cortex'
protoc -I ./vendor:./pkg/ring --gogoslick_out=plugins=grpc:./pkg/ring ./pkg/ring/ring.proto
make: Leaving directory '/go/src/github.com/weaveworks/cortex'
0.00user 0.00system 0:00.55elapsed 1%CPU (0avgtext+0avgdata 19016maxresident)k
0inputs+0outputs (0major+1657minor)pagefaults 0swaps
time docker run --rm --tty -i \
    -v /home/vagrant/src/github.com/weaveworks/cortex/.cache:/go/cache \
    -v /home/vagrant/src/github.com/weaveworks/cortex/.pkg:/go/pkg \
    -v /home/vagrant/src/github.com/weaveworks/cortex:/go/src/github.com/weaveworks/cortex \
    quay.io/weaveworks/cortex-build-image pkg/ingester/client/cortex.pb.go;
make: Entering directory '/go/src/github.com/weaveworks/cortex'
protoc -I ./vendor:./pkg/ingester/client --gogoslick_out=plugins=grpc:./pkg/ingester/client ./pkg/ingester/client/cortex.proto
make: Leaving directory '/go/src/github.com/weaveworks/cortex'
...

I don't understand why the error is "No rule to make" - the rule %.pb.go: is on line 114.

Can you give some more details of your environment - Linux version, Docker version, etc.?

aleks-mariusz commented 6 years ago

on the docker-client side (where i am running make), i am using CentOS 7.5 under WSL (Windows 10 Enterprise, Version 1803, build 17134.165), on the docker-server side, it's also CentOS 7.5 on bare-metal. Here's the output of docker version:

Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-68.gitdded712.el7.centos.x86_64
 Go version:      go1.9.4
 Git commit:      dded712/1.13.1
 Built:           Tue Jul 17 18:34:48 2018
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-68.gitdded712.el7.centos.x86_64
 Go version:      go1.9.4
 Git commit:      dded712/1.13.1
 Built:           Tue Jul 17 18:34:48 2018
 OS/Arch:         linux/amd64
 Experimental:    false

bboreham commented 6 years ago

The build process assumes that -v will mount files from where make is running to where Docker is running. That assumption probably falls down if you run them on two different machines.

aleks-mariusz commented 6 years ago

makes sense, to give a try what you mentioned, i moved to build the images onto a system where the docker client and server is the same (so no longer WSL). this succeeded fine, so it was using docker client on a system different than where the docker service ran that caused the issue as you said. as a note, i only use a remote docker service because the kernel under wsl doesn't have enough syscalls yet implemented to actually have a docker server running there :-/

so that solves the build issue component of this issue, and coupled with my one-liner from https://github.com/weaveworks/cortex/issues/911#issuecomment-409880324 this case can be closed

cortexproject / cortex

abandoned/outdated k8s manifests #911