Projects -> Sign In from Console fails - following CD instructions

[stevef1uk]

Following: http://fabric8.io/guide/cdelivery.html

I am fine up to:

Now in the fabric8 console if you click the Projects tab the Repositories sub tab should be available. This prompts you to login to gogs with the user and password (until we can get single sign on working with gogs and OpenShift)

I am unable to login using the gogsadmin RedHat$1 details. In the gogs web console I created a new user for me and created a repository for myself and tried that as the login details in the Projects Sign In link also without success.

[christian-posta]

Yah sometimes there's a race in the ordering of how things get deployed; ie, fabric8-forge app depends on a service from gogs, so will not deploy correctly in that case and you'd see that error you have. Solution is the always install gogs first, then fabric8-forge. If you have them installed already, just kill the fabric8-forge pod and let the replication controller recreate it. As long as gogs service is available, recreating the fabric8-forge pod should do the trick.

On Sun, Jun 21, 2015 at 10:19 AM, stevef1uk notifications@github.com wrote:

Following: http://fabric8.io/guide/cdelivery.html

I am fine up to:

Now in the fabric8 console if you click the Projects tab the Repositories sub tab should be available. This prompts you to login to gogs with the user and password (until we can get single sign on working with gogs and OpenShift)

I am unable to login using the gogsadmin RedHat$1 details. In the gogs web console I created a new user for me and created a repository for myself and tried that as the login details in the Projects Sign In link also without success.

— Reply to this email directly or view it on GitHub https://github.com/fabric8io/fabric8/issues/4288.

Christian Posta twitter: @christianposta http://www.christianposta.com/blog http://fabric8.io

[stevef1uk]

Thanks for the info. I discovered that fabric8-forge app was not deployed so I ran the cdelivery from the Apps -> Run menu. When running 'oc get pods' I am seeing:

letschat-csw65 0/2 Error pulling image (latest) from docker.io/mongo, Driver devicemapper failed to create image rootfs bb2fbb9da757e79f5f81ee609faf57dbc2a1945e733c0ef2cd6472939812987d: Unknown device 065218d54d7d731858c4db4f5dabca651149c9d39059735f03aad43af021da9a 3 20m

Anything to worry about?

[rawlingsj]

We've seen a couple of issues pulling docker images but the error here does look different to the other recent one https://github.com/fabric8io/fabric8/issues/4275

Myself I've run the CD app 10 or 15 times over the last couple of days and I've had an issue pulling images once.

The mongodb image that is being pulled is the official one and a quick bit of googling suggests the error is a race condition with and the devicemapper backend.

You could try to do a manual docker pull mongoafter either ssh'ing into the VM or using the expose docker daemon from the VM..

export DOCKER_HOST=tcp://vagrant.local:2375 && docker pull mongo

or

vagrant ssh
docker pull mongo

Either way to answer your question Anything to worry about? yes if you want to use chat, hubot and see the build notifications, no if not.

I think OpenShift or Kubernetes is either garbage collecting failed pods or backing off scheduling them as that used to cause thousands of pods being scheduled if gone unnoticed, this would cause high CPU and sometimes loose connectivity to the VM. Like I said though I 'think' this is better since 0.6.2 of OpenShift but that would be the only other thing to worry about.

My guess is if you still have problems pulling the image then the filesystem that docker is using is a bit messed up. Short term fix would be to do a vagrant destroy and vagrant up, long term I guess we need to work out if devicemapper is the best for us or if we should look at others.

[stevef1uk]

Well, I deleted my vagrant image and tried again and now I have 3 Error pulling image messages. I haven't used device mapper as a back-end to docker before, but it doesn't appear too stable for me.

[jimmidyson]

Device mapper is the default storage engine for docker because it requires no extra set up to use, other storage engines do. If you install & run docker on fedora/centos with no other configuration then you are using device mapper. It does have some problems, that much is recognised, & it's definitely not fit for production use. The problem is distributing a vagrant box with any other storage engine enabled is pretty tricky.

I wonder if the problem is to do with the version of docker we're running? We've had to use a koji build (pre-release) to work around some issues with openshift & previous docker versions. I'll try to upgrade docker to 1.7 which I hope it's available in testing repos for centos now (it or for fedora).

On 23:21, Sun, 21 Jun 2015 stevef1uk notifications@github.com wrote:

Well, I deleted my vagrant image and tried again and now I have 3 Error pulling image messages. I haven't used device mapper as a back-end to docker before, but it doesn't appear too stable for me.

— Reply to this email directly or view it on GitHub https://github.com/fabric8io/fabric8/issues/4288#issuecomment-113964729.

[stevef1uk]

My experience was with building docker with aufs and I haven't seen these problems. I have just retried from scratch and have the same 3 images errors:

elasticsearch-a4xlx 1/2 could not find image: no such id: 20b42b81932f418db353cab1942be9dbff12eec5468cd843435d5b312a0d6d59
fabric8-forge-tra97 0/1 Error pulling image (2.2.1) from docker.io/fabric8/fabric8-forge, Error mounting '/dev/mapper/docker-253:0-67168046-df2a0347c9d081fa05ecb83669dcae5830c67b0676a6d6358218e55d8a45969c' on '/var/lib/docker/devicemapper/mnt/df2a0347c9d081fa05ecb83669dcae5830c67b0676a6d6358218e55d8a45969c': invalid argument 0 24m hubot-notifier-o2wn9 0/1 Error pulling image (2.2.1) from docker.io/fabric8/hubot-notifier, Error mounting '/dev/mapper/docker-253:0-67168046-df2a0347c9d081fa05ecb83669dcae5830c67b0676a6d6358218e55d8a45969c' on '/var/lib/docker/devicemapper/mnt/df2a0347c9d081fa05ecb83669dcae5830c67b0676a6d6358218e55d8a45969c': invalid argument 0 24m

I can't manually pull the images because something is already trying to download them. Is there a simple command to stop fabric8 so that I can manually pull the images & restart.

On the start-up order issue surely there is a configuration option to resolve this?

Regards

Steve

[rawlingsj]

You could restart the docker service..

systemctl restart docker

fwiw below is a list of images that make up the cd app..

docker pull fabric8/elasticsearch-k8s:1.5.2
docker pull fabric8/elasticsearch-logstash-template:2.2.1
docker pull fabric8/fabric8-console
docker pull fabric8/fabric8-forge:2.2.1
docker pull fabric8/gerrit
docker pull fabric8/gogs
docker pull fabric8/hubot-notifier:2.2.1
docker pull fabric8/jenkins
docker pull fabric8/kibana4:4.0.2
docker pull mongo
docker pull fabric8/lets-chat:latest
docker pull fabric8/nexus
docker pull fabric8/eclipse-orion
docker pull tpires/sonar-server
docker pull tpires/sonar-mysql
docker pull postgres
docker pull rawlingsj/taiga-back:test
docker pull rawlingsj/taiga-front-dist:test

Thanks to @paoloantinori for the ninja grep command..

curl http://repo1.maven.org/maven2/io/fabric8/apps/cdelivery/2.2.1/cdelivery-2.2.1-kubernetes.json | grep -Po '"image" :.?[^]",' | grep -Po '(?<=image" : ").(?=",)' | xargs -n1 -ITOKEN echo "docker pull TOKEN"

If this is useful maybe we could bake shell scripts into the vagrant image to prime a local docker registry before running an app for the first time? Might help avoid these devicemapper issues.. although of course it might not.

[rawlingsj]

I think there was a formatting issue in the comment above.. here it is again.

curl http://repo1.maven.org/maven2/io/fabric8/apps/cdelivery/2.2.1/cdelivery-2.2.1-kubernetes.json | grep -Po '"image" :.*?[^\\]",' | grep -Po '(?<=image" : ").*(?=",)' | xargs -n1 -ITOKEN echo "docker pull TOKEN"

[davsclaus]

yeah wonder if we can have a flag to do this, as otherwise pulling all those images may take a long time and i would also like to shown ppl that starting fabric8 is fast.

[rawlingsj]

ah yeah. although the script above will only print the images to pull for an app group it wont do the actual docker pull. might be useful if a user wants to run it and pull the required images first.

[stevef1uk]

Nice ninja script indeed. However, restarting docker doesn't cure the root problem as the containers are restarted as well so something is already attempting to pull the docker images and sadly oc get pods shows the same errors as before.

[rawlingsj]

Ah ok you could delete all the fabric8 RC's pods and services using..

oc delete all -l provider=fabric8

from memory there are a couple of RC's that dont have that label yet so you might need to manually delete them oc delete rc taiag

[stevef1uk]

Right. I decided to start again:

1. vagrant destory
2. vagrant up
3. vagrant ssh

docker pull fabric8/elasticsearch-k8s:1.5.2 docker pull fabric8/elasticsearch-logstash-template:2.2.1 docker pull fabric8/fabric8-console docker pull fabric8/fabric8-forge:2.2.1 docker pull fabric8/gerrit docker pull fabric8/gogs docker pull fabric8/hubot-notifier:2.2.1 docker pull fabric8/jenkins docker pull fabric8/kibana4:4.0.2 docker pull mongo docker pull fabric8/lets-chat:latest docker pull fabric8/nexus docker pull fabric8/eclipse-orion docker pull tpires/sonar-server docker pull tpires/sonar-mysql docker pull postgres docker pull rawlingsj/taiga-back:test docker pull rawlingsj/taiga-front-dist:test

No errors encountered by running the pulls serially :-)

1. Hawtio console (http://fabric8.vagrant.local) as namespace = default, Apps -> Run cdelivery

Then everything is showing up as running and I hope to be ok now.

Thanks for your help.

[stevef1uk]

Ok some final comments on this from me.

1. After pulling the docker images I used the vagrant snapshot plugin to save the state.
2. I have not been able to complete the demo of creating a project and watching it build with Jenkins as I have experienced my MBA crashing (the VM is using a lot of CPU & memory) and I have seen the jerkins jobs seem to pause before a 504 Gateway timeout error.
3. I was unable to configure the Chat functionality using the oc route as the room Id wasn't recognised. I tried various flavours here.
4. After a docker restart I found I had lost the repository

What is the approach to saving your work and continuing going forward?

[christian-posta]

On Mon, Jun 22, 2015 at 8:40 AM, stevef1uk notifications@github.com wrote:

Ok some final comments on this from me.

1. After pulling the docker images I used the vagrant snapshot plugin to save the state.
2. I have not been able to complete the demo of creating a project and watching it build with Jenkins as I have experienced by MBA crashing and I have seen the jerkins jobs seem to pause before a 504 Gateway timeout error.
3. I was unable to configure the Chat functionality using the oc route as the room Id wasn't recognised. I tried various flavours here.
4. After a docker restart I found I had lost the repository

What is the approach to saving your work and continuing going forward?

we are going to work on persisting the state of jenkins/gerrit/gogs/taiga, etc into PersistentVolumes so that pods can retain some of their state even across restarts. As of know, each time you recreate the pods, you'll have to re-create/re-import your projects into Gogs.

— Reply to this email directly or view it on GitHub https://github.com/fabric8io/fabric8/issues/4288#issuecomment-114157219.

Christian Posta twitter: @christianposta http://www.christianposta.com/blog http://fabric8.io