search

fabric8io / fabric8

fabric8 is an open source microservices platform based on Docker, Kubernetes and Jenkins
http://fabric8.io/
1.76k stars 504 forks source link

Jenkins builds fail in multi-node clusters #6882

Open pdudits opened 7 years ago

pdudits commented 7 years ago

Out of the box, Fabric8 builds do not work on multi-node cluster due to default, insecure, configuration of fabric8-docker-registry.

I installed fabric8 on two-node cluster using gofabric8 on Azure ACS. However, the sample applications (in this case a .war project) fail to build:

[INFO] F8> Running generator webapp

[INFO] F8> webapp: Using fabric8/tomcat-8:1.1.5 as base image for webapp

[INFO] F8> The push refers to a repository [10.0.223.76:80/example/wa1]

[ERROR] F8> Unable to push 'example/wa1:1.0.1' from registry '10.0.223.76:80'

[ERROR] Failed to execute goal io.fabric8:fabric8-maven-plugin:3.2.9:push (default-cli) on project wa1: Unable to push 'example/wa1:1.0.1' from registry '10.0.223.76:80': Get https://10.0.223.76:80/v1/_ping: http: server gave HTTP response to HTTPS client

The plugin offers no way of forcing insecure connection, nor the registry offers configuration for configuring TLS.

I tried to replace that with registry proxy addon. It is possible then to change the pipeline to not use the service directly.

The build will pass, however the integration test fails:

Initializing Session:633f2e47-c6a5-4670-94a9-676ca5db8682
Using Kubernetes at: https://kubernetes.default/
Creating kubernetes resources inside namespace: default-testing

if you use OpenShift then type this switch namespaces: oc project default-testing
if you use kubernetes then type this to switch namespaces: kubectl namespace default-testing

Applying kubernetes configuration from: file:/home/jenkins/workspace/wa1/target/classes/META-INF/fabric8/kubernetes.yml

Adapting resources to pull images from registry: 10.0.223.76:80
Applying service:wa1
Applying Deployment.
Waiting for container:webapp. Reason:ErrImagePull
Waiting for container:webapp. Reason:ErrImagePull
Waiting for container:webapp. Reason:ImagePullBackOff
Waiting for container:webapp. Reason:ImagePullBackOff
...

SessionListener follows the same logic as the pipeline, and will use the container service.

pdudits commented 7 years ago

So the answer appears to be that Azure, contrary to GKE or OpenShift does not have insecure registries configured at 10.0.0.0/8.

Adding that setting to /etc/docker/daemon.json of all nodes will make things work.