cmoulliard
commented
6 years ago Closed kadel closed 6 years ago
cmoulliard
commented
6 years ago cmoulliard
commented
6 years ago config file
When the initContainer will be started, then it will call a small go application to populate the content of the supervisord.conf using a list of CMDS passed as env var - see
cmoulliard
commented
6 years ago DeploymentConfig
Modifications to be done are available here
cmoulliard
commented
6 years ago Supervisor config
it is not needed to start supervisord with the run command but instead to register the program with the option autostart=false
[program:run-java]
command = /usr/local/s2i/run
autostart = false
stdout_logfile=/dev/stdout
stdout_events_enabled=true
[program:compile-java]
command = /usr/local/s2i/assemble
autostart = false
stdout_logfile=/dev/stdout
stdout_events_enabled=true
[inet_http_server]
port=localhost:9001Status
SB_POD=$(oc get pods -l app=spring-boot-supervisord -o name)
SERVICE_IP=$(minishift ip)
oc rsh $SB_POD /var/lib/supervisord/bin/supervisord ctl status
echo STOPPED
run-java STOPPED
compile-java STOPPED Next, when we call odo push, the code will copied or updated from the local file system of the developer to the pod and a rsh exec command executed to trigger one of the program
E.g
oc rsh $SB_POD /var/lib/supervisord/bin/supervisord ctl start run-java
run-java: started
oc rsh $SB_POD /var/lib/supervisord/bin/supervisord ctl start compile-java
compile-java: started
slemeur
commented
6 years ago @kadel : Should it be an epic?
jorgemoralespou
commented
6 years ago @kadel, are 2 initContainers required?
cmoulliard
commented
6 years ago @kadel @jorgemoralespou Another technical solution to be considered instead of managing the injection of the initContainer by odo is to delegate to k8s this responsability using a webhook
This is what servicemesh - istio does
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
name: istio-sidecar-injector
namespace: {{ .Release.Namespace }}
labels:
app: istio-sidecar-injector
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
webhooks:
- name: sidecar-injector.istio.io
clientConfig:
service:
name: istio-sidecar-injector
namespace: {{ .Release.Namespace }}
path: "/inject"
caBundle: ""
rules:
- operations: [ "CREATE" ]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]where pods are matched using this rule
namespaceSelector:
matchLabels:
istio-injection: enabledusing a configmap containing the definition about what should be changed/enriched
kind: ConfigMap
metadata:
name: istio-inject
namespace: {ISTIO_NAMESPACE}
apiVersion: v1
data:
config: |
policy: enabled
template: |-
initContainers:
...@cmoulliard this is the reason why OpenShift needs to be involved. Currently CRDs, Webhooks, etc.. need to be installed by cluster-admin. Also, having odo relying on a functionality that is not provided by the client or the server but that needs to be installed on the server is not an option, IMHO.
cmoulliard
commented
6 years ago Here is a first prototype using the supervisord's init container : https://github.com/cmoulliard/k8s-supervisor#instructions-to-inject-a-supervisords-initcontainer-and-enrich-the-deployment-of-a-spring-boot-s2i-application
[create.go](https://github.com/cmoulliard/k8s-supervisor#create-the-deploymentconfig-for-the-local-spring-boot-project)Remark : For a reason that I haven't yet identified, I have created a new s2i image to add such permissions as the by default was reporting a permission denied
...
USER root
RUN mkdir -p /tmp/src/target
RUN chgrp -R 0 /tmp/src/ && \
chmod -R g+rw /tmp/src/@jorgemoralespou @kadel
kadel
commented
6 years ago @kadel, are 2 initContainers required?
It is not, it can be done in one, you are right. The initContainer that we have right now just copies /opt/app-root from image to volume. This can be done in one initContainer together with injecting supervisor.
cmoulliard
commented
6 years ago initContainer
here is what I did to add supervisord as initcontainer and which is working. https://github.com/cmoulliard/k8s-supervisor/blob/master/pkg/buildpack/deploymentconfig.go#L142
kadel
commented
6 years ago @slemeur
Should it be an epic?
I've converted this to task and created new epic #588. This issue is now the first part of that epic.
cdrage
commented
6 years ago @kadel At the moment I'm learning / looking into the supervisord image.
Hopefully one of you can help me.
But from my understanding, we no longer need to deploy a BuildConfig to take the GitBuildSource and inject it into a new image (ImageStreamTag).
However, in your explanation, what do you mean by "application container"? Wouldn't we actually need BuildConfig in order to successfully rebuild with local / binary components?
Hopefully this explains it better, but here: https://github.com/redhat-developer/odo/blob/master/pkg/occlient/occlient.go#L738 what would we use instead if we're no longer building the initial image with BuildConfig? I'm a tad confused as to what image we would be pulling instead.
jorgemoralespou
commented
6 years ago @cdrage, maybe I can help in the meantime. The image to use in the DC will be the image for the component to deploy. As this image will most likely be a s2i image, it'll be able to build source code, as it'll have build tools in it and the assemble script. (e.g. wildfly, EAP, Java, Perl, python, nodes,...). There needs to be an initContainer (supervisor image) that will copy the supervisor to the component image and will move the sources to the PV. This needs to be a small image with the minimal stuff and it'll be the init Container added to every DC.
This last part was previously done by a build config, that built the supervisor into the component image. This is no longer needed, hence no need for a BC anymore.
Hope this helps.
cdrage
commented
6 years ago @jorgemoralespou @kadel
I'm getting a failed deployment as I'm nearing the completion of this implementation.
The problem I'm encountering is that I'm unable to use a S2I image within the component without having to build it.
I'm able to successfully inject the supervisord in to the image (at least). But the problem is starting the application container with S2I.
The problem is that odo create will create a failed deployment, as the main application container will encounter a CrashLoopBackoff with this in the logs:
github.com/redhat-developer/odo implement-supervisord ✗ 21h53m ⚑ ⍉
▶ oc logs po/foo-app-1-55lh5
This is a S2I nodejs-8 centos base image:
To use it, install S2I: https://github.com/openshift/source-to-image
Sample invocation:
s2i build https://github.com/sclorg/s2i-nodejs-container.git --context-dir=8/test/test-app/ /nodejs-8-centos7 nodejs-sample-app
You can then run the resulting image via:
docker run -p 8080:8080 nodejs-sample-appIt is only until we use odo push does this partially get resolved, as the files are copied over into the main application container.
I've yet to implement it yet, but I'm assuming: https://github.com/kadel/bootstrap-supervisored-s2i/blob/master/scripts/assemble-and-restart.sh#L10 will fix the issue. It will actually build the container for the first time when odo push is used.
How @cmoulliard get's around this is well... using a BuildConfig, which is exactly what we're trying to avoid using.
See:
Specifically:
var defaultImages = []types.Image{
*CreateTypeImage(true, "dev-s2i", "latest", "quay.io/snowdrop/spring-boot-s2i", false),
*CreateTypeImage(true, "copy-supervisord", "latest", "quay.io/snowdrop/supervisord", true),
}Before he deploys using DeploymentConfig.
I'll come up with some more ideas on how to get around this, but at the moment only a couple of things come to mind:
odo push is used)run command on the application container (endless echo or sleep(9999)) until odo push is ran. This keeps the container running and indicates that it's working but obviously nothing will happen until the user uses odo push.My current / awful code is located here: https://github.com/cdrage/odo/commit/a44ee3529d7ad3702f55c983c690cafc26b26d41
What do you think @cmoulliard @kadel @jorgemoralespou ?
kadel
commented
6 years ago @cdrage that is strange. Are you setting command in the "application" container?
Can you show me your DC definition?
I've just tried this: https://gist.github.com/kadel/e9fc6e47f2a05accf0306b4cb69e70c2 and it worked
kadel
commented
6 years ago Sorry, I've just noticed that you already included a link to your code.
I just did a quick check, and I confirmed my suspicion ;-) You are not overwriting default command from the image with supervisor binary.
cdrage
commented
6 years ago @kadel Ah, you're right.. Dumb mistake! Thanks for the insight.
jorgemoralespou
commented
6 years ago @cdrage also, worth mentioning that in some cases, the supervisord should not start the process automatically, as there are runtimes that will fall off the code/application is not there, e.g openjdk.
Charles has everything set as not start by default, I think we should eventually make this a config as depends on the runtime, but the easy one is not start by default.
Also, I would think that we should think about the following: When creating the components:
cdrage
commented
6 years ago @kadel @jorgemoralespou
So I'm getting an error with regards to mounting /opt/app-root early in the deployment process.
From debugging, it appears that we're mounting an empty container, causing OpenShift to fail:
Containers:
test-app:
Container ID: docker://f917265261a07a22f302568ed8574526869c1dccf9eb4207f311a372e6239706
Image: docker.io/centos/nodejs-8-centos7@sha256:f0da15b9859597156cba3e568bd495b261118a16e232e6cd4a0907c4587d4083
Image ID: docker-pullable://docker.io/centos/nodejs-8-centos7@sha256:f0da15b9859597156cba3e568bd495b261118a16e232e6cd4a0907c4587d4083
Port: 8080/TCP
Command:
/var/lib/supervisord/bin/supervisord
Args:
-c
/var/lib/supervisord/conf/supervisor.conf
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: ContainerCannotRun
Message: oci runtime error: container_linux.go:247: starting container process caused "chdir to cwd (\"/opt/app-root/src\") set in config.json failed: no such file or directory"
Exit Code: 128
Started: Mon, 13 Aug 2018 14:49:11 -0400
Finished: Mon, 13 Aug 2018 14:49:11 -0400
Ready: False
Restart Count: 6
Environment: <none>
Mounts:
/opt/app-root from test-app-s2idata (rw)
/var/lib/supervisord from shared-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-h48xx (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
shared-data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
test-app-s2idata:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: test-app-s2idata
ReadOnly: false
default-token-h48xx:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-h48xx
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: <none>
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
7m 7m 1 default-scheduler Normal Scheduled Successfully assigned test-app-1-l4zb8 to localhost
7m 7m 1 kubelet, localhost Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "pv0050"
7m 7m 1 kubelet, localhost Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "shared-data"
7m 7m 1 kubelet, localhost Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "default-token-h48xx"
7m 7m 1 kubelet, localhost spec.initContainers{copy-supervisord} Normal Pulling pulling image "docker.io/cdrage/supervisord-test:latest"
7m 7m 1 kubelet, localhost spec.initContainers{copy-supervisord} Normal Pulled Successfully pulled image "docker.io/cdrage/supervisord-test:latest"
7m 7m 1 kubelet, localhost spec.initContainers{copy-supervisord} Normal Started Started container
7m 7m 1 kubelet, localhost spec.initContainers{copy-supervisord} Normal Created Created container
7m 6m 4 kubelet, localhost spec.containers{test-app} Normal Pulling pulling image "docker.io/centos/nodejs-8-centos7@sha256:f0da15b9859597156cba3e568bd495b261118a16e232e6cd4a0907c4587d4083"
6m 6m 4 kubelet, localhost spec.containers{test-app} Normal Pulled Successfully pulled image "docker.io/centos/nodejs-8-centos7@sha256:f0da15b9859597156cba3e568bd495b261118a16e232e6cd4a0907c4587d4083"
6m 6m 4 kubelet, localhost spec.containers{test-app} Normal Created Created container
6m 6m 4 kubelet, localhost spec.containers{test-app} Warning Failed Error: failed to start container "test-app": Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "chdir to cwd (\"/opt/app-root/src\") set in config.json failed: no suc
h file or directory"
6m 1m 20 kubelet, localhost spec.containers{test-app} Warning BackOff Back-off restarting failed containerShould we only add this volume to the DC / Pod when doing odo push?
cdrage
commented
6 years ago @cmoulliard
Maybe I'm doing something wrong, but how are you using the s2i run scripts here: https://github.com/snowdrop/k8s-supervisor/blob/master/pkg/buildpack/deploymentconfig.go#L243 ?
Despite me using the official images (nodejs). I'm unable to find the actual s2i directory:
github.com/redhat-developer/odo implement-supervisord ✗ 3d ⚑ ◒ ⍉
▶ oc exec foo-app-1-qbv7f ls /usr/local
bin
etc
games
include
lib
lib64
libexec
sbin
share
srcDespite using the nodejs image:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
8m 8m 1 default-scheduler Normal Scheduled Successfully assigned foo-app-1-qbv7f to localhost
8m 8m 1 kubelet, localhost Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "pv0024"
8m 8m 1 kubelet, localhost Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "shared-data"
8m 8m 1 kubelet, localhost Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "default-token-h48xx"
8m 8m 1 kubelet, localhost spec.initContainers{copy-supervisord} Normal Pulling pulling image "docker.io/cdrage/supervisord-test:latest"
8m 8m 1 kubelet, localhost spec.initContainers{copy-supervisord} Normal Pulled Successfully pulled image "docker.io/cdrage/supervisord-test:latest"
8m 8m 1 kubelet, localhost spec.initContainers{copy-supervisord} Normal Created Created container
8m 8m 1 kubelet, localhost spec.initContainers{copy-supervisord} Normal Started Started container
8m 8m 1 kubelet, localhost spec.containers{foo-app} Normal Pulling pulling image "docker.io/centos/nodejs-8-centos7@sha256:f0da15b9859597156cba3e568bd495b261118a16e232e6cd4a0907c4587d4083"
7m 7m 1 kubelet, localhost spec.containers{foo-app} Normal Pulled Successfully pulled image "docker.io/centos/nodejs-8-centos7@sha256:f0da15b9859597156cba3e568bd495b261118a16e232e6cd4a0907c4587d4083"
7m 7m 1 kubelet, localhost spec.containers{foo-app} Normal Created Created container
7m 7m 1 kubelet, localhost spec.containers{foo-app} Normal Started Started container@cdrage your error with the empty directory used in the config.json might be because the initContainer might not create that directory in the volume as part of the copy-files, hence when the main container starts that directory does not exist. My guess would be that you need to manually create that directory in the copy-files action to prevent this error.
https://github.com/openshift-evangelists/pseudo-vps-quickstart/blob/master/.s2i/bin/assemble#L82-L84
@cdrage For your second question, s2i has a convention on how to denote where the s2i sccripts should be. AFAIK it used to be an ENV (STI_SCRIPTS_URL=image:///usr/libexec/s2i) and now it's a label in the container. Read the docs here: https://github.com/openshift/source-to-image/blob/master/docs/builder_image.md#s2i-scripts
In case any doubt, you can tag bparees on GH issues, as he's the father of the feature.
kadel
commented
6 years ago I've replied at #638 but we should probably join those discussions into one.
https://github.com/redhat-developer/odo/issues/638#issuecomment-415499870
Problem
Right new we are using BuildConfig and https://github.com/kadel/bootstrap-supervisored-s2i to inject supervisor to s2i image. This process is slow and creates unnecessary BuildConfig.
Solution
Use initContainer and Go implementation of supervisor (https://github.com/ochinchina/supervisord)
InitContainer will copy supervisor binary with config file into the shared volume. This volume is shared between initContainer and application container.
You can see an example of initContainer and volumes approach at this small PoC - https://gist.github.com/kadel/35fa7c64c79a845edfe85ada2706019b
1) Create container image Image should include:
run.shscript. Something similar to current run.confutility scripts that we are already using in our current approach https://github.com/kadel/bootstrap-supervisored-s2i/tree/master/scripts Some of the scripts might need to be updated Image should be as small as possible Push that image somewhere where can be accessed by anyone (for example quay.io/odo/....)
Most of this work has already been done at https://github.com/snowdrop/k8s-supervisor/tree/master/supervisord. We should reuse that. It also includes small Go program to generate supervisord config file according to env parameters.
2) update BootstrapSupervisoredS2I
copy-files-to-volumescript (this is what it is currently doing)3) make necessary changes to
component.Updatefunction to makeodo updatework with new implementationChange will impact local and binary components, git component will remain the same (with BuildConfig).