rromannissen / nifi-openshift

8 stars 16 forks source link

Had to fix the given DeploymentConfig #4

Closed keremceliker closed 4 years ago

keremceliker commented 4 years ago

Could you share what you did as Fix in Deployment Config on it ?

Had to fix the given DeploymentConfig

Originally posted by @wilddeuces in https://github.com/rromannissen/nifi-openshift/issues/2#issuecomment-527427193

wilddeuces commented 4 years ago

Hey,

I cannot remember exactly what the issue was but I can share you my current DC & BC. I am running it with the default ServiceAccount which has the permission to create privileged pods

   apiVersion: build.openshift.io/v1
   metadata:
     labels:
       app: ${APPLICATION_NAME}
     name: ${APPLICATION_NAME}
   spec:
     nodeSelector:
     output:
       to:
         kind: ImageStreamTag
         name: ${APPLICATION_NAME}:latest
     runPolicy: Serial
     source:
       binary: {}
       type: Binary
     strategy:
       dockerStrategy: {}
   apiVersion: apps.openshift.io/v1
   metadata:
     name: ${APPLICATION_NAME}
     labels:
       app: ${APPLICATION_NAME}
   spec:
     strategy:
       type: Rolling
     triggers:
     - type: ImageChange
       imageChangeParams:
         automatic: true
         containerNames:
         - ${APPLICATION_NAME}
         from:
           kind: ImageStream
           name: ${APPLICATION_NAME}:latest
     - type: ConfigChange
     replicas: 1
     selector:
       deploymentConfig: ${APPLICATION_NAME}
     template:
       metadata:
         name: ${APPLICATION_NAME}
         labels:
           deploymentConfig: ${APPLICATION_NAME}
           app: ${APPLICATION_NAME}
       spec:
         terminationGracePeriodSeconds: 60
         serviceAccountName: default
         securityContext:
           runAsUser: 0
         containers:
         - name: ${APPLICATION_NAME}
           image: ${APPLICATION_NAME}:latest
           imagePullPolicy: Always
           ports:
           - name: http
             containerPort: 8080
             protocol: TCP
           env:
           - name: "NIFI_JAVA_XMS"
             value: ${NIFI_JAVA_XMS}
           - name: "NIFI_JAVA_XMX"
             value: ${NIFI_JAVA_XMX}
           - name: "NIFI_TIMEZONE"
             value: ${NIFI_TIMEZONE}
           - name: "ENVIRONMENT"
             value: ${TARGET_ENVIRONMENT}
           - name: "HOME"
             value: /tmp
           resources:
             limits:
               cpu: ${CPU_LIMIT}
               memory: ${MEMORY_LIMIT}
             requests:
               cpu: ${CPU_REQUEST}
               memory: ${MEMORY_REQUEST}
           volumeMounts: 
         volumes:
keremceliker commented 4 years ago

Hey Mate, i tried the build it on MiniShift and i got the follows error in POD now while pod creation processing.

Any idea on it to fix ?

container_linux.go:235: starting container process caused "exec: \"../scripts/start.sh\": permission denied"
--

Kerem Çeliker

wilddeuces commented 4 years ago

Hey Mate, i tried the build it on MiniShift and i got the follows error in POD now while pod creation processing.

Any idea on it to fix ?

container_linux.go:235: starting container process caused "exec: \"../scripts/start.sh\": permission denied"
--

Hey,

im not at home atm. That error states you need to chmod that file in the Dockerfile for ease you cam use „chmod 777 /scriptdir/start.sh“

Cheers

keremceliker commented 4 years ago

Hey Mate, I just added as you mentioned chmod 777.. in Dockerfile but im still getting error while im build the code.

Error code the follows below;


chmod: cannot access '/scripts/start.sh': No such file or directory
Removing intermediate container 0fd3ad71a841
The command '/bin/sh -c chmod 777 /scripts/start.sh' returned a non-zero code: 1

Could you edit for me mate, according to what I share below ?

Here is the Currently Dockerfile:


ADD sh/ ${NIFI_BASE_DIR}/scripts/

# Setup NiFi user and create necessary directories
RUN mkdir -p ${NIFI_BASE_DIR} \
    && apt-get update \
    && apt-get install -y jq xmlstarlet procps

# Download, validate, and expand Apache NiFi Toolkit binary.
RUN curl -fSL ${MIRROR_BASE_URL}/${NIFI_TOOLKIT_BINARY_PATH} -o ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip \
    && echo "$(curl ${BASE_URL}/${NIFI_TOOLKIT_BINARY_PATH}.sha256) *${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip" | sha256sum -c - \
    && unzip ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip -d ${NIFI_BASE_DIR} \
    && rm ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip \
    && mv ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION} ${NIFI_TOOLKIT_HOME} \
    && ln -s ${NIFI_TOOLKIT_HOME} ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION} \
    && chmod -R g+rwX ${NIFI_TOOLKIT_HOME}

# Download, validate, and expand Apache NiFi binary.
RUN curl -fSL ${MIRROR_BASE_URL}/${NIFI_BINARY_PATH} -o ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip \
    && echo "$(curl ${BASE_URL}/${NIFI_BINARY_PATH}.sha256) *${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip" | sha256sum -c - \
    && unzip ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip -d ${NIFI_BASE_DIR} \
    && rm ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip \
    && mv ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION} ${NIFI_HOME} \
    && mkdir -p ${NIFI_HOME}/conf \
    && mkdir -p ${NIFI_HOME}/database_repository \
    && mkdir -p ${NIFI_HOME}/flowfile_repository \
    && mkdir -p ${NIFI_HOME}/content_repository \
    && mkdir -p ${NIFI_HOME}/provenance_repository \
    && mkdir -p ${NIFI_HOME}/state \
    && mkdir -p ${NIFI_LOG_DIR} \
    && ln -s ${NIFI_HOME} ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION} \
    && chmod -R g+rwX ${NIFI_HOME}

ADD bootstrap.conf ${NIFI_HOME}/conf/bootstrap.conf

# Clear nifi-env.sh in favour of configuring all environment variables in the Dockerfile
RUN echo "#!/bin/sh\n" > ${NIFI_HOME}/bin/nifi-env.sh

# Web HTTP(s) & Socket Site-to-Site Ports
EXPOSE 8080 8443 10000

WORKDIR ${NIFI_HOME}

USER 1001

# Apply configuration and start NiFi
#
# We need to use the exec form to avoid running our command in a subshell and omitting signals,
# thus being unable to shut down gracefully:
# https://docs.docker.com/engine/reference/builder/#entrypoint
#
# Also we need to use relative path, because the exec form does not invoke a command shell,
# thus normal shell processing does not happen:
# https://docs.docker.com/engine/reference/builder/#exec-form-entrypoint-example
RUN chmod 777 start.sh
ENTRYPOINT ["../scripts/start.sh"]

Kerem Çeliker

wilddeuces commented 4 years ago

You did not specify the correct path for the start.sh script Its more like:


ARG NIFI_VERSION=1.9.2
ARG BASE_URL=https://archive.apache.org/dist
ARG MIRROR_BASE_URL=${MIRROR_BASE_URL:-${BASE_URL}}
ARG NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
ARG NIFI_TOOLKIT_BINARY_PATH=${NIFI_TOOLKIT_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-toolkit-${NIFI_VERSION}-bin.zip}

ENV NIFI_BASE_DIR=/opt/nifi
ENV NIFI_HOME ${NIFI_BASE_DIR}/nifi-current
ENV NIFI_TOOLKIT_HOME ${NIFI_BASE_DIR}/nifi-toolkit-current

ENV NIFI_PID_DIR=${NIFI_HOME}/run
ENV NIFI_LOG_DIR=${NIFI_HOME}/logs

USER root

ADD sh/ ${NIFI_BASE_DIR}/scripts/

# Setup NiFi user and create necessary directories
RUN mkdir -p ${NIFI_BASE_DIR} \
    && apt-get update \
    && apt-get install -y jq xmlstarlet procps python3

# Download, validate, and expand Apache NiFi Toolkit binary.
RUN curl -fSL ${MIRROR_BASE_URL}/${NIFI_TOOLKIT_BINARY_PATH} -o ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip \
    && echo "$(curl ${BASE_URL}/${NIFI_TOOLKIT_BINARY_PATH}.sha256) *${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip" | sha256sum -c - \
    && unzip ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip -d ${NIFI_BASE_DIR} \
    && rm ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip \
    && mv ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION} ${NIFI_TOOLKIT_HOME} \
    && ln -s ${NIFI_TOOLKIT_HOME} ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION} \
    && chmod -R g+rwX ${NIFI_TOOLKIT_HOME}

# Download, validate, and expand Apache NiFi binary.
RUN curl -fSL ${MIRROR_BASE_URL}/${NIFI_BINARY_PATH} -o ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip \
    && echo "$(curl ${BASE_URL}/${NIFI_BINARY_PATH}.sha256) *${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip" | sha256sum -c - \
    && unzip ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip -d ${NIFI_BASE_DIR} \
    && rm ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip \
    && mv ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION} ${NIFI_HOME} \
    && mkdir -p ${NIFI_HOME}/conf \
    && mkdir -p ${NIFI_HOME}/database_repository \
    && mkdir -p ${NIFI_HOME}/flowfile_repository \
    && mkdir -p ${NIFI_HOME}/content_repository \
    && mkdir -p ${NIFI_HOME}/provenance_repository \
    && mkdir -p ${NIFI_HOME}/state \
    && mkdir -p ${NIFI_LOG_DIR} \
    && ln -s ${NIFI_HOME} ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION} \
    && chmod -R g+rwX ${NIFI_HOME}

ADD bootstrap.conf ${NIFI_HOME}/conf/bootstrap.conf

# Clear nifi-env.sh in favour of configuring all environment variables in the Dockerfile
RUN echo "#!/bin/sh\n" > ${NIFI_HOME}/bin/nifi-env.sh

# Web HTTP(s) & Socket Site-to-Site Ports
EXPOSE 8080 8443 10000 8000

WORKDIR ${NIFI_HOME}

RUN chmod 777 ../scripts/start.sh
RUN chmod 777 /opt/nifi/scripts/toolkit.sh

USER 1001

# Apply configuration and start NiFi
#
# We need to use the exec form to avoid running our command in a subshell and omitting signals,
# thus being unable to shut down gracefully:
# https://docs.docker.com/engine/reference/builder/#entrypoint
#
# Also we need to use relative path, because the exec form does not invoke a command shell,
# thus normal shell processing does not happen:
# https://docs.docker.com/engine/reference/builder/#exec-form-entrypoint-example
ENTRYPOINT ["../scripts/start.sh"]
keremceliker commented 4 years ago

Thanks Mate !, u re really rock. it works well according to my another try but there seems a little issue in POD while the creating on it that we got the follows. Any idea what should i do ?

Here is the Issue (after run your code,mate)

replacing target file  /opt/nifi/nifi-current/conf/nifi.properties
--
|/opt/nifi/scripts/toolkit.sh: 18: /opt/nifi/scripts/toolkit.sh: cannot create //.nifi-cli.nifi.properties: Permission denied
keremceliker commented 4 years ago

The problem has been Fixed... The reason for the problem is the permission issue caused by the Dockerfile that came in the catalog. Generally a bug Apache NiFi-sourced on OpenShift when u want to run on it with Docker Image...

Kerem Çeliker

roskoN commented 3 years ago

Hey @keremceliker

could you share your fix, please?

I am stuck with the same problem...

Thanks, Rosko

wilddeuces commented 3 years ago

Hey @keremceliker

could you share your fix, please?

I am stuck with the same problem...

Thanks, Rosko

Did you add the ServiceAccount with priviliged SCC ? Also did you put RunAsUser in the Template?

     serviceAccountName: default
     securityContext:
       runAsUser: 0
roskoN commented 3 years ago

Hi @wilddeuces ,

thanks for your response! Unfortunately, this is not an option, because it's not permitted on the OpenShift cluster where I am running.

wilddeuces commented 3 years ago

Hi @wilddeuces ,

thanks for your response! Unfortunately, this is not an option, because it's not permitted on the OpenShift cluster where I am running.

Try that solution:

FROM openjdk:8-jre

#ARG NIFI_VERSION=1.9.2
ARG NIFI_VERSION=1.12.1
ARG BASE_URL=https://archive.apache.org/dist
ARG MIRROR_BASE_URL=${MIRROR_BASE_URL:-${BASE_URL}}
ARG NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
ARG NIFI_TOOLKIT_BINARY_PATH=${NIFI_TOOLKIT_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-toolkit-${NIFI_VERSION}-bin.zip}

ENV NIFI_BASE_DIR=/opt/nifi
ENV NIFI_HOME ${NIFI_BASE_DIR}/nifi-current
ENV NIFI_TOOLKIT_HOME ${NIFI_BASE_DIR}/nifi-toolkit-current

ENV NIFI_PID_DIR=${NIFI_HOME}/run
ENV NIFI_LOG_DIR=${NIFI_HOME}/logs

USER root
RUN chmod -R 777 /opt/

# Setup NiFi user and create necessary directories
RUN mkdir -p ${NIFI_BASE_DIR} \
    && apt-get update \
    && apt-get install -y jq xmlstarlet procps python3 bash nano

# Download, validate, and expand Apache NiFi Toolkit binary.
RUN curl -fSL ${MIRROR_BASE_URL}/${NIFI_TOOLKIT_BINARY_PATH} -o ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip \
    && echo "$(curl ${BASE_URL}/${NIFI_TOOLKIT_BINARY_PATH}.sha256) *${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip" | sha256sum -c - \
    && unzip ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip -d ${NIFI_BASE_DIR} \
    && rm ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip \
    && mv ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION} ${NIFI_TOOLKIT_HOME} \
    && ln -s ${NIFI_TOOLKIT_HOME} ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION} \
    && chmod -R g+rwX ${NIFI_TOOLKIT_HOME}

# Download, validate, and expand Apache NiFi binary.
RUN curl -fSL ${MIRROR_BASE_URL}/${NIFI_BINARY_PATH} -o ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip \
    && echo "$(curl ${BASE_URL}/${NIFI_BINARY_PATH}.sha256) *${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip" | sha256sum -c - \
    && unzip ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip -d ${NIFI_BASE_DIR} \
    && rm ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip \
    && mv ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION} ${NIFI_HOME} \
    && mkdir -p ${NIFI_HOME}/conf \
    && mkdir -p ${NIFI_HOME}/database_repository \
    && mkdir -p ${NIFI_HOME}/flowfile_repository \
    && mkdir -p ${NIFI_HOME}/content_repository \
    && mkdir -p ${NIFI_HOME}/provenance_repository \
    && mkdir -p ${NIFI_HOME}/state \
    && mkdir -p ${NIFI_LOG_DIR} \
    && ln -s ${NIFI_HOME} ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION} \
    && chmod -R g+rwX ${NIFI_HOME}

USER 1001

ADD bootstrap.conf ${NIFI_HOME}/conf/bootstrap.conf
ADD sh/ ${NIFI_BASE_DIR}/scripts/

# Clear nifi-env.sh in favour of configuring all environment variables in the Dockerfile
RUN echo "#!/bin/sh\n" > ${NIFI_HOME}/bin/nifi-env.sh

# Web HTTP(s) & Socket Site-to-Site Ports
EXPOSE 8080 8443 10000 8000

WORKDIR ${NIFI_HOME}

RUN chmod 777 ../scripts/start.sh
RUN chmod 777 /opt/nifi/scripts/toolkit.sh

# Apply configuration and start NiFi
#
# We need to use the exec form to avoid running our command in a subshell and omitting signals,
# thus being unable to shut down gracefully:
# https://docs.docker.com/engine/reference/builder/#entrypoint
#
# Also we need to use relative path, because the exec form does not invoke a command shell,
# thus normal shell processing does not happen:
# https://docs.docker.com/engine/reference/builder/#exec-form-entrypoint-example
ENTRYPOINT ["../scripts/start.sh"]
roskoN commented 3 years ago

Hey @wilddeuces,

Thank you for your persistence, but that also did not work.

Step 23/25 : RUN chmod 777 ../scripts/start.sh ---> Running in c3847fdd7ddc chmod: changing permissions of '../scripts/start.sh': Operation not permitted  Removing intermediate container c3847fdd7ddc The command '/bin/sh -c chmod 777 ../scripts/start.sh' returned a non-zero code: 1

wilddeuces commented 3 years ago

What OpenShift version are you using? This should run fine on v4.5 Try adding the scripts while building with root:

FROM openjdk:8-jre

#ARG NIFI_VERSION=1.9.2
ARG NIFI_VERSION=1.12.1
ARG BASE_URL=https://archive.apache.org/dist
ARG MIRROR_BASE_URL=${MIRROR_BASE_URL:-${BASE_URL}}
ARG NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
ARG NIFI_TOOLKIT_BINARY_PATH=${NIFI_TOOLKIT_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-toolkit-${NIFI_VERSION}-bin.zip}

ENV NIFI_BASE_DIR=/opt/nifi
ENV NIFI_HOME ${NIFI_BASE_DIR}/nifi-current
ENV NIFI_TOOLKIT_HOME ${NIFI_BASE_DIR}/nifi-toolkit-current

ENV NIFI_PID_DIR=${NIFI_HOME}/run
ENV NIFI_LOG_DIR=${NIFI_HOME}/logs

USER root
ADD sh/ ${NIFI_BASE_DIR}/scripts/
RUN chmod -R 777 /opt/

# Setup NiFi user and create necessary directories
RUN mkdir -p ${NIFI_BASE_DIR} \
    && apt-get update \
    && apt-get install -y jq xmlstarlet procps python3 bash nano

# Download, validate, and expand Apache NiFi Toolkit binary.
RUN curl -fSL ${MIRROR_BASE_URL}/${NIFI_TOOLKIT_BINARY_PATH} -o ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip \
    && echo "$(curl ${BASE_URL}/${NIFI_TOOLKIT_BINARY_PATH}.sha256) *${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip" | sha256sum -c - \
    && unzip ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip -d ${NIFI_BASE_DIR} \
    && rm ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip \
    && mv ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION} ${NIFI_TOOLKIT_HOME} \
    && ln -s ${NIFI_TOOLKIT_HOME} ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION} \
    && chmod -R g+rwX ${NIFI_TOOLKIT_HOME}

# Download, validate, and expand Apache NiFi binary.
RUN curl -fSL ${MIRROR_BASE_URL}/${NIFI_BINARY_PATH} -o ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip \
    && echo "$(curl ${BASE_URL}/${NIFI_BINARY_PATH}.sha256) *${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip" | sha256sum -c - \
    && unzip ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip -d ${NIFI_BASE_DIR} \
    && rm ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip \
    && mv ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION} ${NIFI_HOME} \
    && mkdir -p ${NIFI_HOME}/conf \
    && mkdir -p ${NIFI_HOME}/database_repository \
    && mkdir -p ${NIFI_HOME}/flowfile_repository \
    && mkdir -p ${NIFI_HOME}/content_repository \
    && mkdir -p ${NIFI_HOME}/provenance_repository \
    && mkdir -p ${NIFI_HOME}/state \
    && mkdir -p ${NIFI_LOG_DIR} \
    && ln -s ${NIFI_HOME} ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION} \
    && chmod -R g+rwX ${NIFI_HOME}

ADD bootstrap.conf ${NIFI_HOME}/conf/bootstrap.conf

USER 1001

# Clear nifi-env.sh in favour of configuring all environment variables in the Dockerfile
RUN echo "#!/bin/sh\n" > ${NIFI_HOME}/bin/nifi-env.sh

# Web HTTP(s) & Socket Site-to-Site Ports
EXPOSE 8080 8443 10000 8000

WORKDIR ${NIFI_HOME}

# Apply configuration and start NiFi
#
# We need to use the exec form to avoid running our command in a subshell and omitting signals,
# thus being unable to shut down gracefully:
# https://docs.docker.com/engine/reference/builder/#entrypoint
#
# Also we need to use relative path, because the exec form does not invoke a command shell,
# thus normal shell processing does not happen:
# https://docs.docker.com/engine/reference/builder/#exec-form-entrypoint-example
ENTRYPOINT ["../scripts/start.sh"]