search

telefonicaid / fiware-cygnus

A connector in charge of persisting context data sources into other third-party databases and storage systems, creating a historical view of the context
https://fiware-cygnus.rtfd.io/
GNU Affero General Public License v3.0
64 stars 105 forks source link

JAVA classes are not compiled in docker image #1669

Open jmezzera opened 5 years ago

jmezzera commented 5 years ago

I'm deploying a completely new implementation of the FIWARE stack and came across an issue I had last time. I'm trying to run the Docker image (docker.io/fiware/cygnus-common, ID 2514238c292f) but when I do so, I get the following errors:


java.lang.ClassNotFoundException: com.telefonica.iot.cygnus.handlers.OrionRestHandler
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at org.apache.flume.source.http.HTTPSource.configure(HTTPSource.java:102)
        at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
        at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:331)
        at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
        at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
19/06/17 19:22:07 ERROR node.AbstractConfigurationProvider: Source http-source has been removed due to an error during configuration
java.lang.RuntimeException: java.lang.ClassNotFoundException: com.telefonica.iot.cygnus.handlers.OrionRestHandler
        at com.google.common.base.Throwables.propagate(Throwables.java:156)
        at org.apache.flume.source.http.HTTPSource.configure(HTTPSource.java:114)
        at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
        at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:331)
        at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
        at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.telefonica.iot.cygnus.handlers.OrionRestHandler
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at org.apache.flume.source.http.HTTPSource.configure(HTTPSource.java:102)
        ... 11 more

as if it couldn't find the compiled .class files.

Inspecting the container I found a script, compile.sh (implemented in PR #1220) which takes care of the entire process of downloading dependencies and compiling the project. If I remember correctly, this is how I figured it out last time. However, I don't think this is the correct aproach.

Any help on this matter would be appreciated

fgalan commented 5 years ago

Maybe the Dockerfile at https://github.com/telefonicaid/fiware-cygnus/blob/master/docker/cygnus-common/Dockerfile has gone obsolete as this "flavour" of Cygnus is not used too often (most of the people uses cygnus-ngsi).

I'd suggest to have a look to cygnus-ngsi Dockerfile (https://github.com/telefonicaid/fiware-cygnus/blob/master/docker/cygnus-ngsi/Dockerfile) which do work. Maybe comparing both the problem can be found.

If you found it at the end and contribute with the fix with a PR to the repo it would be great. Thanks!

jmezzera commented 5 years ago

Thank you for your quick response. You are right, cygnus-ngsi is what I wanted in the first place, but I got confused while reading the docs. I checked again and it was entirely my mistake.

However, when running the new docker image I get an error in the entrypoint script (/cygnus-entrypoint.sh: line 862: syntax error: unexpected end of file)

Should I open a new issue on this or can we follow the discussion here?

Thanks!

fgalan commented 5 years ago

It seems there was some syntax error in the entroypoint.sh. They should have been fixed by this PR https://github.com/telefonicaid/fiware-cygnus/pull/1676.

Please re-pull the fiware-cygnus from dockerhub and try again.

jmezzera commented 5 years ago

The image from dockerhub still crashes. Lets try with the docker file

jmezzera commented 5 years ago

Could not build the container with the docker file


The command '/bin/sh -c adduser ${CYGNUS_USER} &&     yum -y install nc java-${JAVA_VERSION}-openjdk-devel git &&     export JAVA_HOME=/usr/lib/jvm/java-${JAVA_VERSION}-openjdk &&     export MAVEN_OPTS="-Xmx4096m -XX:MaxPermSize=256m -Dfile.encoding=UTF-8 -Dproject.build.sourceEncoding=UTF-8 -Dmaven.compiler.useIncrementalCompilation=false -DdependencyLocationsEnabled=false -XX:+TieredCompilation -XX:TieredStopAtLevel=1" &&     export MAVEN_ARGS="-B -T4" &&     echo "INFO: Getting apache preferred site and obtain URLs for Maven and Flume..." &&     APACHE_DOMAIN="$(curl -s 'https://www.apache.org/dyn/closer.cgi?as_json=1' | python -c 'import json,sys;obj=json.load(sys.stdin);print obj["preferred"]')"       || APACHE_DOMAIN="http://archive.apache.org/dist/" &&     MVN_URL="${APACHE_DOMAIN}maven/maven-3/${MVN_VER}/binaries/apache-maven-${MVN_VER}-bin.tar.gz" &&     FLUME_URL="${APACHE_DOMAIN}flume/${FLUME_VER}/apache-flume-${FLUME_VER}-bin.tar.gz" &&     echo -e $'INFO: Java version <'${JAVA_VERSION}'>\n'$(java -version)'\nINFO: Apache domain <'${APACHE_DOMAIN}'>\nINFO: URL MAVEN <'${MVN_URL}'>\nINFO: URL FLUME <'${FLUME_URL}'>' &&     echo "INFO: Download and install Maven and Flume..." &&     curl --remote-name --location --insecure --silent --show-error "${MVN_URL}" &&     tar xzf apache-maven-${MVN_VER}-bin.tar.gz &&     mv apache-maven-${MVN_VER} ${MVN_HOME} &&     curl --remote-name --location --insecure --silent --show-error "${FLUME_URL}" &&     tar zxf apache-flume-${FLUME_VER}-bin.tar.gz &&     mv apache-flume-${FLUME_VER}-bin ${FLUME_HOME} &&     mkdir -p ${FLUME_HOME}/plugins.d/cygnus &&     mkdir -p ${FLUME_HOME}/plugins.d/cygnus/lib &&     mkdir -p ${FLUME_HOME}/plugins.d/cygnus/libext &&     chown -R cygnus:cygnus ${FLUME_HOME} &&     echo "INFO: Cloning Cygnus using <${GIT_URL_CYGNUS}> and <${GIT_REV_CYGNUS}>" &&     git clone ${GIT_URL_CYGNUS} ${CYGNUS_HOME} &&     cd ${CYGNUS_HOME} &&     git checkout ${GIT_REV_CYGNUS} &&     echo "INFO: Build and install cygnus-common" &&     cd ${CYGNUS_HOME}/cygnus-common &&     ${MVN_HOME}/bin/mvn ${MAVEN_ARGS} clean compile exec:exec assembly:single &&     cp target/cygnus-common-${CYGNUS_VERSION}-jar-with-dependencies.jar ${FLUME_HOME}/plugins.d/cygnus/libext/ &&     ${MVN_HOME}/bin/mvn install:install-file -Dfile=${FLUME_HOME}/plugins.d/cygnus/libext/cygnus-common-${CYGNUS_VERSION}-jar-with-dependencies.jar -DgroupId=com.telefonica.iot -DartifactId=cygnus-common -Dversion=${CYGNUS_VERSION} -Dpackaging=jar -DgeneratePom=false &&     echo "INFO: Build and install cygnus-ngsi" &&     cd ${CYGNUS_HOME}/cygnus-ngsi &&     ${MVN_HOME}/bin/mvn ${MAVEN_ARGS} clean compile exec:exec assembly:single &&     cp target/cygnus-ngsi-${CYGNUS_VERSION}-jar-with-dependencies.jar ${FLUME_HOME}/plugins.d/cygnus/lib/ &&     echo "INFO: Install Cygnus Application script" &&     cp ${CYGNUS_HOME}/cygnus-common/target/classes/cygnus-flume-ng ${FLUME_HOME}/bin/ &&     chmod +x ${FLUME_HOME}/bin/cygnus-flume-ng &&     echo "INFO: Instantiate some configuration files" &&     cp ${CYGNUS_HOME}/cygnus-common/conf/log4j.properties.template ${FLUME_HOME}/conf/log4j.properties &&     cp ${CYGNUS_HOME}/cygnus-ngsi/conf/grouping_rules.conf.template ${FLUME_HOME}/conf/grouping_rules.conf &&     cp ${CYGNUS_HOME}/cygnus-ngsi/conf/name_mappings.conf.template ${FLUME_HOME}/conf/name_mappings.conf &&     echo "INFO: Create Cygnus log folder" &&     mkdir -p /var/log/cygnus &&     echo "INFO: Cleanup to thin the final image... doing optimizations..." &&     cd ${CYGNUS_HOME}/cygnus-common &&     ${MVN_HOME}/bin/mvn ${MAVEN_ARGS} clean &&     cd ${CYGNUS_HOME}/cygnus-ngsi &&     ${MVN_HOME}/bin/mvn ${MAVEN_ARGS} clean &&     rm -rf /root/.m2 && rm -rf ${MVN_HOME} && rm -rf ${FLUME_HOME}/docs && rm -rf ${CYGNUS_HOME}/doc && rm -f /*.tar.gz &&     echo "INFO: Java runtime not needs JAVA_HOME... Unsetting..." &&     unset JAVA_HOME &&     yum erase -y git java-${JAVA_VERSION}-openjdk-devel &&     rpm -qa redhat-logos gtk2 pulseaudio-libs libvorbis jpackage* groff alsa* atk cairo libX* | xargs -r rpm -e --nodeps && yum -y erase libss &&     yum clean all && rpm -vv --rebuilddb && rm -rf /var/lib/yum/yumdb && rm -rf /var/lib/yum/history &&     find /usr/share/locale -mindepth 1 -maxdepth 1 ! -name 'en_US' ! -name 'locale.alias' | xargs -r rm -r && rm -f /var/log/*log &&     bash -c 'localedef --list-archive | grep -v -e "en_US" | xargs localedef --delete-from-archive' &&     /bin/cp -f /usr/lib/locale/locale-archive /usr/lib/locale/locale-archive.tmpl &&     build-locale-archive && find ${CYGNUS_HOME} -name '.[^.]*' 2>/dev/null | xargs -r rm -rf' returned a non-zero code: 18

Sorry about the format, but it's only one command.

jmezzera commented 5 years ago

So, I decided to give it another try just to make sure and apparently it worked this time. At least the container is running now. May there was some delay in dockerhub, I don´t really know how Docker handles the reupload of images.

I'm leaving the office at this moment and will continue with testing tomorrow. If it works correctly, I'll close the issue. But I wanted all you to know that there was some improvement.

Thanks!

fgalan commented 5 years ago

May there was some delay in dockerhub

Most probably that's the cause. I had happened to me similar cases in the past.

Happy to know you are progressing :)