use jars from vendor/jars?

[mooreniemi]

I'm deploying an app in Docker. I won't have maven or the .m2 repo in that environment. I used jbundle install --vendor to put the jars in a vendor/jars directory. How do I get JBUNDLER_LOCAL_REPO to point to vendor/jars or otherwise just not point to .m2 dir?

[mooreniemi]

So far what I've been doing is adding this line to the classpath.rb file but this feels hacky to me: ENV_JAVA['jars.home'] = File.expand_path( '../../vendor/jars', __FILE__ )

[mooreniemi]

I also have been seeing an error in the Docker environment when I run gem install bundler, which is:

RuntimeError:

you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with lock_jars command

no such file to load -- jar/my/app/requires (LoadError) do_require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:309 block in require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:228 require_jar_with_block at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:252 require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:227 require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:318 block in require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/classpath.rb:46 block in process at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/lock.rb:69 each_line at org/jruby/RubyString.java:4764 process at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/lock.rb:55 process at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/classpath.rb:66 require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/classpath.rb:42 require_jars_lock! at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:194 require_jars_lock at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:215 require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:226 require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:318

at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/psych_jars.rb:5 require at org/jruby/RubyKernel.java:937 (root) at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1 at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54 require at org/jruby/RubyKernel.java:937 (root) at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/psych.rb:3 (root) at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1 require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54 load_yaml at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems.rb:609 load_file at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/config_file.rb:328 initialize at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/config_file.rb:197 at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/gem_runner.rb:74 load at org/jruby/RubyKernel.java:955 at /opt/jruby/jruby-9.0.5.0/bin/gem:4

[mkristian]

quick question before I look into this: do you need to use jbundler or would jar-dependencies from jruby itself do the job for you. it can use the same Jarfile but comes with a command lock_jars which produces a Jars.lock file. and vendoring works better with it.

[mooreniemi]

@mkristian i'm actually not sure, to be honest. i've found it tough to sort out what my best option is. briefly, i have a gem that depends on a jar. (that much seems to be working fine) then i have an app that requires that gem, but in addition depends on another jar. the app itself i want to package as a jar ultimately. i'm happy to take your guidance on what the right practice is for my case.

[mooreniemi]

Perhaps you can help me better understand how the process works? Edited: I misunderstood my error messages before, the error is happening at the gem install stage. That makes sense with the stack trace too. It looks as if doing any kind of gem install is triggering an attempt to grab my app's jar dependency, which seems wrong to me.

[mkristian]

@mooreniemi when installing a gem whether bundler or gem install a post-install hook is triggered which does look if the gems has jar dependencies and installs them. to avoid this trigger set 'JARS_SKIP=true on your shell environment. if rubocop tries to install a the gem with jar dependencies then it will trigger the same post-install hook.

this problem is with a docker container ? is it possible for you to share the Dockerfile which fails ?

packing the app as a jar - with warbler or manually ? there are many ways to pack a ruby app as a jar ;)

[mooreniemi]

the post-install hook of an independent gem triggering a dependence on my project's jars seems wrong though, no? i can see it trying to trigger jars it or its transitive dependencies depend on, but that's not the case here.

this is the entirety of the script that is still failing, even with the exported env var you suggested:

#!/bin/bash
set -x
set -e

echo "STARTING STATIC ANALYSIS"

cd app_directory
export JARS_SKIP=true
jruby --version
jgem install rubocop

rubocop

i will dig up the Dockerfile for the container my app is using FROM

as for packing up into a jar, i've been using warble and that's be alright so far

[mkristian]

@mooreniemi if you use warbler you need to stick to jbundler

the post-install hook only does something on a gem with its jar dependencies. any thing else is bug. I would appreciate if can provide a reproducable case for me. or setting JARS_DEBUG=true should produce some output which can be helpful.

[mooreniemi]

ill paste up the debug output soon, but fwiw i grabbed the Dockerfile

FROM java_base

RUN apk update && apk add --update bash

ENV JRUBY_VERSION 9.0.5.0
RUN mkdir -p /opt/jruby \
  && curl -fSL https://s3.amazonaws.com/jruby.org/downloads/${JRUBY_VERSION}/jruby-bin-${JRUBY_VERSION}.tar.gz -o /tmp/jruby.tar.gz \
  && tar -zxf /tmp/jruby.tar.gz -C /opt/jruby \
  && rm /tmp/jruby.tar.gz
ENV PATH /opt/jruby/jruby-${JRUBY_VERSION}/bin:$PATH

RUN echo 'gem: --no-rdoc --no-ri' >> ~/.gemrc

ENV GEM_HOME /usr/local/bundle
ENV PATH $GEM_HOME/bin:$PATH
RUN gem install bundler \
    && bundle config --global path "$GEM_HOME" \
    && bundle config --global bin "$GEM_HOME/bin"

# don't create ".bundle" in all our apps
ENV BUNDLE_APP_CONFIG $GEM_HOME

[mooreniemi]

and the java_base image looks like this:

From alpine-base

RUN apk update && apk add --update wget curl
RUN apk --no-cache add ca-certificates

RUN wget --no-check-certificate https://github.com/andyshinn/alpine-pkg-glibc/releases/download/2.23-r1/glibc-2.23-r1.apk
RUN apk add --allow-untrusted glibc-2.23-r1.apk

RUN wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u65-b17/server-jre-8u65-linux-x64.tar.gz

RUN mkdir -p /usr/lib/jvm

RUN tar zxfv server-jre-8u65-linux-x64.tar.gz -C /usr/lib/jvm/
RUN mv /usr/lib/jvm/jdk1.8.0_65 /usr/lib/jvm/oracle_jdk8

ENV J2REDIR /usr/lib/jvm/oracle_jdk8
ENV PATH $PATH:/usr/lib/jvm/oracle_jdk8/bin
ENV JAVA_HOME /usr/lib/jvm/oracle_jdk8

RUN rm server-jre-8u65-linux-x64.tar.gz && \
    apk del wget ca-certificates && \
    rm -rf /tmp/* /var/cache/apk/* \
           "$JAVA_HOME"/*src.zip \
           "$JAVA_HOME"/*src.zip \
           "$JAVA_HOME"/lib/missioncontrol \
           "$JAVA_HOME"/lib/visualvm \
           "$JAVA_HOME"/lib/*javafx* \
           "$JAVA_HOME"/jre/lib/plugin.jar \
           "$JAVA_HOME"/jre/lib/ext/jfxrt.jar \
           "$JAVA_HOME"/jre/bin/javaws \
           "$JAVA_HOME"/jre/lib/javaws.jar \
           "$JAVA_HOME"/jre/lib/desktop \
           "$JAVA_HOME"/jre/plugin \
           "$JAVA_HOME"/jre/lib/deploy* \
           "$JAVA_HOME"/jre/lib/*javafx* \
           "$JAVA_HOME"/jre/lib/*jfx* \
           "$JAVA_HOME"/jre/lib/amd64/libdecora_sse.so \
           "$JAVA_HOME"/jre/lib/amd64/libprism_*.so \
           "$JAVA_HOME"/jre/lib/amd64/libfxplugins.so \
           "$JAVA_HOME"/jre/lib/amd64/libglass.so \
           "$JAVA_HOME"/jre/lib/amd64/libgstreamer-lite.so \
           "$JAVA_HOME"/jre/lib/amd64/libjavafx*.so \
           "$JAVA_HOME"/jre/lib/amd64/libjfx*.so

# Hotfix for glibc hack that fixes the order of DNS resolving (i.e. check /etc/hosts first and then lookup DNS-servers).
# To fix this we just create /etc/nsswitch.conf and add the following line:
RUN echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf

COPY ./newrelic ./newrelic
COPY ./health.sh ./health.sh
COPY ./bootstrap.sh ./bootstrap.sh

[mkristian]

thanx, will work through them tomorrow

[mooreniemi]

thanks @mkristian -- i'm not seeing any additional output using the env var you suggested. i will see what i can dig up from my end.

[mooreniemi]

oh this is alpine_base Dockerfile too

FROM alpine:3.3

RUN apk update && apk add --update openntpd

[mooreniemi]

fyi:

i added a puts on L390 of jar_dependencies.rb:

 def require_jar( *args )
   return nil unless Jars.require?
   puts "#{args}"

and ran gem install rubocop in several different directories

1. not my app

   C02LX2KVFD58:~/random_dir gem install rubocop warning: --1.9 ignored Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0 jruby: warning: unknown property jruby.cext.enabled ["org.yaml", "snakeyaml", "1.14"] ["org.bouncycastle", "bcpkix-jdk15on", "1.54"] ["org.bouncycastle", "bcprov-jdk15on", "1.54"] Successfully installed rubocop-0.40.0 1 gem installed

2. my app (has a Jarfile and vendor/jars)

   C02LX2KVFD58:~/my_app gem install rubocop warning: --1.9 ignored Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0 jruby: warning: unknown property jruby.cext.enabled ["org.yaml", "snakeyaml", "1.14"] ["custom", "internal-lib", "1.7.9"] ["org.javassist", "javassist", "3.19.0-GA"] ["com.google.code.findbugs", "annotations", "2.0.1"] ["log4j", "log4j", "1.2.17"] ["org.yaml", "snakeyaml", "1.14"] ["org.slf4j", "slf4j-api", "1.7.12"] ["commons-collections", "commons-collections", "3.2.1"] ["joda-time", "joda-time", "2.8.1"] ["commons-lang", "commons-lang", "2.6"] ["commons-pool", "commons-pool", "1.6"] ["org.reflections", "reflections", "0.9.10"] ["org.apache.commons", "commons-lang3", "3.0"] ["com.google.code.findbugs", "jsr305", "3.0.0"] ["commons-codec", "commons-codec", "1.10"] ["com.googlecode.json-simple", "json-simple", "1.1.1"] ["org.postgresql", "postgresql", "9.4-1201-jdbc41"] ["commons-validator", "commons-validator", "1.4.1"] ["com.rabbitmq", "amqp-client", "3.5.3"] ["commons-io", "commons-io", "2.4"] ["commons-beanutils", "commons-beanutils", "1.8.3"] ["com.mchange", "c3p0", "0.9.5.1"] ["org.jodd", "jodd-core", "3.6.6"] ["commons-digester", "commons-digester", "1.8.1"] ["org.jsoup", "jsoup", "1.8.3"] ["com.google.guava", "guava", "18.0"] ["commons-logging", "commons-logging", "1.2"] ["org.jodd", "jodd-upload", "3.6.6"] ["junit", "junit", "4.10"] ["javax.validation", "validation-api", "1.1.0.Final"] ["org.jodd", "jodd-http", "3.6.6"] ["org.slf4j", "slf4j-log4j12", "1.7.12"] ["com.mchange", "mchange-commons-java", "0.2.10"] ["org.hamcrest", "hamcrest-core", "1.1"] ["org.bouncycastle", "bcpkix-jdk15on", "1.54"] ["org.bouncycastle", "bcprov-jdk15on", "1.54"] Successfully installed rubocop-0.40.0 1 gem installed

[mooreniemi]

as further confirmation something is amiss, i changed the script being executed on my dockerized CI pipeline:

#!/bin/bash
set -x
set -e

echo "STARTING STATIC ANALYSIS"

jruby --version
export JARS_DEBUG=true
export JARS_SKIP=true
jgem install rubocop
cd app_directory
rubocop

and get the following error (the gem installs fine outside the app directory) but:

RuntimeError:

you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with lock_jars command

no such file to load -- zipcar/cheetah-sdk/1.7.9/cheetah-sdk-1.7.9 (LoadError) do_require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:309 block in require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:228 require_jar_with_block at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:252 require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:227 require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:318 block in require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/classpath.rb:46 block in process at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/lock.rb:69 each_line at org/jruby/RubyString.java:4764 process at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/lock.rb:55 process at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/classpath.rb:66 require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/classpath.rb:42 require_jars_lock! at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:194 require_jars_lock at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:215 require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:226 require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:318

at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/psych_jars.rb:5 require at org/jruby/RubyKernel.java:937 require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54 at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/psych.rb:1 require at org/jruby/RubyKernel.java:937 require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54 at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/psych.rb:3 require at org/jruby/RubyKernel.java:937 require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54 at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/yaml.rb:1 require at org/jruby/RubyKernel.java:937 require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54 at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/yaml.rb:5 require at org/jruby/RubyKernel.java:937 (root) at /usr/local/bundle/gems/rubocop-0.40.0/lib/rubocop/config_loader_resolver.rb:1 at /usr/local/bundle/gems/rubocop-0.40.0/lib/rubocop/config_loader_resolver.rb:4 load at org/jruby/RubyKernel.java:955 at /usr/local/bundle/bin/rubocop:23

[mkristian]

@mooreniemi this looks ok for me. installing gems need yaml and yaml needs psych and psych needs a jar and then all jars gets installed when there is Jars.lock file. is there such file inside the vendor/jars directory ?

one thing I was thinking about: the error with you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars withlock_jarscommand means there jar-dependencies can find the jars.

I thought you have the jars vendored ?

[mooreniemi]

@mkristian i'm really confused. if i am installing rubocop -- i expect that to install the same in any directory. why would it depend on directory whether it loads extraneous jars it doesn't even use? this is precisely what i don't want in a dep manager: i want to deterministically know i'm loading stuff i need. "and then all jars gets installed" is just a huge leap to me.

i do have the jars vendored, but that was my original question: it never seems to pick that up, and always tries to go for .m2/ if i could force my project to use the jars in vendor/jars this problem would be papered over, but i still think it is bad behavior

[mkristian]

export JARS_HOME=path/to/vendor/jars - not sure whether this what you tried.

since I do not fully understand why you see all those jars loaded, I can not say it is intended or not. is there a Jars.lock file some where in app_directory or app_directory/vendor or app_directory/vendor/jars ?

[mooreniemi]

vendor/jars contains a Jars.lock. i will try exporting that env var for my docker containers. i bet it will fix this, but i'd still say just calling gem or other ruby executables that require a jar shouldnt pull in all jars if, by chance, some other Jars.lock file exists

[mkristian]

looking into those vendor dirs is too much, you convinced me. ignoring the Jars.lock in the current directory - not sure. so you rather have the application call Jars.setup or something ? I appreciate your input !

[mooreniemi]

exporting of the env var resolved my issue for the moment. maybe i could PR some wiki/readme changes to help the next peeps along?

i do think i would rather the application call Jars.setup or like lock_jar jars = LockJar.load. do you know how bundler handles this generally? they might be good to model off of, i don't know in depth :/

thanks for all your responses by the way, really appreciate the work you do on this.