Grapes does not work with bundled config

[mkutz]

When starting a script using Grapes, the dependency resolution fails using the bundled Grapes config:

Local Ivy config file 'jar:file:/opt/groovy/lib/groovy-3.0.4.jar!/groovy/grape/defaultGrapeConfig.xml' appears corrupt - ignoring it and using default config instead
Error was: failed to load settings from jar:file:/opt/groovy/lib/groovy-3.0.4.jar!/groovy/grape/defaultGrapeConfig.xml: impossible to add configured child for ivy on class org.apache.ivy.plugins.resolver.FileSystemResolver: ivy pattern must be absolute: ?/.groovy`/grapes/[organisation]/[module]/ivy-[revision].xml

The configuration as found within the container is

<!--
     Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
     regarding copyright ownership.  The ASF licenses this file
     to you under the Apache License, Version 2.0 (the
     "License"); you may not use this file except in compliance
     with the License.  You may obtain a copy of the License at
       http://www.apache.org/licenses/LICENSE-2.0
     Unless required by applicable law or agreed to in writing,
     software distributed under the License is distributed on an
     "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
     KIND, either express or implied.  See the License for the
     specific language governing permissions and limitations
     under the License.
-->
<ivysettings>
  <settings defaultResolver="downloadGrapes"/>
  <resolvers>
    <chain name="downloadGrapes" returnFirst="true">
      <filesystem name="cachedGrapes">
        <ivy pattern="${user.home}/.groovy/grapes/[organisation]/[module]/ivy-[revision].xml"/>
        <artifact pattern="${user.home}/.groovy/grapes/[organisation]/[module]/[type]s/[artifact]-[revision](-[classifier]).[ext]"/>
      </filesystem>
      <ibiblio name="localm2" root="${user.home.url}/.m2/repository/" checkmodified="true" changingPattern=".*" changingMatcher="regexp" m2compatible="true"/>
      <!-- TODO: add 'endorsed groovy extensions' resolver here -->
      <ibiblio name="jcenter" root="https://jcenter.bintray.com/" m2compatible="true"/>
      <ibiblio name="ibiblio" m2compatible="true"/>
    </chain>
  </resolvers>
</ivysettings>

It seams that ${user.home} resolves the ? making the file resolver fail.

In effect any import that requires a dependency fails, as the depencency resoltion does not work at all.

[keeganwitt]

I'm not able to reproduce this, could you provide more information about your setup? Which image are you using? What Docker environment and OS are you running on?

PS C:\Users\Keegan\Desktop\scripts> docker run --rm -v "${pwd}:/home/groovy/scripts" -w /home/groovy/scripts groovy:latest groovy  -D ivy.message.logger.level=3 grapesTest.groovy
:: loading settings :: url = jar:file:/opt/groovy/lib/groovy-3.0.4.jar!/groovy/grape/defaultGrapeConfig.xml
no default ivy user dir defined: set to /home/groovy/.ivy2
no default cache defined: set to /home/groovy/.ivy2/cache
settings loaded (126ms)
        default cache: /home/groovy/.ivy2/cache
        default resolver: downloadGrapes
...

grapesTest.groovy

@Grab(group='org.apache.commons', module='commons-lang3', version='3.7')
import org.apache.commons.lang3.SystemUtils

println "Using Java ${SystemUtils.JAVA_VERSION}"

[mkutz]

Here's the minimal version of a pipeline to reproduce the problem:

pipeline {
  agent { docker 'groovy:3.0' }

  stages {
    stage("test") {
      steps {
        writeFile encoding: 'UTF-8', file: 'script.groovy',
        text: '''#!/usr/bin/env groovy
@Grab("com.squareup.retrofit2:retrofit:2.8.1")
import retrofit2.Retrofit'''
        sh "groovy script.groovy"
      }
    }
  }
}

Here's the resulting output:

+ docker inspect -f . groovy:3.0
$ docker run -t -d -u 109:115 -w /var/lib/jenkins/jobs/groovy-container/workspace -v /var/lib/jenkins/jobs/groovy-container/workspace:/var/lib/jenkins/jobs/groovy-container/workspace:rw,z -v /var/lib/jenkins/jobs/groovy-container/workspace@tmp:/var/lib/jenkins/jobs/groovy-container/workspace@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** groovy:3.0 cat
$ docker top e092372aa7714c0d5d2066d7614773f65b57911828e8eed18b5060ba98162cf5 -eo pid,comm
+ groovy script.groovy
Local Ivy config file 'jar:file:/opt/groovy/lib/groovy-3.0.4.jar!/groovy/grape/defaultGrapeConfig.xml' appears corrupt - ignoring it and using default config instead
Error was: failed to load settings from jar:file:/opt/groovy/lib/groovy-3.0.4.jar!/groovy/grape/defaultGrapeConfig.xml: impossible to add configured child for ivy on class org.apache.ivy.plugins.resolver.FileSystemResolver: ivy pattern must be absolute: ?/.groovy/grapes/[organisation]/[module]/ivy-[revision].xml
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
/var/lib/jenkins/jobs/groovy-container/workspace/script.groovy: 2: unable to resolve class retrofit2.Retrofit
 @ line 2, column 1.
   @Grab("com.squareup.retrofit2:retrofit:2.8.1")
   ^

1 error

[keeganwitt]

Ah, OK. I'm able to reproduce it now with the -u 109:115 bit. I'm not sure yet the cause.

[mkutz]

I'll try to find out why Jenkins adds this and what options are there to avoid it.

[keeganwitt]

It's probably added so the permissions in the container are such that the volume mounted can be read. I'm not an Ivy expert, I'm trying to find if there's a Java prop you can set to tell it where to look instead, since the user it runs as won't have a home. If there's not, another option would be to provide your own Ivy XML settings (-Dgrape.config).

[mkutz]

So, I found out that UID 109 is mapped to user jenkins and GID 115 is group jenkins. So this basically gives permissions to the user to edit the files located in the workspace.

$ echo $HOME
/var/lib/jenkins

[keeganwitt]

It's clunky, but it works docker run -u 109:115 --rm -v "${pwd}:/home/groovy/scripts" -w /home/groovy/scripts groovy:latest groovy -D grape.config=/home/groovy/scripts/grapesConfig.xml grapesTest.groovy

grapesConfig.xml

<!-- same thing as https://github.com/apache/groovy/blob/master/src/resources/groovy/grape/defaultGrapeConfig.xml, but with ${user.home} replaced with /home/groovy -->
<ivysettings>
  <settings defaultResolver="downloadGrapes"/>
  <resolvers>
    <chain name="downloadGrapes" returnFirst="true">
      <filesystem name="cachedGrapes">
        <ivy pattern="/home/groovy/.groovy/grapes/[organisation]/[module]/ivy-[revision].xml"/>
        <artifact pattern="/home/groovy/.groovy/grapes/[organisation]/[module]/[type]s/[artifact]-[revision].[ext]"/>
      </filesystem>
      <ibiblio name="localm2" root="/home/groovy/.m2/repository/" checkmodified="true" changingPattern=".*" changingMatcher="regexp" m2compatible="true"/>
      <ibiblio name="jcenter" root="https://jcenter.bintray.com/" m2compatible="true"/>
      <ibiblio name="ibiblio" m2compatible="true"/>
    </chain>
  </resolvers>
</ivysettings>

I'm still trying to find a prop to override to avoid having to make your own config XML.

[keeganwitt]

Found it (though I don't see it anywhere in official documentation): docker run -u 109:115 --rm -v "${pwd}:/home/groovy/scripts" -w /home/groovy/scripts groovy:latest groovy -D user.home=/tmp grapesTest.groovy.

docker run -u 109:115 --rm -v "${pwd}:/home/groovy/scripts" -w /home/groovy/scripts groovy:test groovy -D user.home=/home/groovy grapesTest.groovy would work too, but only if I change the image to chmod /home/groovy so other users use it.

I maintain the Gradle image too, and there was a lot of back and forth there about volumes, users, and permissions. Some CI systems don't even give you the ability to run an image as another user, so in the end I gave up and just ran the image as root. Possibly that's what I should consider here. Let me mull it over a bit. I'm a bit concerned about breaking other users, but this seems like an important use case to support.

In the mean time, you can override to use /tmp (or anywhere else any user can write to).

[mkutz]

Nice. I made it work using the -D user.home workaround.

Thanks @keeganwitt.

[keeganwitt]

I've started a conversation on the dev list to see if folks have an opinion on how to proceed: https://lists.apache.org/thread.html/r5dd47fdbc2415688d8b27646c238a17e52016c423d222b60513805a8%40

[keeganwitt]

I forgot to mention this before, but another option would be to run as root (-u root). I've already symlinked the groovy and root user home directories. Running as root would probably be able to read whatever volume you're mounting, and will find the home already in the container, so then there'd be no need to override the home arg.

[mkutz]

Would do in my case, thanks for the hint. However, if your job produces some files, those would belong to root and Jenkins might not be able to read them in that case.

[keeganwitt]

Possibly, depends on how it's chmodded. I'm curious, is the sticky bit set on that volume in your case? And what's the chmod on the directory?

[mkutz]

I just tried it and here are my observations:

Using the agent definition

agent {
    docker {
      image 'groovy:3.0'
      args '-u root'
    }
}

The docker command executed by Jenkins is

docker run -t -d -u 109:115 -u root -w /var/lib/jenkins/jobs/groovy-container/workspace -v /var/lib/jenkins/jobs/groovy-container/workspace:/var/lib/jenkins/jobs/groovy-container/workspace:rw,z -v /var/lib/jenkins/jobs/groovy-container/workspace@tmp:/var/lib/jenkins/jobs/groovy-container/workspace@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** groovy:3.0 cat

So the additional -u parameter practically works as it is put behind the one set by Jenkins. However, this seems dangerous to me.

Running the script

#!/usr/bin/env groovy
@Grab(group='org.apache.commons', module='commons-lang3', version='3.7')
import org.apache.commons.lang3.SystemUtils

new File("newfile.txt").createNewFile()

println "Using Java ${SystemUtils.JAVA_VERSION}"

Is now working, but the created newfile.txt indeed belongs to root, as ls -al shows:

drwxr-xr-x 2 jenkins jenkins 4096 Jul 14 08:00 .
drwxr-xr-x 6 jenkins jenkins 4096 Jul 14 08:00 ..
-rw-r--r-- 1 root    root       0 Jul 14 08:00 newfile.txt
-rwxr-xr-x 1 jenkins jenkins  230 Jul 14 08:00 script.groovy

So, for me /tmp is the best of all workarounds we came up with untill now.

[keeganwitt]

Given that, I guess the fix that'd make the most sense would be to make the groovy user home writeable, so you can use it instead of /tmp (so you can take advantage of the grapes volume). Another nice thing about that approach that there shouldn't be any downsides for other users who don't have this use case.