search

mojohaus / rpm-maven-plugin

http://www.mojohaus.org/rpm-maven-plugin/
Other
56 stars 48 forks source link

Files copied to buildroot do not preserve symlinks #42

Closed Icedmang closed 8 years ago

Icedmang commented 8 years ago

We have a number of files checked into an SVN repository that contain symbolic links to x86_64 binaries. We are trying to package up these symbolic links as part of the package but the symbolic links are replaced with the actual contents of the files with the name of the symbolic link.

The CI biuld is done through Jenkins and I've confirmed the files are checked out of SVN as symbolic links, however, during the copy to tmp-buildroot, it appears the symbolic links are lost in this process.

[INFO] Copying files to /home/jenkins/jenkins/workspace/test/target/rpm/test/tmp-buildroot/opt/test

I can't find any information on this topic and whether or not there is an option to preserve symbolic links except for the parameter, but I don't want to recreate the symbolic links through configuration, I would like to package them as-is.

      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>rpm-maven-plugin</artifactId>
        <version>2.1.4</version>
        <extensions>true</extensions>
        <executions>
          <execution>
            <id>attach-rpm</id>
            <goals>
              <goal>attached-rpm</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <group>Applications/Engineering</group>
          <needarch>x86_64</needarch>
          <targeOS>Linux</targeOS>

          <defineStatements>
            <defineStatement>_unpackaged_files_terminate_build 0</defineStatement>
          </defineStatements>

          <mappings>
            <mapping>
              <directory>/opt/test/</directory>
              <configuration>false</configuration>
              <username>user</username>
              <groupname>user</groupname>

              <sources>
                <source>
                  <location>test/</location>
                  <noDefaultExcludes>true</noDefaultExcludes>
                  <targetArchitecture>x86_64</targetArchitecture>
                  <targetOSName>linux</targetOSName>
                </source>
              </sources>

            </mapping>
          </mappings>
        </configuration>
      </plugin>
dantran commented 8 years ago

java copy does not preserve symlinks until java 7 this plugin also does not use any java 7 specifics either.

So it is best to manage the sym link yourself via rpm-plugin configurations

Icedmang commented 8 years ago

Thanks Dan, I will have a look and try it and report back. (I was just looking at source code of the version 2.1.4 and just ended up at the org.codehaus.plexus.util.IOUtil class and was looking for a quick and dirty way to pass the NOFOLLOW_LINKS CopyOption https://docs.oracle.com/javase/tutorial/essential/io/copy.html)

edit: i just noticed you may have deleted your comment to try 2.1.5-SNAPSHOT.

dantran commented 8 years ago

😎 we need to fix up this plugin to support symlink which is possible now with minimal effort

dantran commented 8 years ago

@Icedmang could you give 2.1.5-SNAPSHOT deployed at sonatype a try or build this https://github.com/dantran/rpm-maven-plugin

I upgrade plexus-archiver to get symlink support.

Icedmang commented 8 years ago

@dantran 2.1.5-SNAPSHOT from Sonatype seems to work! (It took me a bit to get our Nexus configured properly and my pom.xml to properly access SNAPSHOTs from Sonatype). The symbolic links are preserved (and I have many)

I did notice an Exception in the output but I'm unsure of the impact (yet): [DEBUG] Cannot find ArtifactResolver with hint: project-cache-aware org.codehaus.plexus.component.repository.exception.ComponentLookupException: java.util.NoSuchElementException role: org.apache.maven.artifact.resolver.ArtifactResolver roleHint: project-cache-aware at org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:257) at org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:233) at org.apache.maven.shared.repository.DefaultRepositoryAssembler.contextualize(DefaultRepositoryAssembler.java:721) at org.sonatype.guice.plexus.lifecycles.PlexusLifecycleManager.contextualize(PlexusLifecycleManager.java:317) at org.sonatype.guice.plexus.lifecycles.PlexusLifecycleManager.manageLifecycle(PlexusLifecycleManager.java:292) at org.sonatype.guice.plexus.lifecycles.PlexusLifecycleManager.onProvision(PlexusLifecycleManager.java:148) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:108) at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:55) at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:68) at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1018) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.Scopes$1$1.get(Scopes.java:59) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:965) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1011) at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:961) at org.sonatype.guice.bean.locators.LazyBeanEntry.getValue(LazyBeanEntry.java:83) at org.sonatype.guice.plexus.locators.LazyPlexusBean.getValue(LazyPlexusBean.java:49) at org.sonatype.guice.bean.locators.EntryListAdapter$ValueIterator.next(EntryListAdapter.java:112) at org.apache.maven.plugin.assembly.archive.DefaultAssemblyArchiver.createArchive(DefaultAssemblyArchiver.java:181) at org.apache.maven.plugin.assembly.mojos.AbstractAssemblyMojo.execute(AbstractAssemblyMojo.java:436) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) at org.jvnet.hudson.maven3.launcher.Maven3Launcher.main(Maven3Launcher.java:117) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239) at org.jvnet.hudson.maven3.agent.Maven3Main.launch(Maven3Main.java:178) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at hudson.maven.Maven3Builder.call(Maven3Builder.java:136) at hudson.maven.Maven3Builder.call(Maven3Builder.java:71) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:328) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.NoSuchElementException at java.util.Collections$EmptyIterator.next(Collections.java:4189) at org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:253) ... 54 more

dantran commented 8 years ago

@Icedmang about the exception, I dont see it at my maven 3.3.3 java 8 production build

what do you have?

dantran commented 8 years ago

@Icedmang did you test your RPM?

Icedmang commented 8 years ago

@dantran I am using the following:

Then I tried using

Note that I am using the -X option and the above dump is a DEBUG log message so it's likely hidden? -X deploy site-deploy -Prun-its,coverage -Djacoco.outputDir=${WORKSPACE}/target

I get the same result for either Maven version 3.0.5 or 3.3.3 - the DEBUG message with the stack trace shows up. It will not show up without the -X option.

I am currently testing my packaged RPM and will report back

Icedmang commented 8 years ago

There does not appear to be anything functionally different in the .rpm vs. the .tar.gz that we currently create... There is one caveat: the binary files have a different size in the .rpm vs. the .tar.gz. However, if I strip all binaries, the file sizes between the .rpm and .tar.gz are the same and there is no diff. The binaries in the .rpm are the ones that changed from their original form during packaging.

e.g.

# The file sizes are different for some reason.  The original source filesize is 1486163
[root@centos opt]# ls -l test-rpm/bin/binfile.x64Linux test-tarball/bin/binfile.x64Linux
-rwxrwxr-x. 1 user group 1478254 Feb 16 16:39 test-rpm/bin/binfile.x64Linux
-rw-r--r--. 1 user group 1486163 Feb 16 09:56 test-tarball/bin/binfile.x64Linux

# Permissions not set correctly in the tarball, so I set it manually
[root@centos opt]# chmod 775 test-tarball/bin/binfile.x64Linux
[root@centos opt]# ls -l test-rpm/bin/binfile.x64Linux test-tarball/bin/binfile.x64Linux
-rwxrwxr-x. 1 user group 1478254 Feb 16 16:39 test-rpm/bin/binfile.x64Linux
-rwxrwxr-x. 1 user group 1486163 Feb 16 09:56 test-tarball/bin/binfile.x64Linux

# I stripped the remaining symbols from the binaries
[root@centos opt]# strip test-rpm/bin/binfile.x64Linux test-tarball/bin/binfile.x64Linux

# Now the filesizes are the same
[root@centos opt]# ls -l test-rpm/bin/binfile.x64Linux test-tarball/bin/binfile.x64Linux
-rwxrwxr-x. 1 user group 1388056 Feb 16 17:04 test-rpm/bin/binfile.x64Linux
-rwxrwxr-x. 1 user group 1388056 Feb 16 17:04 test-tarball/bin/binfile.x64Linux

# Now there is no difference :/
[root@centos opt]# diff test-rpm/bin/binfile.x64Linux test-tarball/bin/binfile.x64Linux
[root@centos opt]#

While I don't think this is a significant problem, I'm curious as to why the files are different while packaging the RPM

dantran commented 8 years ago

Glad it works out ( somewhat). I can't explain what you see. Must have something do with the rpmbuild during packaging

Icedmang commented 8 years ago

I'm going to close this issue as it addresses my original question. Thank you so much for your help on this. You were very quick and responsive!

I'm looking forward to an official 2.1.5 release :)

dantran commented 8 years ago

@Icedmang please help test 2.1.5 staged at https://oss.sonatype.org/content/groups/staging

Icedmang commented 8 years ago

@dantran Absolutely. I will report back in a few hours.

Icedmang commented 8 years ago

@dantran I am building the RPM now with version 2.1.5 from Staging. In the meantime, regarding the file size differences I mentioned earlier, I think it's related to what's described on this page

Is there a way through the plugin to include the following in the spec file? I could not find it %global __os_install_post %{nil}

dantran commented 8 years ago

try defineStatements. here is mine

<defineStatements>
          <defineStatement>_unpackaged_files_terminate_build 0</defineStatement>
          <defineStatement>_tmppath ${project.build.directory}/rpm/tmp</defineStatement>
</defineStatements>
Icedmang commented 8 years ago

@dantran Version 2.1.5 of rpm-maven-plugin works as expected. It still has different file sizes which I believe is due to reason I mentioned above and requiring the %global __os_install_post %{nil}. I don't believe the %define's will work here. I already have the following in place before I logged this issue:

        <defineStatements>
          <defineStatement>_unpackaged_files_terminate_build 0</defineStatement>
        </defineStatements>
dantran commented 8 years ago

maybe you can set it at your ~/.rpmrc ?

Icedmang commented 8 years ago

@dantran I apologize, I was wrong to state that it wouldn't work in the defineStatement as you mentioned. I decided I should actually test it and you're right, it works as you said. Mine looks like:

          <defineStatements>
            <defineStatement>_unpackaged_files_terminate_build 0</defineStatement>
            <defineStatement>__os_install_post %{nil}</defineStatement>
          </defineStatements>

I didn't want to include it in the ~/.rpmrc because I would have to do that to all the Jenkins nodes we have configured. I like it contained to the .rpm and not the user.

What should we do next so 2.1.5 is promoted from Staging to a Release?

dantran commented 8 years ago

nice, glad it works out. I need to announce a vote by tonight at mojohaus dev, it takes 3 days for the vote to pass

dantran commented 8 years ago

https://groups.google.com/forum/#!topic/mojohaus-dev/cQjCGaEGtNw

Icedmang commented 8 years ago

Thanks again, @dantran