Packaging fails with multi module project

nicenemo commented 7 years ago

Plugin name and version

   <pluginManagement>
            <plugins>
   ...
                <plugin>
                    <groupId>com.microsoft.azure</groupId>
                    <artifactId>azure-functions-maven-plugin</artifactId>
                    <version>0.1.6</version>
                </plugin>
            </plugins>
        </pluginManagement>

Plugin configuration in your `pom.xml`

...
    <build>
...
        <plugins>
            <plugin>
                <groupId>com.microsoft.azure</groupId>
                <artifactId>azure-functions-maven-plugin</artifactId>
                <configuration>
                    <resourceGroup>java-functions-group</resourceGroup>
                    <appName>${functionAppName}</appName>
                    <region>${functionAppRegion}</region>
                    <appSettings>
                        <property>
                            <name>FUNCTIONS_EXTENSION_VERSION</name>
                            <value>beta</value>
                        </property>
                    </appSettings>
                </configuration>
                <executions>
                    <execution>
                        <id>package-functions</id>
                        <goals>
                            <goal>package</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            ...

Expected behavior

As a developer in a project using Azure Functions I want to use the Java Azure SDK with components from private repositories.

mvn package works.
mvn azure-functions:run works.
I want to use a self created library from my Azure Function
I want to use proprietary libraries within that function or another function. These libraries cannot be deployed to Maven Central or another public repository for legal reasons.

Actual behavior

Both fail with a complaint of missing a class from a jar in a private repo.

Steps to reproduce the problem

In an Azure Functions project mvn package fails if you refer components in private repositories.

Normal package works

If you remove azure-functions-maven-plugin plugin from the pom.xml a normal package works. So it is for sure something in the azure-functions-maven-plugin

Reproduce

Create an Azure Function with the Azure Functions Archetype: https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-first-java-maven
create another java project that just produces a jar with an Hello World function.
install the jar in your local .m2 cache
Refer to it from the Function project pom file.
Use the Hello World from within the function
mvn compile and mvn test should already work
observe that mvn package does not work.

Private repo

After Googleing I noticed that for package a repository should be available and that packaging does not work with only the local cache in ~/.m2

I "Fixed" this by publishing to a local repository from the library project and refering to that from the Azure Function project.

Publish to "remote" repository by adding the following to the pom.xml of the library project:

<plugin>
  <artifactId>maven-deploy-plugin</artifactId>
  <version>2.8.2</version>
  <configuration>

 <altDeploymentRepository>internal.repo::default::file:${project.basedir}/../mvn-repo</altDeploymentRepository>
  </configuration>
</plugin>

Do mvn clean package install deploy in the library project folder.
Verify that the library is deployed to the new local file repo.
clear my .m2/repository
Go to the functions project project
Add the following to the pom.xml file

<repositories>
    <!--other repositories if any-->
    <repository>
        <id>project.local</id>
        <name>project</name>
        <url>file:${project.basedir}/../mvn-repo</url>
    </repository>
</repositories>

do mvn clean compile test
Observe that the tests are run (If you do not add the repository. The code will fail to compile/test if your .m2 cache is empty too.)
Do mvn package --> fails with library classes that cannot be found
Try mvn azure-functions:run It tries to package first and fails.

Error:

INFO] Reflections took 78 ms to scan 1 urls, producing 1 keys and 1 values
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 38.051 s
[INFO] Finished at: 2017-11-09T15:41:27+01:00
[INFO] Final Memory: 43M/286M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.microsoft.azure:azure-functions-maven-plugin:0.1.6:package (package-functions) on project helloworldFunction: Execution package-functions of goal com.microsoft.azure:azure-functions-mave
[ERROR] -----------------------------------------------------
[ERROR] realm =    plugin>com.microsoft.azure:azure-functions-maven-plugin:0.1.6
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] = file:/C:/Users/hkruse/.m2/repository/com/microsoft/azure/azure-functions-maven-plugin/0.1.6/azure-functions-maven-plugin-0.1.6.jar
[... <<<cut>>> ...]
[ERROR] urls[82] = file:/C:/Users/hkruse/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar
[ERROR] Number of foreign imports: 1
[ERROR] import: Entry[import  from realm ClassRealm[maven.api, parent: null]]

Side note: using Github as repo

We looked into using hosted Nexus as a repo but did not do that for cost reasons.

We finally want to use either Github site plugin or Wagon-git I think wagon-git is finally the way to go because the site plugin is more for publishing sites. I did not want to complicate matters more yet by using git based repo plugins now. So I used the local file system to try it out.

xscript commented 7 years ago

Thanks for reporting this issue! I will try to reproduce and get back to you later.

nicenemo commented 6 years ago

I suspect that the plugin only supports one jar, and one jar only. I turned my project into multi module project now. That is actually the scenario that should work. If I make a "fat jar" myself from my modules would it work then?

xscript commented 6 years ago

You can try maven-shade-plugin, which is the recommended approach right now for extra dependencies. Sample configuration as below:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <configuration>
        <artifactSet>
            <excludes>
                <exclude>com.microsoft.azure:azure-functions-java-core:jar:</exclude>
            </excludes>
            <includes>
                ...
            </includes>
        </artifactSet>
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
        </execution>
    </executions>
</plugin>

nicenemo commented 6 years ago

Thanks looking into that today (CET timezone).

It is a "partial success":

mvn clean package no longer breaks but:
mvn azure-functions:package breaks.
mvn azure-functions:run breaks.

I get an error about missing or incorrect parameters. They are actually the same as on a generated project.

full redacted build log

Library names are changed.

java (feature/NEW-195-azurefunctionplugintrouble=) $ mvn clean azure-functions:package
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] My Java Projects
[INFO] My ViewModel
[INFO] My Special Library
[INFO] My Other Library  
[INFO] My Function  
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building My Java Projects 1.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ parent-project ---
[INFO]
[INFO] --- azure-functions-maven-plugin:0.1.6:package (default-cli) @ parent-project ---
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] My Java Projects .................................. FAILURE [  3.292 s]
[INFO] My ViewModel ..................................... SKIPPED
[INFO] My Special Library .............................. SKIPPED
[INFO] My Other Library   ..................................... SKIPPED
[INFO] My Function   ................................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.460 s
[INFO] Finished at: 2017-11-16T12:34:35+01:00
[INFO] Final Memory: 34M/232M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.microsoft.azure:azure-functions-maven-plugin:0.1.6:package (default-cli) on project parent-project: The parameters 'resourceGroup', 'appName' for goal com.microsoft.azure:azure-functions-maven-plugin:0.1.6:package are missing or invalid -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginParameterException

xscript commented 6 years ago

Forgot to mention that azure-functions:package should be after mvn package, because it parses the generated JAR file and generate function configurations for you. So in your pom.xml, typical settings should be:

<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <configuration>
            <artifactSet>
                <excludes>
                    <exclude>com.microsoft.azure:azure-functions-java-core:jar:</exclude>
                </excludes>
                <includes>
                    ...
                </includes>
            </artifactSet>
        </configuration>
        <executions>
            <execution>
                <phase>package</phase>
                <goals>
                    <goal>shade</goal>
                </goals>
            </execution>
        </executions>
    </plugin>
    <plugin>
        <groupId>com.microsoft.azure</groupId>
        <artifactId>azure-functions-maven-plugin</artifactId>
        <configuration>
            <resourceGroup>java-functions-group</resourceGroup>
            <appName>${functionAppName}</appName>
            <region>${functionAppRegion}</region>
            <appSettings>
                <property>
                    <name>FUNCTIONS_EXTENSION_VERSION</name>
                    <value>beta</value>
                </property>
            </appSettings>
        </configuration>
        <executions>
            <execution>
                <id>package-functions</id>
                <goals>
                    <goal>package</goal>
                </goals>
            </execution>
        </executions>
    </plugin>
</plugins>

Also from your error message, seems that you forgot to configure resourceGroup and appName in azure-functions-maven-plugins.

The parameters 'resourceGroup', 'appName' for goal com.microsoft.azure:azure-functions-maven-plugin:0.1.6:package are missing or invalid -> [Help 1]

nicenemo commented 6 years ago

Looked into the build order config stuff you mentioned above.
I did not mention that I did a manual mvn package first
No the resource group and app name are configured .(The version of the plugin comes from the parent pom):

<!-- ... -->
<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <configuration>
          <artifactSet>
            <excludes>
                <exclude>com.microsoft.azure:azure-functions-java-core:jar:</exclude>
            </excludes>
            <includes>
               <include>no.identification:my-lib0:jar:</include>
               <include>no.identification:my-lib1:jar:</include>
               <include>no.identification:my-lib2:jar:</include>
            </includes>
          </artifactSet>
        </configuration>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
          </execution>
        </executions>
  </plugin>
 <plugin>
        <groupId>com.microsoft.azure</groupId>
        <artifactId>azure-functions-maven-plugin</artifactId>
        <executions>
          <execution>
            <id>package-functions</id>
            <goals>
              <goal>package</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <resourceGroup>java-functions-group</resourceGroup>
          <appName>myFunction-20171109104637551</appName>
          <region>westeurope</region>
          <appSettings>
            <property>
              <name>FUNCTIONS_EXTENSION_VERSION</name>
              <value>beta</value>
            </property>
          </appSettings>
        </configuration>
      </plugin>
<!--  ... -->

Looks similar to yours?

nicenemo commented 6 years ago

updated the above comment.

xscript commented 6 years ago

Your pom.xml looks all right. But I do see that you run mvn clean azure-functions:package. The clean goal will delete the target folder.

nicenemo commented 6 years ago

Sorry that should read mvn clean package azure-functions:package

nicenemo commented 6 years ago

Ok good news. I have it working again. But I am sorry to tell that the azure-maven-functions-plugin does not support multi module projects.

Workaround:

Exclude the Azure functions project from the multi module project.
Build the multi module project with mvn clean package install No need for nexus or alike yet. This installs it in the local package cache..
Build the azure functions project separately. Then you can package it up and run it on the local simulator. I am using the shade plugin to build an Über jar.

xscript commented 6 years ago

Thanks @nicenemo for such detailed information! I will look into it. We will fix it once we confirm the issue.

nicenemo commented 6 years ago

It runs in integration test but the function freezes when run from the simulator when called. See also my comment https://github.com/Azure/azure-functions-java-worker/issues/35#issuecomment-345047272 Will look into getting the dependencies bundled correctly today (CET timezone).

nicenemo commented 6 years ago

Suggestion: https://github.com/xscript work together with https://github.com/JunyiYi to get the docs and samples improved.

nicenemo commented 6 years ago

Shade plugin does not work with signed jars such as Bouncy Castle. If I add some more directions for the Shade can get it compiled and integration tested if I turn off the azure-maven-plugin. Building with the plugins gives warnings about class not found and it does freeze when run as before.

JunyiYi commented 6 years ago

Shade plugin definitely works with signed jars. You just need to know how to configure it:

<excludes>
  <exclude>META-INF/*.SF</exclude>
  <exclude>META-INF/*.DSA</exclude>
  <exclude>META-INF/*.RSA</exclude>
</excludes>

nicenemo commented 6 years ago

We filed a ticket on our Azure subscription on this issue on work with Microsoft from there.

I did as you suggested. My maven config.

 <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.1.0</version>
        <configuration>
          <artifactSet>
            <excludes>
              <exclude>com.microsoft.azure:azure-functions-java-core:jar:</exclude>
               <exclude>META-INF/*.SF</exclude>
               <exclude>META-INF/*.DSA</exclude>
               <exclude>META-INF/*.RSA</exclude>
            </excludes>
            <includes>
              <include>*:*</include>
            </includes>
          </artifactSet>
        </configuration>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

This will produce the following error when mvn verify is run.

[INFO] --- maven-resources-plugin:3.0.2:copy-resources (copy-resources) @ my-function ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-failsafe-plugin:2.20.1:integration-test (integration-tests) @ my-function ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running no.identification.functions.FunctionIT
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.922 s <<< FAILURE! - in my.functions.functions.FunctionIT
[ERROR] integrationTest(my.functions.functions.FunctionIT)  Time elapsed: 2.656 s  <<< ERROR!
java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
        at my.functions.FunctionIT.test(FunctionIT.java:37)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   FunctionIT.test:37 ▒ Security Invalid signature file digest for Manifes...
[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-failsafe-plugin:2.20.1:verify (integration-tests) @ di-signpdf ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 19.308 s
[INFO] Finished at: 2017-11-20T10:26:20+01:00
[INFO] Final Memory: 47M/464M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:2.20.1:verify (integration-tests) on project my-function: There are test failures.
[ERROR]
[ERROR] Please refer to C:\Users\hkruse\git\my\java\my-function\target\failsafe-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

nicenemo commented 6 years ago

Made a sample project that is simpler. And filed a new issue.

microsoft / azure-maven-plugins