Open mikezerosix opened 2 months ago
@mikezerosix - I believe you just want to perform fire and forget style deployments and not wait for the deployments to complete. The default option is that so that you can include triggering the automated testing after knowing that deployment is complete.
If you do not want that - you can change the deploy options. Just use -Dapigee.options=async
or update the apigee.options
value in the pom profile to <apigee.options>async</apigee.options>
That will just run the import and deploy command and not poll for the deployment to complete. The same is documented as well
@ssvaidyanathan Thanks. I am aware of the async option. But I do want to wait for confirmation, I just do not run deployments sequentially. I have maven multimodule project with 20 sub modules, each is apigee flow or proxy. if I run the maven normally mvn install
it takes maven almost an hour to run the build as it runs each proxy deployment one at time wating for one to complete before running the next. What I do want to, is to run Maven with mvn -T 20
which would run maven in 20 threads, running all 20 submodules at once and it would complete in 2 minutes (time of single polled deployment)
But when I try to run maven in parallel (even just "-T 2", cutting build time to half), I get errors from this plugin.
[ERROR] Failed to execute goal io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.5.0:deploy (deploy-bundle) on project shared-request-player: MojoExecutionException: com.google.api.client.http.HttpResponseException: 404 Not Found
[ERROR] {
[ERROR] "error": {
[ERROR] "code": 404,
[ERROR] "message": "generic::not_found: organizations/xxx/sharedflows/shared-xxx/revisions/218 not found",
So it looks like this plugin does not support multithreading and it would be amazing if it did.
Or I would love have ability to run async deploy and then run separate process to poll projects deployment
First run mvn apigee-edge-maven-plugin:deploy -Dapigee.options=async
Which would run the 20 sub modules each with proxy in less than minute.
Then run mvn apigee-edge-maven-plugin:POLL-ASYNC-DEPLOYMENTS
Which would poll and untill each of the previously deployed proxies and shared flows report having been deployes or report failure. This should complete in 2-3 minutes assuming Apigee itself runs deployments paraller.
Problem here would be that this would need to persist information between these two runs (like mvn release plugin) so it knows which revisions to poll for. So it does not get confused from 3rd party deploys.
@mikezerosix - Thank you for the detailed summary. Great pointers and suggestions. Let me see what can be done. Will try them out. This is the first time someone's asking this. So thats great 😄
I will need a favor - if I end up implementing, I will need you to help test it for me and see if it works as expected. We can fine tune as you provide more inputs.
I created sample project for the issue. I have not tested it that it actually runs, but it at least demonstrates the structure https://gitlab.com/mike06/public/apigee-paraller-deploy-sample
Failure is random, but on thread value 4 or higher it should be almost 100%.
Brilliant.. I will use this to test it out
I updated the sample little bit. I also added the Maven warning about threads
[WARNING] [WARNING] Your build is requesting parallel execution, but this [WARNING] project contains the following plugin(s) that have goals not [WARNING] marked as thread-safe to support parallel execution. [WARNING] While this /may/ work fine, please look for plugin updates [WARNING] and/or request plugins be made thread-safe. [WARNING] If reporting an issue, report it against the plugin in [WARNING] question, not against Apache Maven. [WARNING] [WARNING] The following plugins are not marked as thread-safe in proxynn: [WARNING] io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.5.0 [WARNING] [WARNING] Enable debug to see precisely which goals are not marked as thread-safe. [WARNING] *****
@mikezerosix - I cloned your repo I had to make some changes for example
<apigee.apiversion>v1</apigee.apiversion>
to main pom.xml<module>proxy18</module>
to the list of modules in the main pom.xml<BasePath>
in Proxy04
module. apigee.googletoken.email
and oauthClientId
from the profile in the main pom.xmlI ran the test with -T4C
and also with -T10C
, I saw errors popping out mainly 503s from the Apigee APIs (I think its rate limiting) when the plugin is polling for the Deployment state
However, when I ran the same test with <apigee.options>async</apigee.options>
, it worked without any issues
The entire build ran in 10 seconds
[INFO] Reactor Summary for apigee-maven-parallel 0.1.0-SNAPSHOT:
[INFO]
[INFO] apigee-maven-parallel .............................. SUCCESS [ 0.910 s]
[INFO] proxy01 ............................................ SUCCESS [ 4.447 s]
[INFO] proxy02 ............................................ SUCCESS [ 4.220 s]
[INFO] proxy03 ............................................ SUCCESS [ 8.624 s]
[INFO] proxy04 ............................................ SUCCESS [ 5.868 s]
[INFO] proxy05 ............................................ SUCCESS [ 7.956 s]
[INFO] proxy06 ............................................ SUCCESS [ 3.782 s]
[INFO] proxy07 ............................................ SUCCESS [ 5.510 s]
[INFO] proxy08 ............................................ SUCCESS [ 7.240 s]
[INFO] proxy09 ............................................ SUCCESS [ 4.837 s]
[INFO] proxy10 ............................................ SUCCESS [ 3.058 s]
[INFO] proxy11 ............................................ SUCCESS [ 5.210 s]
[INFO] proxy12 ............................................ SUCCESS [ 6.941 s]
[INFO] proxy13 ............................................ SUCCESS [ 6.275 s]
[INFO] proxy14 ............................................ SUCCESS [ 7.607 s]
[INFO] proxy15 ............................................ SUCCESS [ 6.581 s]
[INFO] proxy16 ............................................ SUCCESS [ 9.361 s]
[INFO] proxy17 ............................................ SUCCESS [ 2.542 s]
[INFO] proxy18 ............................................ SUCCESS [ 3.417 s]
[INFO] proxy19 ............................................ SUCCESS [ 8.312 s]
[INFO] proxy20 ............................................ SUCCESS [ 8.954 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10.383 s (Wall Clock)
[INFO] Finished at: 2024-05-29T17:52:04-07:00
[INFO] ------------------------------------------------------------------------
I think you can ignore the threadsafe warning. Thats not really impacting the plugin.
With the default "override" option, the plugin is polling the deployments API and if the response from the Apigee API is 503, then the plugin just exits and marks the build as Failed.. But the proxy is eventually getting deployed.
What I tried is pipe the mvn response to a log file
mvn clean install -P destination1 -Dbearer="$(gcloud auth print-access-token)" -T10C > a.log
Then looked at the log in detail
@ssvaidyanathan I added your fixes to repo. Thanks.
I forgot to mention that parallel deploy run might not fail on empty apigee. Because errors were about revisions. When the deploy actually overrides the existing revisions, it should fail. I have once successfull ran maven parallel with -T1C on 2 core cpu for 16 proxies and flows. But that could just been because the other core was buzy and never ran the second thread. As re-running it failed. Also most times error seems to come from flows, but proxies do fail also. If you can not reproduce the problem with this sample, I can add flows to it.
I know that warning is not critical, I added it just for information.
I would be interested to know how long does that take to run in default override option, in single thread. It should be something like 40 minutes, even though the proxies have no policies. I have no Apigee that I can run this sample myself as I do not want to mess around in customer's enviroment.
We tried asýnc option and it run the project without problems. But that async does not meet our requirements. We need to assert the proxies are successfully deployed. Using async the solution would be just horrible; unreliable and a lot of work to create and maintain
If we used async, we would
We could do the work, but the fact that it would be unreliable is a deal breaker.
Got it.. The issue with "override" is that the Apigee APIs are rate limiting.. So you really cannot do anything in the plugin. I tried the async multiple times so that new revisions of the proxy gets created. It always worked.
But I was always having issues with "override" since the Apigee APIs were rate limiting and that was causing failures. Dont think adding flows or policies would matter here.
Let me think about your mvn apigee-edge-maven-plugin:POLL-ASYNC-DEPLOYMENTS
option. The issue here is - its the same set of GET API that I will have to call for polling which is used with the "override". So not sure how this will help.
@ssvaidyanathan About ratelimit, what about adding plugin option for poll rate ? At what rate does the plugin now poll Apigee ? Does Apigee give specific error on rate limit exceeded, so could pluging throttle & wait when receiving rate limit exceed error and resume polling ?
There is a throttling trick I have used in polling, instead of running immediately and sleeping for pollRate. I start the poll on
even clock seconds of the pollRate. Setting different projects pollRate to prime numbers, you can get five ticks that never tick at same time.
For example is I would set 4 sub modules
In Java language:
Instead of doing
sleet(31) ...
Might cause all pols to tick at same time even if rate is different
But doing:
cronSchedule("0/31 * * * * ?") ...
Guarantees that with 20 divided to prime pollRates the parallel polls to only ever run 4 polls per 3 seconds and 20 polls a minute.
@mikezerosix - did u close this issue by mistake?
yes, I closed by mistake
@mikezerosix - when I run your setup, I only see 503s and no 404s Are you getting both 503s and 404s? or just 404s?
@mikezerosix - Just released v2.5.1 Can you update your pom to point to the above version and re-run your cases. I ran it a few times and it worked for me
@ssvaidyanathan I was consistently getting 404 error when they plugin tried to retrieve the revisions for sharedflow or proxy.
I updated the Gitlab sample to 2.5.1 and I updated the real project to 2.5.1. I tried to run the real project with plugin 2.5.1 and mvn run parameter -T5C, which for the 2 core runner machine would have tried to run 2*5=10 threads. That gave the same 404 erro on trying to retrieve some proxy's revisions.
Running with -T1C that should run 21=2 threads succeeded and reduced total runtime by half as expected. Running with -T2C that should run 22=4 threads failed on last sharedflow with 404 on revisions. Otherwise that would had reduced the total runtime by factor of 4.
I only ran -T1C (=2 threads) once, so it might not be stable. I have once succeeded running that o version 2.5.0, which succeeds something like 1 out of 10 times.
On both 2.5.0 and 2.5.1 errors are always something like
[ERROR] Failed to execute goal io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.5.0:deploy (deploy-bundle) on project shared-request-player: MojoExecutionException: com.google.api.client.http.HttpResponseException: 404 Not Found
[ERROR] {
[ERROR] "error": {
[ERROR] "code": 404,
[ERROR] "message": "generic::not_found: organizations/xxx/sharedflows/shared-xxx/revisions/218 not found",
Either on sharedflow or proxy.
@mikezerosix - I never got a 404 Can you try running on the GitLab code you shared. We can test using that set of proxies
@ssvaidyanathan Sorry, like I said earlier, I can not run the sample on Apigee. As Apigee instances require (very expensive) license I do not have playground Apigee instance. I only have real customer Apigee instance and I can not clutter that with the sample. I ran the real customer project with 20 mixed sharedflows and proxies using plugin v2.5.1 to the Apigee with above results.
@mikezerosix - then its difficult for me to fix this issue. From my tests, its working as expected as it pushed all the proxies multiple times. Below are some file runs that worked for me debug3.log debug5.log
@ssvaidyanathan I got permission to run my sample in customer's Apigee. I let you know the result when I can run it.
@ssvaidyanathan Amazing work. It works like charm. Can you do same fix for sharedflows too now ?
Result:
root .................................. SUCCESS [ 3.205 s]
proxy01 ............................................ SUCCESS [02:23 min]
proxy02 ............................................ SUCCESS [01:31 min]
proxy03 ............................................ SUCCESS [01:30 min]
proxy04 ............................................ SUCCESS [01:30 min]
proxy05 ............................................ SUCCESS [01:29 min]
proxy06 ............................................ SUCCESS [01:29 min]
proxy07 ............................................ SUCCESS [01:29 min]
proxy08 ............................................ SUCCESS [01:30 min]
proxy09 ............................................ SUCCESS [01:31 min]
proxy10 ............................................ SUCCESS [01:30 min]
proxy11 ............................................ SUCCESS [01:29 min]
proxy12 ............................................ SUCCESS [01:28 min]
proxy13 ............................................ SUCCESS [01:31 min]
proxy14 ............................................ SUCCESS [01:29 min]
proxy15 ............................................ SUCCESS [01:29 min]
proxy16 ............................................ SUCCESS [01:29 min]
proxy17 ............................................ SUCCESS [01:29 min]
proxy18 ............................................ SUCCESS [01:30 min]
proxy19 ............................................ SUCCESS [01:30 min]
proxy20 ............................................ SUCCESS [01:29 min]
------------------------------------------------------------------------
BUILD SUCCESS
------------------------------------------------------------------------
Total time: 31:01 min
root .................................. SUCCESS [ 3.058 s]
proxy01 ............................................ SUCCESS [ 2.850 s]
proxy02 ............................................ SUCCESS [ 1.587 s]
proxy03 ............................................ SUCCESS [ 2.319 s]
proxy04 ............................................ SUCCESS [ 2.175 s]
proxy05 ............................................ SUCCESS [ 2.225 s]
proxy06 ............................................ SUCCESS [ 1.538 s]
proxy07 ............................................ SUCCESS [ 1.526 s]
proxy08 ............................................ SUCCESS [ 1.491 s]
proxy09 ............................................ SUCCESS [ 1.617 s]
proxy10 ............................................ SUCCESS [ 1.608 s]
proxy11 ............................................ SUCCESS [ 2.273 s]
proxy12 ............................................ SUCCESS [ 2.255 s]
proxy13 ............................................ SUCCESS [ 1.520 s]
proxy14 ............................................ SUCCESS [ 2.240 s]
proxy15 ............................................ SUCCESS [ 2.857 s]
proxy16 ............................................ SUCCESS [ 1.516 s]
proxy17 ............................................ SUCCESS [ 2.241 s]
proxy18 ............................................ SUCCESS [ 2.843 s]
proxy19 ............................................ SUCCESS [ 1.696 s]
proxy20 ............................................ SUCCESS [ 2.916 s]
------------------------------------------------------------------------
BUILD SUCCESS
------------------------------------------------------------------------
Total time: 44.519 s
[INFO] root .................................. SUCCESS [ 3.111 s]
proxy01 ............................................ SUCCESS [ 50.535 s]
proxy02 ............................................ SUCCESS [ 48.935 s]
proxy03 ............................................ SUCCESS [01:52 min]
proxy04 ............................................ SUCCESS [ 48.934 s]
proxy05 ............................................ SUCCESS [01:52 min]
proxy06 ............................................ SUCCESS [01:52 min]
proxy07 ............................................ SUCCESS [ 50.268 s]
proxy08 ............................................ SUCCESS [ 48.627 s]
proxy09 ............................................ SUCCESS [01:53 min]
proxy10 ............................................ SUCCESS [ 58.232 s]
proxy11 ............................................ SUCCESS [01:10 min]
proxy12 ............................................ SUCCESS [01:08 min]
proxy13 ............................................ SUCCESS [01:10 min]
proxy14 ............................................ SUCCESS [ 59.356 s]
proxy15 ............................................ SUCCESS [01:00 min]
proxy16 ............................................ SUCCESS [ 59.079 s]
proxy17 ............................................ SUCCESS [01:08 min]
proxy18 ............................................ SUCCESS [ 58.854 s]
proxy19 ............................................ SUCCESS [ 57.502 s]
proxy20 ............................................ SUCCESS [ 57.626 s]
------------------------------------------------------------------------
BUILD SUCCESS
------------------------------------------------------------------------
Total time: 03:01 min (Wall Clock)
Note the total on wall clock as those deploys are are paraller.
@ssvaidyanathan And because why not ? Running -T10C also succeed and in 2 minutes wall clock. Again, amazing work, thank you. I hope you can add this fix to work with sharedflows too. As in real project is mix of shared flows and proxies.
@mikezerosix - Thats great news. Thanks for testing them out. Appreciate it.
Just to be sure,
The code change should include the change for sharedflows as well. As its the same function for both. Just that you need to pass <apigee.apitype>sharedflow</apigee.apitype>
in the Maven profile
@ssvaidyanathan
I added 10 sharedflows to the Sample project. They seem to working. I might add another 10 later when I have more time. But I think 10 + 20 is pretty good sample.
Thats awesome!! Keep me posted. We can close this issue after your tests.
@ssvaidyanathan Unfortunately the real project still fails. Still with 404 and
it is using io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.5.1:deploy
There should not be any other difference between the real project and sample than that real project had more content on the proxies and shared flows than the sample.
Real project always fails on sharedflows, never on proxies. Were both proxies and sharedflows supposed to be fixed for parallel run in plugin version 2.5.1 or was it just proxies ? And were you going to do yet another release where you add the same fix to sharedflows deploy too ?
No - no other change needed for sharedflows.
If you are sure that its failing only for sharedflows, can you confirm that for those modules if <apigee.apitype>sharedflow</apigee.apitype>
is set correctly
It is and project deploys fine sequentially, without -T mvn run parameter.
Any chance you can share the log file?
I am working on it. It will take me a while to check and anonymize it.
just mvn -X
or is there some special debug option for the plugin ?
Lets run it without "-X" for now.
What you could do is - since the sharedflows are the ones that are failing. You can comment out the proxy modules and just run the sharedflow modules.
Once you have that, please share it here. Make sure you go through it and not share anything sensitive
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:06 min (Wall Clock)
[INFO] Finished at: 2024-06-07T23:28:00Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.5.1:deploy (deploy-bundle) on project External-Reporting: MojoExecutionException: com.google.api.client.http.HttpResponseException: 404 Not Found
[ERROR] {
[ERROR] "error": {
[ERROR] "code": 404,
[ERROR] "message": "generic::not_found: resource organizations/xxxx/apis/yyyy/revisions/155 not found",
[ERROR] "status": "NOT_FOUND",
[ERROR] "details": [
[ERROR] {
[ERROR] "@type": "type.googleapis.com/google.rpc.RequestInfo",
[ERROR] "requestId": "17594140766235609332"
[ERROR] }
[ERROR] ]
[ERROR] }
[ERROR] }
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.5.1:deploy (deploy-bundle) on project External-Reporting:
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:375)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:210)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:195)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
at java.lang.Thread.run (Thread.java:840)
Caused by: org.apache.maven.plugin.MojoExecutionException:
at io.apigee.buildTools.enterprise4g.mavenplugin.DeployMojo.processHelpfulErrorMessage (DeployMojo.java:332)
at io.apigee.buildTools.enterprise4g.mavenplugin.DeployMojo.execute (DeployMojo.java:317)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:210)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:195)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
at java.lang.Thread.run (Thread.java:840)
Caused by: java.io.IOException: com.google.api.client.http.HttpResponseException: 404 Not Found
{
"error": {
"code": 404,
"message": "generic::not_found: resource organizations/xxxx/apis/yyyy/revisions/155 not found",
"status": "NOT_FOUND",
"details": [
{
"@type": "type.googleapis.com/google.rpc.RequestInfo",
"requestId": "17594140766235609332"
}
]
}
}
at io.apigee.buildTools.enterprise4g.rest.RestUtil.activateBundleRevision (RestUtil.java:852)
at io.apigee.buildTools.enterprise4g.rest.RestUtil.activateBundleRevision (RestUtil.java:802)
at io.apigee.buildTools.enterprise4g.mavenplugin.DeployMojo.doActivateBundle (DeployMojo.java:234)
at io.apigee.buildTools.enterprise4g.mavenplugin.DeployMojo.execute (DeployMojo.java:297)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:210)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:195)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
at java.lang.Thread.run (Thread.java:840)
Caused by: com.google.api.client.http.HttpResponseException: 404 Not Found
{
"error": {
"code": 404,
"message": "generic::not_found: resource organizations/xxxx/apis/yyyy/revisions/155 not found",
"status": "NOT_FOUND",
"details": [
{
"@type": "type.googleapis.com/google.rpc.RequestInfo",
"requestId": "17594140766235609332"
}
]
}
}
at com.google.api.client.http.HttpRequest.execute (HttpRequest.java:1137)
at io.apigee.buildTools.enterprise4g.rest.RestUtil.executeAPI (RestUtil.java:926)
at io.apigee.buildTools.enterprise4g.rest.RestUtil.activateBundleRevision (RestUtil.java:847)
at io.apigee.buildTools.enterprise4g.rest.RestUtil.activateBundleRevision (RestUtil.java:802)
at io.apigee.buildTools.enterprise4g.mavenplugin.DeployMojo.doActivateBundle (DeployMojo.java:234)
at io.apigee.buildTools.enterprise4g.mavenplugin.DeployMojo.execute (DeployMojo.java:297)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:210)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:195)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
at java.lang.Thread.run (Thread.java:840)
[ERROR] Failed to execute goal io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.5.1:deploy (deploy-bundle) on project shared-apikey-failure-processing: MojoExecutionException: com.google.api.client.http.HttpResponseException: 404 Not Found
[ERROR] {
[ERROR] "error": {
[ERROR] "code": 404,
[ERROR] "message": "generic::not_found: organizations/xxxx/sharedflows/zzzz/revisions/155 not found",
[ERROR] "status": "NOT_FOUND",
[ERROR] "details": [
[ERROR] {
[ERROR] "@type": "type.googleapis.com/google.rpc.RequestInfo",
[ERROR] "requestId": "15113595828044375745"
[ERROR] }
[ERROR] ]
[ERROR] }
[ERROR] }
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.5.1:deploy (deploy-bundle) on project shared-apikey-failure-processing:
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:375)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:210)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:195)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
at java.lang.Thread.run (Thread.java:840)
Caused by: org.apache.maven.plugin.MojoExecutionException:
at io.apigee.buildTools.enterprise4g.mavenplugin.DeployMojo.processHelpfulErrorMessage (DeployMojo.java:332)
at io.apigee.buildTools.enterprise4g.mavenplugin.DeployMojo.execute (DeployMojo.java:317)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:210)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:195)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
at java.lang.Thread.run (Thread.java:840)
Caused by: java.io.IOException: com.google.api.client.http.HttpResponseException: 404 Not Found
{
"error": {
"code": 404,
"message": "generic::not_found: organizations/xxxx/sharedflows/zzzz/revisions/155 not found",
"status": "NOT_FOUND",
"details": [
{
"@type": "type.googleapis.com/google.rpc.RequestInfo",
"requestId": "15113595828044375745"
}
]
}
}
at io.apigee.buildTools.enterprise4g.rest.RestUtil.activateBundleRevision (RestUtil.java:852)
at io.apigee.buildTools.enterprise4g.rest.RestUtil.activateBundleRevision (RestUtil.java:799)
at io.apigee.buildTools.enterprise4g.mavenplugin.DeployMojo.doActivateBundle (DeployMojo.java:234)
at io.apigee.buildTools.enterprise4g.mavenplugin.DeployMojo.execute (DeployMojo.java:297)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:210)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:195)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
at java.lang.Thread.run (Thread.java:840)
Caused by: com.google.api.client.http.HttpResponseException: 404 Not Found
{
"error": {
"code": 404,
"message": "generic::not_found: organizations/xxxx/sharedflows/zzzz/revisions/155 not found",
"status": "NOT_FOUND",
"details": [
{
"@type": "type.googleapis.com/google.rpc.RequestInfo",
"requestId": "15113595828044375745"
}
]
}
}
at com.google.api.client.http.HttpRequest.execute (HttpRequest.java:1137)
at io.apigee.buildTools.enterprise4g.rest.RestUtil.executeAPI (RestUtil.java:926)
at io.apigee.buildTools.enterprise4g.rest.RestUtil.activateBundleRevision (RestUtil.java:847)
at io.apigee.buildTools.enterprise4g.rest.RestUtil.activateBundleRevision (RestUtil.java:799)
at io.apigee.buildTools.enterprise4g.mavenplugin.DeployMojo.doActivateBundle (DeployMojo.java:234)
at io.apigee.buildTools.enterprise4g.mavenplugin.DeployMojo.execute (DeployMojo.java:297)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:210)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:195)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
at java.lang.Thread.run (Thread.java:840)
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <args> -rf :yyyy
That was running the real project with mvn -X -T10C ...
One thing came to mind that is different in this real project. This Apigee project/org has 5 Apigee environments. All those have this project deployed to them. Which means up 5 different revisions may be deployed (and not deletable)
ok.. for the ones that are failing the logs, I notice that all the revisions for those APIs and sharedflows are pointing to 155
Can you confirm from your logs that though the APIs and sharedflows are different, you are seeing the same revision number across all those that are failing in the logs?
This is for the Proxy "yyyy" after the failed deployment (another one runnnig)
This if for shared flow "zzzz"
but is the Maven plugin stuck to one revision number for all the poxies and sharedflows?
If you see the error logs you shared earlier, they all have the same revision number 155
So am wondering if that revision is somehow being used across the threads
yes, it seems so that is mixes up the revision numbers. That log is direct cut&paste, I only masked the org & proxy names. They should not have same number. The proxy is older and it's counter is around 300 and sharedflow is new counter at 90-ish.
P.S. Problem is exceeding max number of revisions, we have clenaup script to delete all undeployed. This scpirt was not running on this failed test.
it could be that sample work because in that project all proxies and flows same revision number. Yeah, in Sample all the proxies have same rev numbe deployed (8) as they are all deployed exactly same number of times. All sample flows had rev numbet deployed (2).
I guess you could reproduce same state by. commenting out all modules in root pom, except one, and runninf deploy. Then uncomment another one and re-run the project deploy. Repeating this to all or at least for 5-10 so get different revision numbers for each proxy and flow. You might need to delete undeployed so the no proxy and flow have same rev numbers. Slow work.
Or if you know a better way to manipulate proxies and flows revision numbers, so that none of them have matching numbers.
Yes - Let me try that and get back to you
Now I got same error on Sample project. All flows were rev 2 and all proxies were rev 8. This run was now deploying proxies as rev 9. Somehow this value leaked into flows this time. I think it was just by change the plugin did not raise error before and because there were only two different numbers. All proxies had one same number and all flow another same number. By luck it had not mismatched numbers before.
[ERROR] Failed to execute goal io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.5.1:deploy (deploy-bundle) on project flow07: MojoExecutionException: com.google.api.client.http.HttpResponseException: 404 Not Found [ERROR] { [ERROR] "error": { [ERROR] "code": 404, [ERROR] "message": "generic::not_found: organizations/xxxx/sharedflows/flow07/revisions/9 not found", [ERROR] "status": "NOT_FOUND", [ERROR] "details": [ [ERROR] { [ERROR] "@type": "type.googleapis.com/google.rpc.RequestInfo", [ERROR] "requestId": "11660799613087284182" [ERROR] } [ERROR] ] [ERROR] } [ERROR] } [ERROR] -> [Help 1]
Seems the variable holding the rev number is not thread safe.
Got it.. Will work on it and have a fix shortly.. I guess this issue will be there for "async" runs too
@ssvaidyanathan Please let me know when you have the fix so I can try it.
@mikezerosix - I just made some changes and published a release candidate. Update your pom to point to Apigee Maven 2.5.2-rc1
and re-test the scenarios that were failing. Once you confirm, I will release it as 2.5.2
@ssvaidyanathan 2.5.2-rc1 still failing the same way as before when running -T10C on real project. 5 out of 18 submodules, both proxies and shared flows, failed on HTTP 404 on revision number not found.
@mikezerosix - can you try with 2.5.2-rc2
please
@ssvaidyanathan Great success. I ran it successfully twice. It is huge help as it shortens 30 min build to 2 min. Let us test it for a day and I will close this issue. Thank you very much !
Thats awesome news @mikezerosix
Thanks for working with me on this. Keep testing and let me know once you find its working as expected.
I will then release the 2.5.2 version
We have a lot of proxies and running deploy takes looooong time.
We structured our solution as Maven multi module project. So simply running from root all we can easily deploy them all and we also can easily choose to deploy them individually from each submodule.
But would be great if we would deploy all of them at once using Maven parallel build:
mvn -T 10 install
(install phase runs the apigee plugin)We have tried parallel build, but it fail with strange errors like :
That shared flow has no dependencies and it deploys without any issues in non-parallel build. Cutting down the half hour deploy time to just 3 minutes being able to run it in parallel would be amazing.
So if possible, please suppport Maven parallel builds.