Apigee Network call error

[srinivasmurari]

hi, when we tried to deploy API proxies using the cicd pipeline we are not able to deploy and got Apigee Network to call error and connection timed out issue,we need to connect via proxy to do any deployments from cicd server we had added proxy in our seetings.xml file and using 2.3.0 maven deploy plugin.

we have tested sample management APIs using bearer token we can make post & get from the same cicd server, but using the plugin we are getting the below-attached issue deploy command used mvn clean install -phybrid-apiproxy -Dorg=orgname -Denv=envname -Dbearer=token

[ssvaidyanathan]

@srinivasmurari - Looks like the machine where this is running is not able to access apigee.googleapis.com Can you please confirm the machine can access? Do you need to pass any proxy settings?

[srinivasmurari]

we can access apigee.googleapis.com from the same machine (we are tested a management API from the machine we have created a target server) Yes, we have proxy settings, we have added a proxy in our settings.xml, and we are using the 2.3.0 maven plugin.

[ssvaidyanathan]

Can you share the contents of your settings.xml? Hope you are using "https". Thats the only supported protocol.

If the proxy is setup correctly, you should see something like this in your console output. Can you check your console to see if anything similar is logged?

By the way the latest version is v2.4.3

[srinivasmurari]

`

/root/.m2/repository/ org.apache.maven.plugins org.codehaus.mojo false false nexus * file:///root/.m2/repository snapshot release nexus

<proxies>   
 <proxy>
    <id>https_proxy</id>
    <active>true</active>
    <protocol>https</protocol>
    <host>proxyip</host>
    <port>proxyport</port>
    <nonProxyHosts></nonProxyHosts>
 </proxy>

` setting.xml

Yes we are using https proxy in setting.xml. we can see the proxy as per the link above in our console output.

[srinivasmurari]

Hi @ssvaidyanathan ,

Kindly do the needful, currently, we are blocked at this point. if you need more info let us know.

[ssvaidyanathan]

@srinivasmurari - I am assuming you have actual proxy host and port in your settings.xml and thats what the log is showing?

If thats true, thats the extent I can help you with. I know other customers using this plugin with the proxy settings and its working for them without issues.

[srinivasmurari]

Yes, we are using an actual proxy only. We have tested a management API call with the proxy and it worked fine we can deploy the target server using it.

But when we are trying to use the plugin we are facing issues with the same proxy.

[ssvaidyanathan]

Are you trying the maven call from the same machine where you tested the Mgmt API?

[srinivasmurari]

Yes we have tested from the same machine where we are running the maven call

[ssvaidyanathan]

Just noticed that your proxy settings doesn't have any username password fields? Is that correct? Can you check your machine config to see if the proxy requires any authentication?

[srinivasmurari]

Yeah, it does not have a username and password. The proxy doesn't require any authentication.

As we can make management calls from the same machine without authentication that can also conform in 100% that there is no authentication for proxy.

[ssvaidyanathan]

Are you running the direct Mgmt API calls from your machine and the same machine is failing when called via Maven? Or is it from a build server?

[srinivasmurari]

Yes we are running mgmnt API calls from the same machine and when called via maven it is failing on the same machine

[ssvaidyanathan]

You already answered that - My question is - are you trying it in your laptop? If yes, then probably your machine ldap is being passed when you make the direct API call to the proxy from your machine settings. Can you check?

I might have found something in the code but want to be sure before I make that change.

[srinivasmurari]

No, I am not trying it on my laptop or any personal server. We are trying all the management calls and maven calls in the same Jenkins server.

[ssvaidyanathan]

I've just released a new version with a fix (v2.4.4) Will take an hour or so to be available in the Central repo for you to download. Once you see the version here, update your pom to point to this version of the apigee config plugin and re-try. Let me know how it goes.

[ssvaidyanathan]

Were you able to try?

[srinivasmurari]

Not yet, as the Jenkins server has no internet connectivity and is hosted in a private network we need to request the required team to get us the plugin's latest version for testing it might take some time but once we can deploy we will let you know

[srinivasmurari]

@ssvaidyanathan , after installing plugin(config maven plugin :2.4.4, apigee edge maven plugin : 2.3.5 ) in jenkins server we are facing below issue . in jenkins server we have http client version 4.5.5 maven version 3.6.2 java version 11.0.14.1 .

[ssvaidyanathan]

For the deploy plugin - can you update your pom to 2.3.4 and try? Lets first fix the config plugin issue.

[ssvaidyanathan]

@ssvaidyanathan , after installing plugin(config maven plugin :2.4.4, apigee edge maven plugin : 2.3.5 ) in jenkins server we are facing below issue . in jenkins server we have http client version 4.5.5 maven version 3.6.2 java version 11.0.14.1 .

I just tried the deploy plugin (v2.3.5) and it worked for me.

The httpclient jar dependency for this plugin is coming from com.google.http-client:google-http-client-apache:jar:2.0.0 Please check if thats available on your repo

[srinivasmurari]

we have tried with edge plugin(v2.3.4) and config plugin(v2.4.4), here giving below error like config plugin

the com.google.http-client:google-http-client-apache:jar:2.0.0 plugin is available in the repo

[ssvaidyanathan]

am not sure - the same error comes with earlier config plugin version? Lets say you use v2.3.0? Did you see it?

[srinivasmurari]

No with earlier config plugin version we have not seen the same we saw the different error in config plugin version 2.3.0(Apigee network call error)

[ssvaidyanathan]

There were few releases after that v2,3.0 thst included few fixes, features and also updates to dependency due to vulnerability.Cn you try each version released after 2.3.0 and see which version its starting to cause the issue. Unfortunately thats the only way I can think of

[srinivasmurari]

@ssvaidyanathan we are able to deploy target servers and api proxy using Bearer Token But when we tried to deploy using service account we are facing below issue

maven config plugin : 2.4.4 maven deploy plugin : 2.3.5

[ssvaidyanathan]

So if it worked with the token, it means the proxy issue is fixed with the change? Does it work with the previous version as well? That is using v2.3.4 and passing token, did it work or did the proxy issue return a timeout?

I will look into the issue you reported. But please respond to the above question.

[ssvaidyanathan]

I just tried deploying the sample sharedflow from the repo using a service account. It worked as expected. I also tried deleting my .m2/repository directory so that it downloads it freshly. That also worked.

I am using

Java version: 1.8.0_301
Apache Maven: 3.8.4

[srinivasmurari]

So if it worked with the token, it means the proxy issue is fixed with the change? Does it work with the previous version as well? That is using v2.3.4 and passing token, did it work or did the proxy issue return a timeout?

I will look into the issue you reported. But please respond to the above question.

yes, proxy issue is fixed with the change.

No, We tried with 2.3.4 and by passing a token but the proxy issue returned a timeout

[ssvaidyanathan]

cant really tell why it timed out without your logs.

Please do not mix the config and deploy plugin in your pipeline right now. Let's first fix the config plugin. Please remove any deploy plugin tasks in your pom. Point your pom to the latest config plugin v2.4.4 and see if it works? Please confirm if all test cases with config plugin works as expected.

[srinivasmurari]

we have removed the deploy plugin from pom and using only the config plugin v2.4.4. after the changes also we are facing the same issue timed out with the service account, with the bearer token we can deploy.

[ssvaidyanathan]

we have removed the deploy plugin from pom and using only the config plugin v2.4.4. after the changes also we are facing the same issue timed out with the service account, with the bearer token we can deploy.

Can you share the mvn command you are running? and also share the properties of the pom profile you are calling as part of the mvn command

[srinivasmurari]

Properties of hybrid apiproxy `

hybrid-apiproxy https://apigee.googleapis.com v1 ${org} ${env} ${bearer} ${file} override update target/resources/edge target/test/integration ` **MVN Command** : `mvn package apigee-config:targetservers -s ${MAVEN_SETTINGS_XML} -X -Phybrid-apiproxy -Dorg=org -Denv=env  -Dfile=${serviceAccount}  -Dapigee.config.options=update`

[ssvaidyanathan]

{serviceAccount} - is the path of the service account file or the file contents? It needs to be the former.

Are you seeing this log entry in your console logs

Since you are running it with -X, can you also check if this is getting printed

[srinivasmurari]

{serviceAccount} is the service account file that we stored in Jenkins credentials and used in the deployment stage. we are able to see these log entriy in console logs logger.info("Using the service account file to generate a token"); we are not able to see this log info in the console log logger.debug("Access Token "+ accessToken);

[ssvaidyanathan]

The plugin will use the json file and then generate an access token by hitting "https://www.googleapis.com/oauth2/v4/token" behind the scenes. Can you confirm if there are any network issues connecting to that endpoint through the proxy?

Also can you share the logs you are getting when its timing out

[srinivasmurari]

we are not able to connect with these URL from Jenkins server https://www.googleapis.com/oauth2/v4/token. we can connect these OAuth URL https://login.apigee.com/oauth/token.

[ssvaidyanathan]

https://login.apigee.com/oauth/token is for Apigee Edge and not used by the plugin Please work with your network team and see if https://www.googleapis.com/oauth2/v4/token can be opened

[srinivasmurari]

okay, we will work with the network team and open it. will update you once we test after the URL is opened

[ssvaidyanathan]

@srinivasmurari - any update?

[srinivasmurari]

@ssvaidyanathan - we are working with the networking team to whitelist URL https://www.googleapis.com/oauth2/v4/token, Once the URL got whitelisted we will test that one and update you on the progress.

[srinivasmurari]

@ssvaidyanathan after whitelisting the URL( https://www.googleapis.com/oauth2/v4/token) also we are don't able to deploy target servers using a service account file. error: APIGEE NETWORK CALL ERROR PLUGIN USED: 2.4.4 CONFIG PLUGIN 2.3.5 DEPLOY PLUGIN

[srinivasmurari]

@ssvaidyanathan After enabling the OAuth URL ( https://www.googleapis.com/oauth2/v4/token) we are able to deploy the API proxy through bearer token but using the service account we are unable to generate the token through the plugin, Please find the below image error log,

Please suggest, how can we enable exact logging on the plugin to get detailed error log info?

[ssvaidyanathan]

Lets do one thing. Can you create another Jenkins Job and try creating a simple pipeline that uses your service account key file to generate an access token? See if you can get that. If that works, then there is no connectivity issues.

[srinivasmurari]

@ssvaidyanathan, we created another Jenkins job and created a simple and generated access token with the help of a service account. we used the below shell scripts to generate an access token ( service account flow: jwt token >> access token ) https://gist.github.com/ryu1kn/c76aed0af8728f659730d9c26c9ee0ed
note: in the get-access-token. sh script we have added our proxy.

[ssvaidyanathan]

@srinivasmurari - I am not sure why the plugin is not able to do that. I tested it on my Cloud Build instance and also in my Github actions. The plugin is able to generate the token and make the deployments.

So the only option I can think is for you to use the above script to generate the token and then pass the generated token to the mvn command.

[srinivasmurari]

ok, we will try to pass the token into the mvn command, meanwhile can you please check, As per our analysis, we are thinking that the plugin is not supporting the proxy, if possible can you add one more if else condition for the below which supports the proxy. path : src/main/java/com/apigee/edge/config/rest/RestUtil.java Thanks @ssvaidyanathan

[ssvaidyanathan]

I'll take a look. Its quite possible

[ssvaidyanathan]

@srinivasmurari - I have just released a v2.4.5-rc1 with a fix Since I dont have a corporate proxy setup, I cant really test it. So published a rc version Can you please remove the deploy plugin from your pipeline and just use the config plugin pointing to the version 2.4.5-rc1 and try? Pass the service account file and see if its working. Please let me know. Once you confirm, I can push an actual version. Until then we can try out with these rc versions.

[srinivasmurari]

@ssvaidyanathan after using the latest release v2.4.5-rc1 with proxy we are facing the below issue and we have tried with the edge plugin(v2.3.5).