Failed to generate client library: 500 Internal Server Error

[m121212]

When doing: gradle endpointsClientLibs

Execution failed for task ':app:_endpointsClientLibs'.

com.google.api.client.http.HttpResponseException: 500 Internal Server Error

[spuniani]

This is a blocking issue - we can't proceed. How do we bump this up?

What I've tried

• tried with the echo api sample - same result
• tried with endpoints-framework-tools-2.2.1 standalone - same result

the problem seems to be in the call to "https://google-api-client-libraries.appspot.com/generate" - this returns a 500 Internal Server Error - but no idea why.

stacktrace Exception is: org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':servers:api:endpointsClientLibs'. at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77) at org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51) at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59) at org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54) at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59) at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101) at org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44) at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91) at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62) at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59) at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54) at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43) at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34) at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51) at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:300) at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:292) at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:174) at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90) at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31) at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46) at org.gradle.execution.taskgraph.LocalTaskInfoExecutor.execute(LocalTaskInfoExecutor.java:42) at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareWorkItemExecutor.execute(DefaultTaskExecutionGraph.java:277) at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareWorkItemExecutor.execute(DefaultTaskExecutionGraph.java:262) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:135) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:130) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.execute(DefaultTaskPlanExecutor.java:200) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.executeWithWork(DefaultTaskPlanExecutor.java:191) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.run(DefaultTaskPlanExecutor.java:130) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63) at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46) at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55) Caused by: org.gradle.api.UncheckedIOException: com.google.api.client.http.HttpResponseException: 500 Internal Server Error at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:57) at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:40) at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:76) at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:46) at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:39) at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:26) at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:801) at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:768) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$1.run(ExecuteActionsTaskExecuter.java:131) at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:300) at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:292) at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:174) at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90) at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:120) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:99) ... 31 more Caused by: com.google.api.client.http.HttpResponseException: 500 Internal Server Error at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1072) at com.google.api.server.spi.tools.CloudClientLibGenerator.postRequest(CloudClientLibGenerator.java:65) at com.google.api.server.spi.tools.CloudClientLibGenerator.generateClientLib(CloudClientLibGenerator.java:55) at com.google.api.server.spi.tools.GenClientLibAction.genClientLib(GenClientLibAction.java:91) at com.google.api.server.spi.tools.GetClientLibAction.getClientLib(GetClientLibAction.java:87) at com.google.api.server.spi.tools.GetClientLibAction.execute(GetClientLibAction.java:61) at com.google.api.server.spi.tools.EndpointsTool.execute(EndpointsTool.java:84) at com.google.cloud.tools.gradle.endpoints.framework.server.task.EndpointsArtifactTask.generateEndpointsArtifact(EndpointsArtifactTask.java:208) at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:73) ... 44 more

[peterfortuin]

Same problem here. Just started to get the 500 Internal Server Error without any reason. Reverted all my own work and tested out old versions of my own code, always get the 500 Internal Server Error.

[m121212]

Tried all the way aroud. From :

• echo api
• gradle wrapper
• gradle downloaded locally

All components are up-to-date: com.google.guava:guava:28.1-jre com.android.tools.build:gradle:3.5.1 com.google.cloud.tools:endpoints-framework-gradle-plugin:2.0.1 com.google.endpoints:endpoints-framework:2.2.1 com.google.http-client:google-http-client:1.32.1 com.google.api-client:google-api-client:1.30.4

This error is blocking!

[m121212]

Did anyone find a solution?

[loosebazooka]

Sorry, I'll escalate this internally. @tangiel

I'll update this bug as I learn more.

[m121212]

Here's the info from the generate request:

-------------- REQUEST -------------- POST https://google-api-client-libraries.appspot.com/generate Accept-Encoding: gzip User-Agent: Google-HTTP-Java-Client/1.25.0 (gzip) Content-Type: multipart/form-data; boundary=----GoogleApisClientBoundary1571151312395 Content-Length: 68566

curl -v --compressed -X POST -H 'Accept-Encoding: gzip' -H 'User-Agent: Google-HTTP-Java-Client/1.25.0 (gzip)' -H 'Content-Type: multipart/form-data; boundary=----GoogleApisClientBoundary1571151312395' -d '@-' -- 'https://google-api-client-libraries.appspot.com/generate' << $$$ Total: 68.566 bytes (logging first 16.384 bytes) ------GoogleApisClientBoundary1571151312395 Content-Disposition: form-data; name="lang"

java ------GoogleApisClientBoundary1571151312395 Content-Disposition: form-data; name="content"

{ "auth": {.... THIS IS THE DISCOVERY DOC

}

And this is the response:

-------------- RESPONSE -------------- HTTP/1.1 500 Internal Server Error X-Cloud-Trace-Context: dcd352524bb55d5872d79bdcedff934a;o=1 Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000 Server: Google Frontend Cache-Control: no-cache Content-Length: 0 Date: Tue, 15 Oct 2019 14:55:12 GMT Content-Type: text/html; charset=utf-8

[m121212]

Any news on this?

[GuyDviri]

Execution failed for task ':*ClientLibs'.

com.google.api.client.http.HttpResponseException: 500 Internal Server Error what's going on .stack for few hours ,any one with friends at google?

[m121212]

I have also opened the issue here (https://github.com/cloudendpoints/endpoints-java/issues/183) where the problem occurs.

[chingor13]

We are rolling back the bad config change, sorry folks.

[GuyDviri]

thanks . when it will be alive?

On Tue, Oct 15, 2019 at 8:25 PM Jeff Ching notifications@github.com wrote:

We are rolling back the bad config change, sorry folks.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/endpoints-framework-gradle-plugin/issues/98?email_source=notifications&email_token=ACJDN2U6LUCDEUKRQUJJ3STQOX4IPA5CNFSM4JA4SVV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBJSJUQ#issuecomment-542319826, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACJDN2UUSZMP3HJRN4TUMKLQOX4IPANCNFSM4JA4SVVQ .

[chingor13]

The rollback should be complete.

[loosebazooka]

@GuyDviri @m121212 @spuniani @peterfortuin can you verify the service is up again?

[m121212]

It works!

[GuyDviri]

yes, it's working now thanks you.

On Tue, Oct 15, 2019 at 8:58 PM Appu notifications@github.com wrote:

@GuyDviri https://github.com/GuyDviri @m121212 https://github.com/m121212 @spuniani https://github.com/spuniani @peterfortuin https://github.com/peterfortuin can you verify the service is up again?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/endpoints-framework-gradle-plugin/issues/98?email_source=notifications&email_token=ACJDN2TTIEZKNANLHYPE45DQOYAFTA5CNFSM4JA4SVV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBJVQRI#issuecomment-542332997, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACJDN2V6JQ2NUU4EVPROHY3QOYAFTANCNFSM4JA4SVVQ .

[loosebazooka]

Thanks for verifying, we're updating our internal deployment process to prevent this kind of failure in the future. Closing for now.

[peterfortuin]

Yes, it works!

[amarkovits]

I have the same problem:

com.google.api.client.http.HttpResponseException: 500 Internal Server Error {"error": {"message": "Internal Server Error", "code": 500, "errors": [{"message": "unknown exception", "debug_info": "Neither servicePath nor rpcPath is defined."}]}}

[peterfortuin]

It seems that the problem is back:

com.google.api.client.http.HttpResponseException: 500 Internal Server Error
  POST https://google-api-client-libraries.appspot.com/generate
  {"error": {"code": 500, "message": "Internal Server Error", "errors": [{"message": "unknown exception", "debug_info": "cannot use a string pattern on a bytes-like object"}]}}

[loosebazooka]

@peterfortuin is this happening on a regular basis (every build)? or just sometimes?

I received an update from @chingor13 that it should be working and was a temporary issue with serving infrastructure.

[peterfortuin]

It was around 90% of the time failing. But I tested it again and it seems to be working fine. So, thanks for resolving it!

[cmaan]

@loosebazooka It's a little bit different (read timeouts instead of internal server errors), but we're having issues with this cloud service again (has been down for us since yesterday):

Caused by: java.net.SocketTimeoutException: Read timed out
        at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:149)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
        at com.google.api.server.spi.tools.CloudClientLibGenerator.postRequest(CloudClientLibGenerator.java:65)

[loosebazooka]

@chingor13 any ideas?

[loosebazooka]

@cmaan, are you still experiencing this?

[cmaan]

Yes, although looking at it closely, it seems the problem depends on the size of the discovery document. When commenting out endpoints, the task succeeds.

[cmaan]

I created a separate issue for this: #99