search

GoogleCloudPlatform / cloud-sql-jdbc-socket-factory

A collection of Java libraries for connecting securely to Cloud SQL
Apache License 2.0
229 stars 118 forks source link

Getting info messages in application logs : Failed to create ephemeral certificate for the Cloud SQL instance. #1925

Open pjain-sift opened 5 months ago

pjain-sift commented 5 months ago

Bug Description

We are noticing following info messages in our application logs which started from 03/11/2024 and still happening. This does not seem to affect the database connections though adding noise to the application logs and give false alarm to new engineers who are not aware of this issue.

Example code (or command)

No response

Stacktrace

Mar 29 12:02:13  POST https://sqladmin.googleapis.com/sql/v1beta4/projects/sift-flow-runner-prod/instances/flowrunner-db:generateEphemeralCert
Mar 29 12:02:13  <!DOCTYPE html>
Mar 29 12:02:13  <html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content=initial-scale=1, minimum-scale=1, width=device-width>
  <title>Error 502 (Server Error)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>502.</b> <ins>That’s an error.</ins>
  <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.  <ins>That’s all we know.</ins>
Mar 29 12:02:13  
    at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$3.interceptResponse(AbstractGoogleClientRequest.java:466)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:552)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:493)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:603)
    at com.google.cloud.sql.core.DefaultConnectionInfoRepository.fetchEphemeralCertificate(DefaultConnectionInfoRepository.java:276)
    ... 9 common frames omitted
Wrapped by: java.lang.RuntimeException: [sift-flow-runner-prod:us-east4:flowrunner-db] Failed to create ephemeral certificate for the Cloud SQL instance.
    at com.google.cloud.sql.core.DefaultConnectionInfoRepository.addExceptionContext(DefaultConnectionInfoRepository.java:381)
    at com.google.cloud.sql.core.DefaultConnectionInfoRepository.fetchEphemeralCertificate(DefaultConnectionInfoRepository.java:278)
    at com.google.cloud.sql.core.DefaultConnectionInfoRepository.lambda$getConnectionInfo$1(DefaultConnectionInfoRepository.java:112)
    at com.google.common.util.concurrent.CombinedFuture$CallableInterruptibleTask.runInterruptibly(CombinedFuture.java:196)
    ... 7 common frames omitted
Wrapped by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: [sift-flow-runner-prod:us-east4:flowrunner-db] Failed to create ephemeral certificate for the Cloud SQL instance.
    at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:592)
    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:551)
    at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:111)
    at com.google.cloud.sql.core.Refresher.handleRefreshResult(Refresher.java:196)
    at com.google.cloud.sql.core.Refresher.lambda$startRefreshAttempt$1(Refresher.java:188)
    at com.google.common.util.concurrent.CombinedFuture$AsyncCallableInterruptibleTask.runInterruptibly(CombinedFuture.java:165)
    at com.google.common.util.concurrent.CombinedFuture$AsyncCallableInterruptibleTask.runInterruptibly(CombinedFuture.java:153)
    at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)"

Steps to reproduce?

N/A

Environment

N/A

Additional Details

No response

enocom commented 5 months ago

@pjain-sift What version are you using?

pjain-sift commented 5 months ago

Thanks @enocom for your reply. We are using version 1.15.2 and also see version 1.0.13 in the classpath.

enocom commented 5 months ago

It's possible we've fixed this in the latest version (v1.17.1) which I'd recommend trying.

The background certificate refresh will sometimes fail and if it does, its exception might appear in the logs. Assuming you still see this in the latest version, we could revisit this design to see if we can make it less noisy and less alarming.

pjain-sift commented 5 months ago

Thanks @enocom. Really appreciate you for looking into it and providing your feedback.

jackwotherspoon commented 1 month ago

@hessjcg do you want to verify that these logs are now debug and not showing as info logs? If so I think we can close this out