search

cloudfoundry-community / autosleep

Auto sleep service for CloudFoundry
Apache License 2.0
39 stars 21 forks source link

autosleep-app errors when trying to stop app on pivotal.io #274

Closed drnic closed 7 years ago

drnic commented 7 years ago

From autosleep-app, configured against api.run.pivotal.io:

2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT org.cloudfoundry.autosleep.access.cloudfoundry.CloudFoundryException: java.lang.IllegalArgumentException: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT  at [Source: <html>
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT <head><title>404 Not Found</title></head>
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT <body bgcolor="white">
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT <center><h1>404 Not Found</h1></center>
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT <hr><center>nginx</center>
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT </body>
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT </html>
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT ; line: 1, column: 2]
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT         at org.cloudfoundry.autosleep.access.cloudfoundry.CloudFoundryApi.waitForResult(CloudFoundryApi.java:441)
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT         at org.cloudfoundry.autosleep.access.cloudfoundry.CloudFoundryApi.getApplicationActivity(CloudFoundryApi.java:249)
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT         at org.cloudfoundry.autosleep.worker.ApplicationStopper.handleApplicationEnrolled(ApplicationStopper.java:120)
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT         at org.cloudfoundry.autosleep.worker.ApplicationStopper.lambda$run$10(ApplicationStopper.java:197)
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT         at org.cloudfoundry.autosleep.util.ApplicationLocker.executeThreadSafe(ApplicationLocker.java:47)
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT         at org.cloudfoundry.autosleep.worker.ApplicationStopper.run(ApplicationStopper.java:190)
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2017-01-08T22:37:10.43-0800 [APP/PROC/WEB/0]OUT         at java.lang.Thread.run(Thread.java:745)

Suggestions for debugging?

pradyutsarma commented 7 years ago

@drnic Will you be able to provide some more data like the parameters with which you created the service instance within your space? The cf cli command should be enough. Also, if you could share your manifest snapshot?

gberche-orange commented 7 years ago

@drnic It looks like the autosleep call to the CF API recentLog is getting a 404 call, surprisingly from an nginx server. Would be worth checking the autosleep app is properly configured to talk to the CF API (which I would be surprised to host nginx on PWS)

As @pradyutsarma was suggesting, checking the env vars provided to autosleep app would help making sure the CF API endpoint is properly configured.

Next diagnostic steps would then be to enable the CF java client verbose wire traces to trace this exchange with the CC API.

You might want to try with

-Dlogging.level.cloudfoundry-client.request=DEBUG -Dlogging.level.cloudfoundry-client.response=DEBUG -Dlogging.level.stream=INFO -Dlogging.level.reactor.ipc.netty=INFO

drnic commented 7 years ago

Sorry I haven't gotten back to this; will try to follow up soon.

gberche-orange commented 7 years ago

@drnic sorry it took us a bit of time to figure that out, it seems a duplicate of the main blocker bug we have currently open: #264 Thanks for reporting this stack trace variation. I'm therefore closing this one, please reopen as needed.

The error message you experienced is different that the reported messages in #264 because PWS has customized the load balancer handling the TLS termination e.g. curl https://badhost.run.pivotal.io

<!DOCTYPE html>
<html class="bg-dark-11">
  <head>
    <meta charset="utf-8">
    <meta content="IE=edge,chrome=1" http-equiv="X-UA-Compatible">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Pivotal | 404 Not Found</title>
    <link href="build/components.css" rel="stylesheet" type="text/css">
  </head>
...