Direct send of a report request failed because of endpoints.repackaged.com.google.api.client.http.HttpResponseException

[emremogn]

Hello,

I have a simple API built with Endpoints Frameworks (v. 2.0.7) for App Engine Standard (Java 7).

I've added API Management following the documentation at https://cloud.google.com/endpoints/docs/frameworks/java/adding-api-management.

The API has API keys with IP restriction (clients are web servers).

For most of the time all works fine, but sometimes (very rarely, in truth) I see this error in logs:

com.google.api.control.Client flushAndScheduleReports: direct send of a report request failed because of endpoints.repackaged.com.google.api.client.http.HttpResponseException: 400 (Client.java:354)
{
  "error": {
    "code": 400,
    "message": "Precondition check failed.",
    "errors": [
      {
        "message": "Precondition check failed.",
        "domain": "global",
        "reason": "failedPrecondition"
      }
    ],
    "status": "FAILED_PRECONDITION"
  }
}

I cannot actually understand the origin of the error and cannot find any clue in docs.

I noticed it happens when a new key is used for the first time by a client or when an existing key is edited, but this explanation of the error is a pure supposition.

Thank you.

[tangiel]

@emremogn we probably need your project ID and service name to do some investigation. If you click the documentation page, you can click "Send Feedback" at the top right and fill in information there, if you don't want to reveal information here.

[emremogn]

@tangiel sent a product feedback with additional information (project ID, service name, service version). Thank you.

[tangiel]

@emremogn Are you using an autoscaling app or basic/manual?

[emremogn]

@tangiel An autoscaling app.

[tangiel]

Can you estimate how many requests per second or minute you get?

[emremogn]

Requests can have an extreme variance; the API does nothing for long periods and then clients can generate sudden and heavy traffic in very short intervals (thousands of requests in few minutes). I'm sorry, but I cannot estimate, at the moment, the rate (requests / s or requests / min).

[tangiel]

I think this is related. I think if you switch to basic scaling, it will fix this issue. The reason is because API management metrics are typically aggregated and flushed via a background thread, but automatic scaling doesn't support background threads. In this case, metrics are aggregated and flushed after some requests. But if your traffic is too intermittent, stale data gets sent, causing this issue.

[emremogn]

Basic scaling fixed the issue. Metrics in endpoints dashboard are now ok. Thank you.

[emremogn]

Just a clarification: does the background thread used by the API management run indefinitely? Dynamically added instances never shutdown, even if there are no requests and regardless of idle timeout. Am I wrong?

[tangiel]

Yes, the thread runs indefinitely. I'm not sure what you mean by your second question. Auto and basic scaling instances should shut down when they aren't used.

[emremogn]

It seems to me that API Management causes the instances to remain standing.

To test it, I've created a new endpoint project with no API Management. I've set basic scaling in config (appengine-web.xml) and instances have an idle timeout of 3m.

Instances are started and then, ~3m after receiving their last request, properly stopped.

In logs I can see 08:55:43.306 GET 404 234 B 3,1 s Unknown /_ah/start 08:58:47.252 GET 200 2 B 82 ms Unknown /_ah/stop

Now, if I add API Management to the project, instances (all the dinamically added instances) never shut down after receiving their last request, despite of idle timeout set in config.

Does the background thread used by API Management cause the instances to remain standing?

I can see a remarkable consumption and increasing costs also when there's no traffic at all and after hours since the last received request.

Thank you.

[tangiel]

Thanks for the report. It appears you are correct. I'm looking into ways to rectify this.

[emremogn]

@tangiel Hello, any news about this issue? Thank you very much.

[william-ferguson-au]

I'd like an update too. I had noticed the behaviour also, but hadn't traced it as far as @emremogn did.

[william-ferguson-au]

Also, it is really confusing to have the log for the management API intrude on the log for a request.

It would be much better if the management API Thread generated its own entry. After all it should be entirely orthogonal to the requests.

[tangiel]

I have a potential solution that hasn't been implemented yet, involving spinning down the background thread if no requests have been sent for awhile. It should solve this problem, if it works correctly. The log of the background thread in the request log is, unfortunately, beyond the scope of this project. This is an App Engine usability problem.