googleapis / nodejs-logging-winston

Node.js client integration between Stackdriver Logging and Winston.
https://cloud.google.com/logging/
Apache License 2.0
105 stars 50 forks source link

Get the error `GoogleError: Total timeout of API google.logging.v2.LoggingServiceV2` when logging with `httpRequest.latency` #787

Closed domdomegg closed 1 year ago

domdomegg commented 1 year ago

1) Is this a client library issue or a product issue?

Unclear, possibly the root cause is a product issue but the client should probably be able to handle this better.

2) Did someone already solve this?

Have reviewed the suggested resources, and don't think this has already been solved.

3) Do you have a support contract?

No.

Environment details

Steps to reproduce

  1. Set up a winston logging client

  2. Log with httpRequest data with a latency property (which should be a string according to https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#HttpRequest ):

logger.info("This log seems to mess everything up.", {
  httpRequest: {
    latency: "0.021123013s"
  }
})

logger.info("I'd be fine on my own, but I also never get sent now.")

NB: logging requestMethod (string), requestUrl (string), status (number), userAgent (string), remoteIp (string), serverIp (string), referer (string) all seem fine.

  1. Get the error:
GoogleError: Total timeout of API google.logging.v2.LoggingServiceV2 exceeded 60000 milliseconds before any response was received.
    at repeat (/workspace/node_modules/google-gax/build/src/normalCalls/retries.js:66:31)
    at Timeout._onTimeout (/workspace/node_modules/google-gax/build/src/normalCalls/retries.js:101:25)
    at listOnTimeout (node:internal/timers:564:17)
    at process.processTimers (node:internal/timers:507:7)

Desired behaviour

Ideal: All the logs are sent to Google Cloud Logging correctly.

Better than now: The logs that don't have httpRequest properties on them are at least sent correctly. The logs should be independent - if one fails, that doesn't mean I want to give up on storing all my logs.

losalex commented 1 year ago

Thanks @domdomegg for opening this issue and sorry for inconvenience. I believe this issue was addressed in 732 - please let me know if that resolves the matter.

domdomegg commented 1 year ago

I think that resolves the issue with making the request.

However, I think there's still some issue in the API that it times out rather than returning a 400 or similar. It also might be useful to include in the documentation instructions about this, given a few people have gotten stuck here.

losalex commented 1 year ago

The actual problem happens during protobuf encoding stage, so due to error the request is never sent... But I agree with you @domdomegg - the timeout is absolutely not a proper behavior. I opened an issue 1376 to track this problem and going to close this one - please let me know if you have any more concerns.

domdomegg commented 1 year ago

Thanks, happy to track there :+1: