run-gmp-sidecar is terminated by cloud run

[iydoowii]

Hi

initially, I was getting Failed to scrape Prometheus endpoint error. I saw that app redirect /metric to /metrics

url -I https://myurl.run.app/metrics
HTTP/2 307 
location: http://myurl.run.app/metrics/
x-cloud-trace-context: cdc6f442b3ec98e8949ecdd9860a0e04;o=1
content-length: 0
date: Thu, 28 Mar 2024 18:34:33 GMT
content-type: text/html
server: Google Frontend
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

I created custom RunMonitoring

apiVersion: monitoring.googleapis.com/v1beta
kind: RunMonitoring
metadata:
  name: run-gmp-sidecar
spec:
  endpoints:
  - port: 8080
    path: /metrics/
    interval: 20s
    scheme: http

after this change, sidecar start working but it is terminated around 2 min later. main app is running file. It seems cloudrun terminates sidecar.
Any idea? why it happens and how can we fix it?

info    otelcol@v0.94.0/collector.go:258    Received signal from OS {"signal": "terminated"}
info    prometheusreceiver/metrics_receiver.go:316  collector: stopping collector's prometheus receiver
info    prometheusreceiver/metrics_receiver.go:324  collector: final scrape complete. Shutting down rest of pipeline
info    prometheusreceiver/metrics_receiver.go:316  collector: stopping collector's prometheus receiver
info    prometheusreceiver/metrics_receiver.go:324  collector: final scrape complete. Shutting down rest of pipeline
info    extensions/extensions.go:59 Stopping extensions...
info    service@v0.94.0/service.go:195  Shutdown complete.
sidecar exited

[ridwanmsharif]

This doesn't seem to be an issue with the sidecar. Instead, Cloud Run terminates the instance (app container and sidecar together) due to an event like scale down or deleted revision. These logs are from such an event. https://cloud.google.com/run/docs/container-contract#lifecycle-services has more details about when Cloud run will terminate an instance, I suspect its because the instance is idle because no requests are being served.

As soon as the instance boots up again (because new requests are rolling in), the sidecar should start up as well.

[iydoowii]

When I check cloud run, Both seems are running(sidecar and cloud run) but, It says that collector shutdown, but it still sends metric to google managed Prometheus. This is not a case that cloud run instance also terminated.

[hs1n]

I faced similar issue, but after comparison of logs. Found out that it's terminating older reversion.