metrics are not continuously polled/ingested for every minute

[peijunzh]

Hi, team,

I am deploying dynatrace-gcp-monitor as Kubernetes pod and polling metrics from a GCP monitoring scoping projects. In the scoping project, there are about 10 projects monitored. dynatrace-gcp-monitor polls metrics from the scoping project together with all monitored 10 projects.

Initially, I set the polling interval as 3 minutes, in the running log of dynatrace-gcp-monitor, I find there are errors like: [ERROR][MAIN_LOOP] Single polling timed out and was stopped, timeout: 300s

Based on the code, I find the timeout minutes is interval + 2min, then I increase polling interval to 6 mins, there are also some errors like:

[ERROR][MAIN_LOOP] Single polling timed out and was stopped, timeout: 480s

Once the metrics are ingested into Dynatrace, I find the gathered metrics are not continuously shown in the graph and there are sometimes 2 minutes or 3 minutes gap with no metrics. Looks like once polling is time out, the polled metrics are dropped and will not be processed and ingested into Dynatrace backend.

I have thought maybe I added too many monitored project into the scoping project. But based on doc https://www.dynatrace.com/support/help/setup-and-configuration/setup-on-cloud-platforms/google-cloud-platform/gcp-integrations/gcp-guide/monitor-multiple-projects, "This approach should work up to roughly 375 moderately sized projects."

Could you please advise what configuration should be checked to make sure we can get the metrics are continuously polled and ingested for every minutes?

Thanks.

[NematulloKozimov]

Hi, what does self-monitoring metrics say about ingested line count? Log line should start with "[SFM] Metrics SFM:" - part ("Dynatrace MINT accepted lines count")

If it is around 1M then you might have big projects and you might need to setup 2 (or more) integrations. If that is not the case, please contact Dynatrace support so we can take closer look

[peijunzh]

@NematulloKozimov,

Thanks for your kind reply. The log has lines like: [Dynatrace MINT accepted lines count [per project]: {'nbxx-xxxx-xxx1-xxxxxxx': 1826714}]

Yes, it is more than 1M accepted lines. We use one GCP scoping project which combines multiple monitored projects.

Dynatrace doc https://www.dynatrace.com/support/help/setup-and-configuration/setup-on-cloud-platforms/google-cloud-platform/gcp-integrations/gcp-guide/monitor-multiple-projects says "This approach should work up to roughly 375 moderately sized projects." Is "roughly 375 moderately sized projects" based on "1M" lines as you mention?

Thanks.

[NematulloKozimov]

Yes it is

[peijunzh]

@NematulloKozimov, thanks a lot.