search

edx / edx-arch-experiments

A plugin to include applications under development by the architecture team at edx
GNU Affero General Public License v3.0
0 stars 3 forks source link

Celery tasks missing from DD #700

Open robrap opened 5 days ago

robrap commented 5 days ago

Troy created a ticket: https://help.datadoghq.com/hc/en-us/requests/1747459

So far - @robrap is the only collaborator.

From ticket:

It seems like not all of our celery tasks are ingested into DataDog. When we try to count runs of a specific celery task (update_full_content_metadata_task) we find that there are a different counts depending on whether Metrics or Spans are queried, but more importantly we find that the actual task counts determined via Splunk are much higher than either counts determined via DataDog.

Screenshots attached demonstrate queries against Metrics, APM spans, and also a Splunk query which we consider a reliable source.

Questions:

  1. Why might celery task counts be different between metrics and spans? Are my queries non-equivalent?

  2. Is there a common issue with either/both DD queries that would result in several celery tasks discovered via Splunk not appearing at all in DataDog?

This task is to ensure Troy or others follow-up and get this closed.