Adds the spark.stage details attribute at the end of the stage, rather than at the beginning.
Motivation
The details attribute contains a large amount of data, including the full stack trace that initiated the stage. When using long-running spans, the span is flushed multiple times, which can significantly increase the ingestion volume.
By adding the details attribute at the end of the stage, we reduce the ingestion volume while still ensuring the information is available once the stage has completed.
What Does This Do
Adds the spark.stage details attribute at the end of the stage, rather than at the beginning.
Motivation
The details attribute contains a large amount of data, including the full stack trace that initiated the stage. When using long-running spans, the span is flushed multiple times, which can significantly increase the ingestion volume.
By adding the details attribute at the end of the stage, we reduce the ingestion volume while still ensuring the information is available once the stage has completed.
Additional Notes
Contributor Checklist
type:
and (comp:
orinst:
) labels in addition to any usefull labelsclose
,fix
or any linking keywords when referencing an issue.Use
solves
instead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]