The root cause of the flakiness is the function eventsMatch. This function sorts the APMEvents by converting it into strings and later compares them at which point it uses protocmp.SortRepeated for metricset samples. The metricsets are created using maps (ref) which makes the order of metricsets undefined. While the tests use sorting on the metricset samples to get around this we still end up in an indeterministic state since the first sort doesn't sort on metricset samples (ref).
We can fix the eventsMatch function by first sorting the samples in each APMEvent and then sorting the APMEvent's themselves and this is what the PR does. In addition the PR also refactors the test TestConsumeMetricsWithOTelRemapper which would avoid the special issue anyway.
The flakiness in the test
TestConsumeMetricsWithOTelRemapperwas introduced in https://github.com/elastic/apm-data/pull/309 when we added another scope in the metric.The root cause of the flakiness is the function
eventsMatch. This function sorts the APMEvents by converting it into strings and later compares them at which point it usesprotocmp.SortRepeatedfor metricset samples. The metricsets are created using maps (ref) which makes the order of metricsets undefined. While the tests use sorting on the metricset samples to get around this we still end up in an indeterministic state since the first sort doesn't sort on metricset samples (ref).We can fix the
eventsMatchfunction by first sorting the samples in eachAPMEventand then sorting theAPMEvent's themselves and this is what the PR does. In addition the PR also refactors the testTestConsumeMetricsWithOTelRemapperwhich would avoid the special issue anyway.