Aspire Dashboard 'freezes' after 10,000 trace entries

[onionhammer]

This issue is a: (mark with an x)

• [x] bug report -> please search issues before submitting
• [ ] documentation issue or request
• [ ] regression (a behavior that used to work and stopped in a new release)

Issue description

After getting to 10,000 entries, the aspire dashboard ceases to be useful

Steps to reproduce

1. Enable aspire dashboard
2. Log traces to it

Expected behavior [What you expected to happen.] Old logs should disappear

Actual behavior [What actually happened.] New logs never appear

Screenshots

[snehapar9]

Thanks for opening this issue @onionhammer! I was able to reproduce what you're seeing in your screenshot by curling the container app endpoint 15,000 times. However, my observation is traces for 15.000 records are populated but the total results found remains at 10,000. It does not seem to increase beyond that even though more traces are occurring. Do you have the same observation?

[onionhammer]

No, my dashboard traces have since emptied and now no new traces are appearing at all

[snehapar9]

@onionhammer do you mind sharing your sub id/managed cluster name to acasupport@microsoft.com? We will take a look.

[onionhammer]

@onionhammer do you mind sharing your sub id/managed cluster name to acasupport@microsoft.com? We will take a look.

Thanks, sent an email

[snehapar9]

Thanks @onionhammer for providing us this information! We have identified the root cause of the issue.

There was an issue with the otlp authentication because of a race condition while setting the otlp api key. So, even though traces were being transmitted they were not showing up on the dashboard. I performed temporary mitigation steps on your cluster. Can you please confirm you can see traces now?

We will work on getting a long term fix out soon. Thanks again for reporting this. We appreciate it! :)

[onionhammer]

@snehapar9 yep, things are showing again. Thanks!