Describe the issue:
The worker profile has a limited span and older data seems to be lost. For example, with the minimal example below, the total CPU time is 24 hours, but the profile never contains more than 4 h 26 min. At one point during the run, the profile looks like this:
30 minutes later, it looks like this:
The previous data is completely gone, as can be seen by the "activity over time" graph at the bottom.
Describe the issue: The worker profile has a limited span and older data seems to be lost. For example, with the minimal example below, the total CPU time is 24 hours, but the profile never contains more than 4 h 26 min. At one point during the run, the profile looks like this:![image](https://github.com/dask/distributed/assets/52597883/532af79a-0158-4035-91f6-fa76f663229c)
30 minutes later, it looks like this:![image](https://github.com/dask/distributed/assets/52597883/37702c5a-e092-4b4a-8572-ddfb17d3a209)
The previous data is completely gone, as can be seen by the "activity over time" graph at the bottom.
This issue has been occurring for several months, most recently with dask and distributed 2024.4.2. You can look at the full discussion on Discourse: https://dask.discourse.group/t/measuring-the-overall-profile-of-long-runs/1859/11
Minimal Complete Verifiable Example:
Environment: