Closed donhardman closed 1 week ago
We've discovered an issue with fetching the version from Manticore. In some cases, when the instance is large and takes a long time to preload, it doesn't respond with the version string when we're loading the metric thread. As a solution, we can try lazy loading of versions and/or add retries. For a more comprehensive fix, we could pass it in Buddy launch as a parameter --versions
.
Lazy loading was implemented here: https://github.com/manticoresoftware/manticoresearch-buddy/pull/315
I discovered that about 9 months ago, we updated Buddy to print its version with a 'v' prefix, which was causing an issue.
In some rare cases, Manticore might not load the Buddy version in time when we send a request from Buddy. This led to some valid metrics, which is an unusual and rare condition. To address this, I've extended the regex to capture v-prefixed versions and improved it. I've also implemented lazy loading to eliminate daemon errors on fast startups. I've validated that it works as expected and refactored some things in the metrics, which should help.
This is a critical issue for metrics collection, and I think we should test and release it as soon as possible.
I think it's a good idea to write a test that does the following:
--telemetry-period=1
or some short interval for reports--debugv
For columnar, we should have: columnar_version, knn_version, secondary_version
WITHOUT columnar, we should only have: manticore_version, buddy_version
These last two labels must always be present.
@donhardman the PR has been approved. Pls merge and we'll go ahead with the tests.
The issue was merged. Let's proceed with the tests.
@PavelShilin89 pls look into this issue to develop tests. If smth is not clear, pls talk to @donhardman
Bug Description:
Starting with version 6.3.0, we've encountered an issue with telemetry due to a missing label for the Manticore version in our metrics.
We need to investigate and fix this problem.
Manticore Search Version:
6.3.0+
Operating System Version:
Ubuntu Jammy
Have you tried the latest development version?
None
Internal Checklist:
To be completed by the assignee. Check off tasks that have been completed or are not applicable.