VLV index checks

[frasertweedale]

VLV indices are used by Dogtag instances to work out what the next available number is, in the various managed ranges (serials, request IDs, replica IDs, key IDs, etc). If the VLV index becomes incomplete or corrupt, Dogtag attempts to add objects that already exist and operations such as certificate requests fail.

We should add a healthcheck to check relevant VLV indices against the actual database contents and detect inconsistencies.

This is a prime candidate for auto-remediation, too. All that should be required is to create a reindex task.

[rcritten]

How does one know that the VLV index is incomplete or corrupt?

[rcritten]

How does one know that the VLV index is incomplete or corrupt?

[frasertweedale]

@rcritten I'll blog about it, and I shall write a script to automate checking it (it will be helpful for Support). But at a high level it goes like this:

1. Decide an upper bound, e.g. the limit of the identifier range of interest. This is the target attribute value h.
2. Perform a normal (no sort, no VLV) ldapsearch for all the objects of the relevant type
3. Perform a VLV search for the target value h and n preceding values (n = 1 is probably fine)
4. Ignore VLV results where the target attribute value > h.
5. Take VLV result having target attribute of least value, l.
6. Compare the results from the VLV search against the normal search. If there are any objects with target attribute value >= l and <= h, but which are not in the VLV result, then the VLV index is incomplete or corrupt.

[frasertweedale]

Blog post: https://frasertweedale.github.io/blog-redhat/posts/2020-09-17-dogtag-vlv-corruption.html

[tscherf]

@rcritten - Rob, should we clone the ticket into BZ? We have so many customers running into this issue that I'd really like to prioritize the development of this check.

[rcritten]

Sure, feel free.

[tscherf]

Sure, feel free.

https://bugzilla.redhat.com/show_bug.cgi?id=1926909