Rework and fix index paranoia feature

[lebedev]

Hi.

I like this tool a lot. So I invested quite some time in fixing and improving index paranoia feature. Using the magic of virtual machines (vagrant and VirtualBox), m to quickly install/switch between different MongoDB versions and some handwritten scripts and tests, I managed to get this feature working in (almost) every scenario possible.

Now this feature can correctly show index or indexes that was/were used in a query of almost any complexity.

It also shows if a query was covered by index.

And works with shards, showing which indexes were used on each shard.

Compatibility table:

2.4.0  - OK
2.4.14 - OK
2.5.0  - OK
2.5.5  - OK
2.6.0  - OK
2.6.12 - OK
2.7.0  - Bugged, doesn't show names of 2+ indexes used in query
2.7.8  - OK
3.0.0  - OK
3.0.15 - OK
3.1.0  - OK
3.1.9  - OK
3.2.0  - OK
3.2.17 - OK
3.3.0  - OK
3.3.15 - OK
3.4.0  - OK
3.4.9  - OK
3.5.1  - OK
3.5.13 - OK
3.6.0-rc0 - OK

This also closes #159.

[TylerBrock]

Wow, thanks for all the hard work @angly-cat! Any idea if this works with the released version of 3.6.x?

[lebedev]

It should, but to tell for sure some testing needs to be done.

Also, this version has a bug. I detect whether a query is covered by checking if there're FETCH stages. But some time ago I stumbled upon a query, which accessed a collection, but there wasn't a single FETCH stage in execution stats of it. I don't remember the details at all, but I suspect it was related to sharding. So, this implementation on rare occasions acts false-positively.

[TylerBrock]

Thats most likely a covered query (one that can be serviced by just looking at index keys).

For example a query where you project name only and the index used for the match and/or sort is also on name doesn't need to fetch the actual documents to return results.

[lebedev]

No, that query wasn't covered for sure, because it had non-zero docsExamined. But without FETCH stages.

[TylerBrock]

@angly-cat sorry for the delay here, I'd love to get these changes in. Would you still want to work with me to get it over the finish line?

[lebedev]

Would you still want to work with me to get it over the finish line?

Sure. What do you want me to do? I need to warn you, though, that my skills with MongoDB became a little rusty, because I haven't touched it for quite a while. :smile_cat:

[pawelpasterz]

Just FYI, I'm using 4.0.11 and your fix works fine for me

[TylerBrock]

@stennie can we merge this one?

[stennie]

@TylerBrock I'll test this out with modern versions of MongoDB (3.6+). This has been lingering for a while ;-)