Sanitize test names created by pytest-flakefinder

[pamarcos]

Remove the [N] suffix from test names to consider all tests the same so that testmon works well with test names generated by flakefinder.

This closes https://github.com/tarpas/pytest-testmon/issues/200

I'm not sure this is the most elegant way of doing it, but it definitely does the trick. I'd love to hear some feedback about how to make testmon play nicely with flakefinder 🤗

Of course, this only fixes the problem to deselect all duplicated tests generated by flakefinder. They're still stored in the DB as they are. I didn't change the name used to store them because I didn't know for sure whether we could have duplicated entries or if it was assumed they were unique.

[tarpas]

Hi Pablo. It's unlikely I'll merge something hacky, which your solution seems to be. What I'm open to explore is why it doesn't work out of the box.

I think we should work with the pytest_generate_tests (https://github.com/dropbox/pytest-flakefinder/blob/f6fb30fac91d6e85b6a712cf2a1266654a6c14e7/pytest_flakefinder.py#L59) just fine.

I'll have to study a little bit what is the order of collection, tests generation and collection modification hooks. I think it can be made to work in a better way. Your solution is bound to negatively interact with other plugins.

[pamarcos]

Thanks for taking a look at it, Tibor. I completely agree with you.

As I said in the issue, I think it is hacky indeed 😅

It is the workaround I did in local to make tesmon work with flake-finder, and since I am going to be some days off, I preferred to open the issue and submit what I had for now.

I think we should work with the pytest_generate_tests (https://github.com/dropbox/pytest-flakefinder/blob/f6fb30fac91d6e85b6a712cf2a1266654a6c14e7/pytest_flakefinder.py#L59) just fine.

The problem is that while flake-finder uses tryfirst=True, testmon uses trylast=True. I also thought that maybe passing a parameter in the CLI to set that to a different value might be helpful.