Introduced NameError failures due to function names

[jayvdb]

When I add pytest-leaks on a repo that otherwise passes, I see a long list of failures all of the kind NameError: name '...' is not defined, and then one very curious

tests/settings/FunctionMetadataTest.py:104: in check_function_metadata_data_set

    self.assertEqual(metadata.name, name)
E   AssertionError: 't' != 'test_from_function'
E   - t
E   + test_from_function

https://travis-ci.org/jayvdb/coala/jobs/220817392#L1497

The errors all relate to meta-classes, so that could be the cause, and maybe we need to improve the meta-programming of these generated classes to be supported by pytest-leaks. Either way, it would be helpful if pytest-leaks identified the problem automatically or documentation explain the cause.

[abalkin]

@jayvdb - it will be helpful if you could reproduce this issue in a small self-contained example. Your example seems to be using a unittest style test. I wonder if this could be a problem.

[jayvdb]

Also a similar error, and others, in our coala-bears repo which sits on top of the coala repo.

If we cant guess the problem easily, then yes a small test case will be needed. But extracting the problem isnt easy because of the meta programming.

It occurs mostly with the doctest in the source package, not in the tests, so those have no relation to unittest style classes.

[pv]

This seems to be because pytest-leaks runs the test several times, and your test function is not idempotent because it adds a __metadata__ attribute to self.test_from_function, which then breaks it when it runs a second time.

IOW, the documentation could warn about this, but probably there's not something else pytest-leaks should do.