test_ignore_broken_distributions_raises_exceptions_by_default only passes by accident

mdickinson commented 3 years ago

The test envisage.tests.test_egg_basket_plugin_manager.EggBasketPluginManagerTestCase.test_ignore_broken_distributions_raises_exceptions_by_default only passes by accident as a result of side-effects from other tests in the test case. When run standalone, it fails.

Here's what happens when the whole test case is run:

(envisage-py310) mdickinson@mirzakhani envisage % python -m unittest -v envisage.tests.test_egg_basket_plugin_manager.EggBasketPluginManagerTestCase
test_find_plugins_in_eggs_on_the_plugin_path (envisage.tests.test_egg_basket_plugin_manager.EggBasketPluginManagerTestCase) ... ok
test_ignore_broken_distributions_loads_good_distributions (envisage.tests.test_egg_basket_plugin_manager.EggBasketPluginManagerTestCase) ... ok
test_ignore_broken_distributions_raises_exceptions_by_default (envisage.tests.test_egg_basket_plugin_manager.EggBasketPluginManagerTestCase) ... ok
test_ignore_broken_plugins_loads_good_plugins (envisage.tests.test_egg_basket_plugin_manager.EggBasketPluginManagerTestCase) ... ok
test_ignore_broken_plugins_raises_exceptions_by_default (envisage.tests.test_egg_basket_plugin_manager.EggBasketPluginManagerTestCase) ... ok
test_ignore_plugins_matching_a_wildcard_in_the_exclude_list (envisage.tests.test_egg_basket_plugin_manager.EggBasketPluginManagerTestCase) ... ok
test_ignore_plugins_whose_ids_are_in_the_exclude_list (envisage.tests.test_egg_basket_plugin_manager.EggBasketPluginManagerTestCase) ... ok
test_only_find_plugins_matching_a_wildcard_in_the_include_list (envisage.tests.test_egg_basket_plugin_manager.EggBasketPluginManagerTestCase) ... ok
test_only_find_plugins_whose_ids_are_in_the_include_list (envisage.tests.test_egg_basket_plugin_manager.EggBasketPluginManagerTestCase) ... ok
test_reflect_changes_to_the_plugin_path (envisage.tests.test_egg_basket_plugin_manager.EggBasketPluginManagerTestCase) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.552s

OK

And here's the result of just running that single test case.

(envisage-py310) mdickinson@mirzakhani envisage % python -m unittest -v envisage.tests.test_egg_basket_plugin_manager.EggBasketPluginManagerTestCase.test_ignore_broken_distributions_raises_exceptions_by_default
test_ignore_broken_distributions_raises_exceptions_by_default (envisage.tests.test_egg_basket_plugin_manager.EggBasketPluginManagerTestCase) ... ERROR

======================================================================
ERROR: test_ignore_broken_distributions_raises_exceptions_by_default (envisage.tests.test_egg_basket_plugin_manager.EggBasketPluginManagerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/mdickinson/Enthought/ETS/envisage/envisage/tests/test_egg_basket_plugin_manager.py", line 184, in test_ignore_broken_distributions_raises_exceptions_by_default
    iter(plugin_manager)
  File "/Users/mdickinson/Enthought/ETS/envisage/envisage/plugin_manager.py", line 96, in __iter__
    for plugin in self._plugins
  File "/Users/mdickinson/Enthought/ETS/envisage/envisage/egg_basket_plugin_manager.py", line 79, in __plugins_default
    plugins = self._harvest_plugins_in_eggs(self.application)
  File "/Users/mdickinson/Enthought/ETS/envisage/envisage/egg_basket_plugin_manager.py", line 149, in _harvest_plugins_in_eggs
    self.on_broken_plugin(entry_point, exc)
  File "/Users/mdickinson/Enthought/ETS/envisage/envisage/egg_basket_plugin_manager.py", line 57, in handle_broken_plugin
    raise exc
  File "/Users/mdickinson/Enthought/ETS/envisage/envisage/egg_basket_plugin_manager.py", line 137, in _harvest_plugins_in_eggs
    plugin = self._create_plugin_from_entry_point(
  File "/Users/mdickinson/Enthought/ETS/envisage/envisage/egg_basket_plugin_manager.py", line 90, in _create_plugin_from_entry_point
    klass = ep.load()
  File "/Users/mdickinson/.venvs/envisage-py310/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2450, in load
    return self.resolve()
  File "/Users/mdickinson/.venvs/envisage-py310/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2456, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/mdickinson/Enthought/ETS/envisage/envisage/tests/bad_eggs/acme.bad-0.1a1-py3.10.egg/acme/bad/bad_plugin.py", line 18, in <module>
    import busted_module  # noqa: F401
ModuleNotFoundError: No module named 'busted_module'

----------------------------------------------------------------------
Ran 1 test in 0.090s

FAILED (errors=1)

mdickinson commented 3 years ago

Related: #440. (It was in trying to fix that issue that I encountered this one.)

mdickinson commented 3 years ago

Here's the traceback from the SystemError that the test expects to be raised:

======================================================================
ERROR: test_on_broken_distribution_raises_exceptions_by_default (envisage.tests.test_egg_basket_plugin_manager.EggBasketPluginManagerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/mdickinson/Enthought/ETS/envisage/envisage/tests/test_egg_basket_plugin_manager.py", line 184, in test_on_broken_distribution_raises_exceptions_by_default
    iter(plugin_manager)
  File "/Users/mdickinson/Enthought/ETS/envisage/envisage/plugin_manager.py", line 96, in __iter__
    for plugin in self._plugins
  File "/Users/mdickinson/Enthought/ETS/envisage/envisage/egg_basket_plugin_manager.py", line 79, in __plugins_default
    plugins = self._harvest_plugins_in_eggs(self.application)
  File "/Users/mdickinson/Enthought/ETS/envisage/envisage/egg_basket_plugin_manager.py", line 127, in _harvest_plugins_in_eggs
    add_eggs_on_path(
  File "/Users/mdickinson/Enthought/ETS/envisage/envisage/egg_utils.py", line 30, in add_eggs_on_path
    on_error(errors)
  File "/Users/mdickinson/Enthought/ETS/envisage/envisage/egg_basket_plugin_manager.py", line 156, in _handle_broken_distributions
    raise SystemError("Cannot find eggs %s" % errors)
SystemError: Cannot find eggs {acme.foo 0.1a11 (/private/var/folders/07/jbbjv8b53bs5y9xyjdyn1zgw0000gn/T/tmpwisenohd/acme.foo-0.1a11-py3.10.egg): VersionConflict(acme.foo 0.1a1 (/Users/mdickinson/Enthought/ETS/envisage/envisage/tests/eggs/acme.foo-0.1a1-py3.10.egg), Requirement.parse('acme.foo==0.1a11'))}

mdickinson commented 3 years ago

Brain dump from yesterday's investigations, in case I don't get time to look at this again in the near future:

In egg_utils, in add_eggs_on_path, we're using WorkingSet.find_plugins from pkg_resources to determine whether there are eggs in the given paths which could be added to the working set without version conflicts
find_plugins returns error information in a dictionary; the intent of the test we're looking at is to test the behaviour when find_plugins returns that error information
There are two calls to add_eggs_on_path. In the second call, the working set we're using already contains everything from sys.path
When the problematic test test_ignore_broken_distributions_raises_exceptions_by_default is run, sys.path has already been contaminated from changes made by earlier tests in the same test class, and already contains an acme.foo egg with version 0.1a1, so the find_plugins call returns error information, which is translated by Envisage to the expected SystemError
If we fix earlier tests to undo changes to sys.path, then the problematic test starts with a clean working set.

mdickinson commented 3 years ago

A similar issue applies to test_ignore_broken_distributions_loads_good_distributions. I'll try to fix it as part of the same PR #443.

enthought / envisage

test_ignore_broken_distributions_raises_exceptions_by_default only passes by accident #441