Can't iterate over multiprocessing.managers.DictProxy

[GitBharath]

Can't iterate over multiprocessing.managers.DictProxy. through pytest-paralell but works fine in python and pytest.

test_Run.py
---------------
from multiprocessing import Process, Manager

def f(d, l):
    d[1] = '1'
    d['2'] = 2
    d[0.25] = None
    l.reverse()

if __name__ == '__main__':
    with Manager() as manager:
    d = manager.dict()
    l = manager.list(range(10))

    p = Process(target=f, args=(d, l))
    p.start()
    p.join()

    print(d)
    print(l)

If you run the

(venv) [tivo@localhost src]$ python test_run.py 
{0.25: None, 1: '1', '2': 2}
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
(venv) [tivo@localhost src]$ 

for pytest:

from multiprocessing import Process, Manager
def test_f():
    d, l = {}, []
    d[1] = '1'
    d['2'] = 2
    d[0.25] = None
    print(d)

with Manager() as manager:
    d = manager.dict()
    l = manager.list(range(10))
   l.reverse()
   print(l)
   p = Process(target=f)
   p.start()
   p.join()

(venv) [tivo@localhost src]$ pytest -v -s test_run.py 
collecting ... [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
{0.25: None, 1: '1', '2': 2}
collected 1 item                                                                                                                                               

test_run.py::test_f {0.25: None, 1: '1', '2': 2}
PASSED

(venv) [tivo@localhost src]$

But If you run through pytest along with pytest-paralell package, it throws error

(venv) [tivo@localhost src]$ pytest -v -s --tests-per-worker auto --workers auto test_run.py 
===================================================================== test session starts ======================================================================
platform linux -- Python 3.4.4, pytest-4.5.0, py-1.8.0, pluggy-0.11.0 -- /home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/src, inifile: pytest.ini
plugins: xdist-1.28.0, remotedata-0.3.1, pipeline-0.3.0, parallel-0.0.9, forked-1.0.2, flake8-1.0.4, cov-2.7.1
collecting ... [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
{0.25: None, 1: '1', '2': 2}
collected 1 item                                                                                                                                               
pytest-parallel: 2 workers (processes), 1 test per worker (thread)

test_run.py::test_f {0.25: None, 1: '1', '2': 2}
PASSED
Traceback (most recent call last):
  File "/home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/venv/bin/pytest", line 10, in <module> sys.exit(main())
  File "/home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/venv/lib/python3.4/site-packages/_pytest/config/__init__.py", line 79, in main return config.hook.pytest_cmdline_main(config=config)
  File "/home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/venv/lib/python3.4/site-packages/pluggy/hooks.py", line 289, in __call__ return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/venv/lib/python3.4/site-packages/pluggy/manager.py", line 68, in _hookexec return self._inner_hookexec(hook, methods, kwargs)
  File "/home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/venv/lib/python3.4/site-packages/pluggy/manager.py", line 62, in <lambda> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/venv/lib/python3.4/site-packages/pluggy/callers.py", line 208, in _multicall return outcome.get_result()
  File "/home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/venv/lib/python3.4/site-packages/pluggy/callers.py", line 80, in get_result raise ex[1].with_traceback(ex[2])
  File "/home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/venv/lib/python3.4/site-packages/pluggy/callers.py", line 187, in _multicall res = hook_impl.function(*args)
  File "/home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/venv/lib/python3.4/site-packages/_pytest/main.py", line 242, in pytest_cmdline_main return wrap_session(config, _main)
  File "/home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/venv/lib/python3.4/site-packages/_pytest/main.py", line 235, in wrap_session session=session, exitstatus=session.exitstatus
  File "/home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/venv/lib/python3.4/site-packages/pluggy/hooks.py", line 289, in __call__ return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/venv/lib/python3.4/site-packages/pluggy/manager.py", line 68, in _hookexec return self._inner_hookexec(hook, methods, kwargs)
  File "/home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/venv/lib/python3.4/site-packages/pluggy/manager.py", line 62, in <lambda> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/venv/lib/python3.4/site-packages/pluggy/callers.py", line 203, in _multicall gen.send(outcome)
  File "/home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/venv/lib/python3.4/site-packages/_pytest/terminal.py", line 678, in pytest_sessionfinish self.summary_stats()
  File "/home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/venv/lib/python3.4/site-packages/_pytest/terminal.py", line 876, in summary_stats (line, color) = build_summary_stats_line(self.stats)
  File "/home/tivo/workspace/ServicePortal/autotestscripts/CAT/scripts/ServerQE/brat/venv/lib/python3.4/site-packages/_pytest/terminal.py", line 1034, in build_summary_stats_line for found_type in stats:
  File "<string>", line 2, in __getitem__
  File "/usr/local/lib/python3.4/multiprocessing/managers.py", line 747, in _callmethod raise convert_to_error(kind, result)
 KeyError: 0
(venv) [tivo@localhost src]$ 

I have the following packages with me

pytest-parallel==0.0.9
pytest-pipeline==0.3.0

Q. What workaround can I do to get this above code PASSED without Error logs ? The issue being the results are not giving me output as to how many test cases are PASSED.

Why is it taking pytest-parallel: 2 workers (processes), 1 test per worker (thread) ? I have provided only one function there !

HINT:

If I add flags --worker 1; this above error does not appear; but usually fails my scripts and hence I am forced to use --tests-per-worker 1 along with it. But paralellism is not present here !

[GitBharath]

Looks like it is similar to the bug https://github.com/enewe101/word2vec/issues/3.

Is the pytest-paralell package supported for python 2 only 'coz @enewe101 says so.

[GitBharath]

Upgraded with python 3.6; I see same error

[blueyed]

See https://github.com/browsertron/pytest-parallel/issues/36 for more info.