search

mtreinish / stestr

A parallel Python test runner built around subunit
Apache License 2.0
27 stars 33 forks source link

Add a --pdb option to stestr run #14

Closed mtreinish closed 4 years ago

mtreinish commented 7 years ago

Right now if you want to run with pdb stestr breaks that. The way to workaround this is to call subunit.run or testtools.run directly. We point this out in the stestr development docs:

http://stestr.readthedocs.io/en/latest/developer_guidelines.html#running-the-tests

However there is no reason we can't just fast path stestr to call the testr runner directly if we want to use pdb. We probably don't want to store results in the repository in that case.

For an example of this being done, we added this to os-testr: https://github.com/openstack/os-testr/blob/master/os_testr/ostestr.py#L197-L210 we probably could steal that logic and dump it into stestr.commands.run.

mtreinish commented 7 years ago

So we added the - -no-discover option already which does the same thing and calls subunit.run directly. All we need to do here is add to the docs around pdb usage to tell people what to do.

mtreinish commented 7 years ago

I actually don't know if --no-discover will actually work for using pdb. It still subprocesses out which might cause an issue with pdb. Someone will need to test it works correctly with --no-discover before we call it implemented and just document how to do it.

aspiers commented 5 years ago

I just tested it and it works:

$ stestr --test-path=./nova/tests/functional run -n nova.tests.functional.test_servers.TraitsTrackingTests.test_resource_provider_traits
> /home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/placement/requestlog.py(44)__call__()
-> accept = environ.get('HTTP_ACCEPT')
(Pdb)

In contrast:

$ stestr --test-path=./nova/tests/functional run TraitsTrackingTests
> /home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/placement/requestlog.py(44)__call__()
-> accept = environ.get('HTTP_ACCEPT')
(Pdb)
{0} nova.tests.functional.test_servers.TraitsTrackingTests.test_resource_provider_traits [4.816931s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/mock/mock.py", line 1724, in _inner
        return f(*args, **kw)
      File "nova/tests/functional/test_servers.py", line 2047, in test_resource_provider_traits
        self._delete_trait('CUSTOM_FOO')
      File "nova/tests/functional/integrated_helpers.py", line 445, in _delete_trait
        return self.placement_api.delete('/traits/%s' % trait, version='1.6')
      File "nova/tests/functional/fixtures.py", line 45, in delete
        self.fixture._fake_delete(None, url, **kwargs))
      File "nova/tests/functional/fixtures.py", line 158, in _fake_delete
        headers={'x-auth-token': self.token})
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 366, in delete
        return self.request(url, 'DELETE', **kwargs)
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 213, in request
        return self.session.request(url, method, **kwargs)
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/keystoneauth1/session.py", line 814, in request
        resp = send(**kwargs)
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/keystoneauth1/session.py", line 903, in _send_request
        resp = self.session.request(method, url, **kwargs)
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/requests/sessions.py", line 533, in request
        resp = self.send(prep, **send_kwargs)
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/requests/sessions.py", line 646, in send
        r = adapter.send(request, **kwargs)
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/requests/adapters.py", line 449, in send
        timeout=timeout
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
        chunked=chunked)
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/urllib3/connectionpool.py", line 377, in _make_request
        httplib_response = conn.getresponse(buffering=True)
      File "/usr/lib64/python2.7/httplib.py", line 1118, in getresponse
        response = self.response_class(*args, **kwds)
      File "/usr/lib64/python2.7/httplib.py", line 368, in __init__
        self.fp = sock.makefile('rb')
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/wsgi_intercept/__init__.py", line 449, in makefile
        app_result = self.app(environ, start_response)
    wsgi_intercept.WSGIAppError: BdbQuit() at /usr/lib64/python2.7/bdb.py:68
dshcherb commented 5 years ago

-n or --no-discover works for me too for executing a single test case with traces set up:

source .tox/py37/bin/activate
stestr run --no-discover unit_tests.test_actions.TestKeystoneSAMLMellonActions                   
> /home/foobar/src/openstack/charm-keystone-saml-mellon/unit_tests/test_actions.py(45)test_get_sp_metadata()
     44 
---> 45         service_name = 'keystone-foobar-mellon'
     46         self.service_name.return_value = service_name
tox -e py37 -- -n unit_tests.test_actions.TestKeystoneSAMLMellonActions                     
py37 installed: asn1crypto==0.24.0,attrs==19.1.0,backcall==0.1.0,cffi==1.12.3,charmhelpers==0.19.13,charms.openstack==0.0.1.dev1,charms.reactive==1.2.1,cliff==2.15.0,cmd2==0.9.13,colorama==0.4.1,coverage==4.5.3,cryptography==2.7,decorator==4.4.0,entrypoints==0.3,extras==1.0.0,fixtures==3.0.0,flake8==3.7.7,future==0.17.1,ipdb==0.12,ipython==7.5.0,ipython-genutils==0.2.0,jedi==0.14.0,Jinja2==2.10.1,linecache2==1.0.0,lxml==4.3.4,MarkupSafe==1.1.1,mccabe==0.6.1,mock==3.0.5,netaddr==0.7.19,nose==1.3.7,parso==0.5.0,pbr==5.3.1,pexpect==4.7.0,pickleshare==0.7.5,pkg-resources==0.0.0,prettytable==0.7.2,prompt-toolkit==2.0.9,ptyprocess==0.6.0,pyaml==19.4.1,pycodestyle==2.5.0,pycparser==2.19,pyflakes==2.1.1,Pygments==2.4.2,pyparsing==2.4.0,pyperclip==1.7.0,python-mimeparse==1.6.0,python-subunit==1.3.0,PyYAML==5.1.1,six==1.12.0,stestr==2.3.1,stevedore==1.30.1,Tempita==0.5.2,testtools==2.3.0,traceback2==1.4.0,traitlets==4.3.2,unittest2==1.1.0,voluptuous==0.11.5,wcwidth==0.1.7
py37 run-test-pre: PYTHONHASHSEED='0'
py37 runtests: commands[0] | stestr run -n unit_tests.test_actions.TestKeystoneSAMLMellonActions
> /home/foobar/src/openstack/charm-keystone-saml-mellon/unit_tests/test_actions.py(45)test_get_sp_metadata()
     44 
---> 45         service_name = 'keystone-foobar-mellon'
     46         self.service_name.return_value = service_name
mtreinish commented 5 years ago

Ok cool, then lets document this as the recommended way to run tests with pdb in the stestr man page and close the issue.

aspiers commented 5 years ago

@mtreinish commented on June 27, 2019 2:39 PM:

Ok cool, then lets document this as the recommended way to run tests with pdb in the stestr man page and close the issue.

Just as a reminder, even with -n there are issues: https://github.com/mtreinish/stestr/issues/186#issuecomment-474595964

mtreinish commented 5 years ago

@aspiers when you get a second can you try running your pdb workflow with: https://github.com/mtreinish/stestr/pull/271 The code there isn't quite ready to merge yet (it's been a wish list feature I've been working on and off for some time in the background that I finally got running) but it launches things in parallel using multiprocessing.Process and passing the subunit stream as a pipe instead of a subprocess and reading stdout. I'm hoping this will have better support for parallel pdb usage than what was there before.

aspiers commented 5 years ago

After adding import pdb; pdb.set_trace() to a unit testcase and fixing the TypeError: 'sys.version_info' object is not callable issue, I still see issues:

nova $ stestr run --dynamic -n nova.tests.unit.virt.libvirt.test_designer
/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/run.py:374: UserWarning: WARNING: The dynamic scheduler is still experimental. You might encounter issues while using it
  warnings.warn("WARNING: The dynamic scheduler is still experimental. "
'Popen' object has no attribute 'read'
Traceback (most recent call last):
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
    result = cmd.run(parsed_args)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/run.py", line 237, in take_action
    all_attachments=all_attachments, dynamic=args.dynamic)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/run.py", line 410, in run_command
    return run_tests()
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/run.py", line 407, in run_tests
    all_attachments=all_attachments)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/load.py", line 234, in load
    all_attachments)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/load.py", line 267, in _load_case
    case.run(result)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/testtools/testsuite.py", line 158, in run
    for test, route_code in tests:
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/load.py", line 208, in make_tests
    stream, non_subunit_name='stdout')
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/subunit/v2.py", line 272, in __init__
    self.source = subunit.make_stream_binary(source)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/subunit/__init__.py", line 1278, in make_stream_binary
    return _unwrap_text(stream)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/subunit/__init__.py", line 1298, in _unwrap_text
    if type(stream.read(0)) is unicode_type:
AttributeError: 'Popen' object has no attribute 'read'
Traceback (most recent call last):
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/bin/stestr", line 10, in <module>
    sys.exit(main())
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/cli.py", line 118, in main
    return cli.run(argv)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/cliff/app.py", line 281, in run
    result = self.run_subcommand(remainder)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
    result = cmd.run(parsed_args)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/run.py", line 237, in take_action
    all_attachments=all_attachments, dynamic=args.dynamic)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/run.py", line 410, in run_command
    return run_tests()
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/run.py", line 407, in run_tests
    all_attachments=all_attachments)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/load.py", line 234, in load
    all_attachments)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/load.py", line 267, in _load_case
    case.run(result)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/testtools/testsuite.py", line 158, in run
    for test, route_code in tests:
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/load.py", line 208, in make_tests
    stream, non_subunit_name='stdout')
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/subunit/v2.py", line 272, in __init__
    self.source = subunit.make_stream_binary(source)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/subunit/__init__.py", line 1278, in make_stream_binary
    return _unwrap_text(stream)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/subunit/__init__.py", line 1298, in _unwrap_text
    if type(stream.read(0)) is unicode_type:
AttributeError: 'Popen' object has no attribute 'read'
nova $ Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: oslo_context.context, urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
--- Logging error ---
Traceback (most recent call last):
  File "/home/adam/.pyenv/versions/3.6.1/lib/python3.6/logging/__init__.py", line 996, in emit
    self.flush()
  File "/home/adam/.pyenv/versions/3.6.1/lib/python3.6/logging/__init__.py", line 976, in flush
    self.stream.flush()
BrokenPipeError: [Errno 32] Broken pipe
Call stack:
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/oslo_log/log.py", line 203, in logging_excepthook
    getLogger(product_name).critical('Unhandled error', **extra)
Message: 'Unhandled error'
Arguments: ()
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe