xunit vs multiprocess

[jpellerin]

What steps will reproduce the problem?

1. Run xunit unit test w/ --processes=2
2. See fails

What is the expected output? What do you see instead?

All of nose's tests should pass under --processes=2

Please use labels and text to provide additional information.

Problem is, xunit creates a file and writes to it when it sees reportable events. For that to work under mp, all the reporting has to happen in the main process. That will require a major redesign of how mp worker processes report their results to the main process.

Google Code Info: Issue #: 267 Author: jpelle...@gmail.com Created On: 2009-05-16T13:04:45.000Z Closed On:

[jpellerin]

Any update to this issue? I would love to see a fix soon as this issue is plaguing me.

Google Code Info: Author: boneill...@gmail.com Created On: 2010-12-22T22:50:58.000Z

[jpellerin]

I don't have time to work on nose myself right now, but would happily accept a patch (with tests) that fixes this issue.

Google Code Info: Author: jpelle...@gmail.com Created On: 2010-12-23T14:19:35.000Z

[jpellerin]

I wish I had the knowledge to fix it myself.

Google Code Info: Author: boneill...@gmail.com Created On: 2010-12-27T18:59:50.000Z

[jpellerin]

I was able to do a quick fix using multiprocessing.Queue and multiprocessing.Array to manage passing data across the processes. Because the multiprocess plugin also uses the python multiprocessing module, this seemed like the most natural extension.

In order for this work, the xunit plugin also needs to be loaded into each new process, this is currently not being done in the mulitprocess plugin.

I'm attaching two files: a new multiprocess plugin and xunit plugin called Xunitmp:

The multiprocess plugin also fixes the timeouts, generator issues as covered in Issue 399:

http://code.google.com/p/python-nose/issues/detail?id=399

The Xunitmp plugin can be enabled with --with-xunitmp

The only disadvantage here is that the queues are global variables. I don't think this will be a problem...

rosen diankov,

Google Code Info: Author: ffxvzero@gmail.com Created On: 2011-02-23T11:18:28.000Z

[jpellerin]

I'm attaching a new version of xunitmultiprocess.py that accurately measures the timings.

rosen diankov,

Google Code Info: Author: ffxvzero@gmail.com Created On: 2011-02-23T12:02:42.000Z

[jpellerin]

Google Code Info: Author: jpelle...@gmail.com Created On: 2011-02-23T13:21:05.000Z

[jpellerin]

I'm attaching a new version of xunitmultiprocess

After doing a little more testing, it became clear that multiprocessing.Manager is a better alternative than multiprocessing.Queue since it doesn't block on exiting processes.

I also made it play nicer with the Capture plugin, now it will fill the XML tags instead of embedding it into the error stream.

Google Code Info: Author: ffxvzero@gmail.com Created On: 2011-02-25T05:35:22.000Z

[jpellerin]

Please get the latest xunitmultiprocess.py from:

https://openrave.svn.sourceforge.net/svnroot/openrave/trunk/test/noseplugins/xunitmultiprocess.py

In order to use it with the new multiprocess.py, have to do:

multiprocess._instantiate_plugins = [xunitmultiprocess.Xunitmp]

Google Code Info: Author: rosen.di...@gmail.com Created On: 2011-03-25T05:51:09.000Z

[jpellerin]

Rosen:

I'm glad to see you're using Manager.Queue rather than the raw Queue.

It's troubling to see coupling between the multiprocess and xunit plugins. They shouldn't need to be aware of each other.

Earlier, I added a fix to the plugintest plugin which may have some bearing here. The cleanest solution was to always use multiprocessing-aware interface whenever the the multiprocessing module was available. This meant that the plugintest system didn't need to be aware of the multiprocess plugin, and vice versa. It would continue to "just work" regardless of whether multiprocessing was used or not.

Would an analogous approach be useful here?

Google Code Info: Author: workitha...@gmail.com Created On: 2011-05-03T04:09:15.000Z

[jpellerin]

does the new multiprocess code already address this issue? Some (maybe all) of the patches here have been applied already in Issue 399

Google Code Info: Author: kumar.mcmillan Created On: 2011-05-03T14:38:58.000Z

[jpellerin]

I tried it today but the problem is still there and was not fixed by Issue 399.

As far as I understand it after trying a few things each test process creates its own Xunit instances, then report, addSucess, addFailure and so on gets called on different instances. As suggested in previous comments I guess it could be solved by sharing same instance of the errorlist. I don't see the need of this affecting the multiprocess plugin at all, but maybe I am missing something?

Google Code Info: Author: anders.r...@gmail.com Created On: 2011-11-22T14:07:00.000Z

[jpellerin]

Ok, I see now why you need to add things in the multiprocess plugin, I thought the multiprocessing module worked differently.

Google Code Info: Author: anders.r...@gmail.com Created On: 2011-11-22T14:40:12.000Z

[santiycr]

What's the status of this bug? I do see lots of updates in mp. I would love to help if that's needed. Is there anything else for this one to happen?

[vogxn]

I'm trying the use of Rosen's plugin for some of my integration tests. The first problem I had was that its options --xunit-file conflicts with xunit plugin's option of the same name. So I changed Rosen's plugin to have --xml-file instead and got it working. Secondly, my tests do not have an attribute capturedOutput which the plugin refers to:

    if test.capturedOutput is not None:
        systemout = '<system-out><![CDATA['+escape_cdata(str(test.capturedOutput))+']]></system-out>'
    xml = """<testcase classname=%(cls)s name=%(name)s time="%(taken)f">

I've commented out the if condition for now and kicked off a run. Will report back any issues I see. Would love to very much see this plugin/ this issue fixed. The multiprocess runs saved 1/3rd of our run time.

Would it be hard to just have xunit plugin send each of its run reports into a seperate file? say moduleName-report.xml? Because this is how unittest-xml-reporting splits the xUnit style reports and they are rendered fine by jenkins CI.

Also - anyone actively working on this issue?

[vogxn]

Ok the xunitmp plugin works for me on my setup. I've been able to run tests in parallel again. Any chance of it being merged with nose mainline? I'm happy to provide any testing data if reqd.

[vogxn]

The nose version I am using the plugin with is 1.2.1

$nosetests -p Plugin xunit Plugin deprecated Plugin skip Plugin multiprocess Plugin failuredetail Plugin capture Plugin logcapture Plugin xunitmp Plugin coverage Plugin marvin Plugin attributeselector Plugin doctest Plugin profile Plugin id Plugin allmodules Plugin collect-only Plugin isolation Plugin pdb

[vogxn]

ok - i'm back. there seem to be a couple of problems with the xunitmp plugin

1) It doesn't show test classes in the XML of the fixtures that were run

2) I often see the following awkward error that doesn't tell me much. I followed some threads on nose-devs that had tests with fixtures created using generators. But I have no such tests:

nose.plugins.multiprocess: ERROR: Worker 1 error running test or returning results Traceback (most recent call last): File "/tests/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 699, in runner test(result) File "/tests/lib/python2.7/site-packages/nose/suite.py", line 176, in __call return self.run(_arg, _kw) File "/tests/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 796, in run test(orig) File "/tests/lib/python2.7/site-packages/nose/suite.py", line 176, in call return self.run(_arg, _kw) File "/tests/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 796, in run test(orig) File "/tests/lib/python2.7/site-packages/nose/suite.py", line 176, in call return self.run(_arg, _kw) File "/tests/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 777, in run result.addError(self, self._exc_info()) File "/tests/lib/python2.7/site-packages/nose/proxy.py", line 134, in addError plugins.addError(self.test, err) File "/tests/lib/python2.7/site-packages/nose/plugins/manager.py", line 99, in call return self.call(_arg, _kw) File "/tests/lib/python2.7/site-packages/nose/plugins/manager.py", line 167, in simple result = meth(_arg, *_kw) File "/tests/lib/python2.7/site-packages/nose/plugins/manager.py", line 334, in addError return self.plugin.addError(test.test, err, capt) AttributeError: 'NoSharedFixtureContextSuite' object has no attribute 'test' Failure: AttributeError ('NoSharedFixtureContextSuite' object has no attribute 'test') ... ERROR

[jszakmeister]

Oooh. That looks like plugins/manager.py is expecting the test attribute to exist but it doesn't. On a plugin I'm working on, I've seen this be the case when there is an error is something like setUpClass(). I'm not sure about this particular trace though.

[donald-ngo]

Hey guys I think I am having the same problem. When I run nosetests -s -v --xml-report-file=donald.xml --xml-accumulate --with-xunit --processes=3

I always get the following output in my xml file. <?xml version="1.0" encoding="UTF-8"?>

It looks like my tests are being run in parallel but the xml file is always like that. Is this the same problem you guys are having?

[winhamwr]

@donald-ngo Yup. That's the problem.

[twiggy]

Is there a way we can vote on this? processes gives us a huge speed up, but having no xunit output blows when the build fails.

[donald-ngo]

I will gladly vote to get this fixed

[vogxn]

+1, would be nice to have this fixed in the upcoming release.

Prasanna.,

From: Donald Ngo [mailto:notifications@github.com] Sent: Wednesday, April 10, 2013 05:43 AM To: nose-devs/nose nose@noreply.github.com Cc: Prasanna Santhanam Subject: Re: [nose] xunit vs multiprocess (#2)

I will gladly vote to get this fixed

— Reply to this email directly or view it on GitHubhttps://github.com/nose-devs/nose/issues/2#issuecomment-16142306.

[vultron81]

Also vote to get this issue fixed

[daubman]

+1, critical for builds with very large numbers of tests

[kylejmcintyre]

I was asked to join github just to vote for this issue being fixed. This has a very negative effect on our CI.

[santiycr]

+1 for getting this fixed. I'll happily buy dinner for whoever takes on this.

[niedbalski]

+1 for fix this issue, would be a great milestone for large tests batteries runners!

[kahunacohen]

Hi, don't mean to nag, but I am just curious if this is actively being worked on by someone. Thanks for any info!

[jszakmeister]

@kahunacohen Not that I know of. For one, that's a particular hard area of nose to work with. Second, I think most of the maintainers (including myself) having a growing interest in Nose 2. I'm not sure what changes need to be made to use Nose 2 (I haven't dived deeply into it just yet), but it does support multiple processes.

[Ignas]

https://pypi.python.org/pypi/nose_xunitmp/0.2 - works on my machine certified https://github.com/Ignas/nose_xunitmp - for bugs and fixes

[twiggy]

Thanks Ignas!!!!!!!!!!!!!!

[senseysensor]

Thanks!!!

[castulo]

I'm using the xunitmp plugin and still is not working :(

[Ignas]

Well, you should probably complain about that in xunitmp github, with details like nose version, operating system and what is failing, and not here...

[nichochar]

Thanks Ignas, that's really helpful.

[crusaderky]

Any plan about fixing this problem in the main nose distribution?

[jszakmeister]

@crusaderky If you'd like to see this fixed, I suggest putting together a PR to fix it. It's not likely to climb up the list otherwise.

I'd also suggest taking a look at Nose 2. It's really the future of Nose, and it has both JUnit XML support and MP support.

[kahunacohen]

nose2 hasn't gained much traction. I've stuck with nose1 after trying out

2. nose2 isn't quite there yet and nobody's really picked it up. No disrespect, just the truth. There's a nose1 plugin that can deal with this issue more or less.

On Wed, Jan 28, 2015 at 6:09 AM, John Szakmeister notifications@github.com wrote:

@crusaderky https://github.com/crusaderky If you'd like to see this fixed, I suggest putting together a PR to fix it. It's not likely to climb up the list otherwise.

I'd also suggest taking a look at Nose 2. It's really the future of Nose, and it has both JUnit XML support and MP support.

— Reply to this email directly or view it on GitHub https://github.com/nose-devs/nose/issues/2#issuecomment-71817042.

[jszakmeister]

nose2 hasn't gained much traction.

It won't unless more people use it, and more importantly, help oout.

I've stuck with nose1 after trying out

2. nose2 isn't quite there yet and nobody's really picked it up. No disrespect, just the truth. There's a nose1 plugin that can deal with this issue more or less.

None taken. OTOH, Nose will go away at some point. It's pretty crufty as it stands, and fixing some issues would require breaking plugins. Others, such as the multiprocessing brokeness, can only be fixed with some significant effort and likely a number of architecture changes--and I apologize, but it's more time than I can commit to.

So at the end of the day, Nose only grows harder to maintain with time--and quite frankly, with all the stuff I've got going on right now, I'm seriously considering dropping out of the open source scene for a while.

Nose 2 is really where the attention needs to be.

[kahunacohen]

I agree that it's up to users to move nose forward, but at the same time if nose2 were really that needed it would be getting done. This says to me that either everyone's leaving nose for other things, or nose 1 suffices enough for most people that they don't see value in nose2, or a combination of the two. I'd say the latter. A lot of people are moving on to other things, and those who continue to use nose1 (I do for many cases) think it's good enough for most situations. The only real issue I've ever had with nose1, was this mp issue, which I've generally been able to work around. We're all busy and generally concentrate on what we need to do to get our jobs done. We contribute to open-source only when we feel the benefit of contributing outweighs the cost. The fact that nose1 may be full of cruft only becomes a problem when it really interferes with our work. Apparently that's not the case for most people (yet).

On Wed, Jan 28, 2015 at 8:35 AM, John Szakmeister notifications@github.com wrote:

nose2 hasn't gained much traction.

It won't unless more people use it, and more importantly, help oout.

I've stuck with nose1 after trying out

2. nose2 isn't quite there yet and nobody's really picked it up. No disrespect, just the truth. There's a nose1 plugin that can deal with this issue more or less.

None taken. OTOH, Nose will go away at some point. It's pretty crufty as it stands, and fixing some issues would require breaking plugins. Others, such as the multiprocessing brokeness, can only be fixed with some significant effort and likely a number of architecture changes--and I apologize, but it's more time than I can commit to.

So at the end of the day, Nose only grows harder to maintain with time--and quite frankly, with all the stuff I've got going on right now, I'm seriously considering dropping out of the open source scene for a while.

Nose 2 is really where the attention needs to be.

— Reply to this email directly or view it on GitHub https://github.com/nose-devs/nose/issues/2#issuecomment-71835136.

[p00j4]

any idea if its fixed or does it work in nose2 now? or --with-xunit & --processes are still not compatible for any of nose version?

However, @jpellerin : thanks for the great help 👍 xunitmp comes as rescue

[therewillbeblood]

i use nose 1.3.7 inside a virtualenv where install junit_xml==1.6 and nose_xunitmp==0.4.0 and then i can successfully run: python ${venv_home}/bin/nosetests -vv --processes=8 --process-timeout=21600 --exe --with-xunitmp --xunitmp-file "${xml_file}" ${files_to_check}

The process timeout is necessary because the default value is ridiculously low