Multiprocess plugin fails to pickle nose config, so added plugins are ignored in each process

[justiniso]

It seems that nose's config cannot be pickled (I suspect due to open file handles in logging plugins), which causes it to silently drop plugins on the config.plugins instance when passing the config to child processes (repro at bottom).

The problem is when the config is pickled and passed into the runner:

    # Line 301 of nose.plugins.multiprocess.py
    def startProcess(self, iworker, testQueue, resultQueue, shouldStop, result):
        currentaddr = Value('c',bytes_(''))
        currentstart = Value('d',time.time())
        keyboardCaught = Event()
        p = Process(target=runner,
                   args=(iworker, testQueue,
                         resultQueue,
                         currentaddr,
                         currentstart,
                         keyboardCaught,
                         shouldStop,
                         self.loaderClass,
                         result.__class__,
                         pickle.dumps(self.config)))   # <-- Here we have n plugins

When it is loaded in the spawned process, the config does not have the same plugins as the unpickled config had in master process. It seems to be dropping 3rd-party plugins.

# Line 620 of nose.plugins.multiprocess.py
def runner(ix, testQueue, resultQueue, currentaddr, currentstart,
           keyboardCaught, shouldStop, loaderClass, resultClass, config):
    config = pickle.loads(config)  # <-- Here we have (n - 1) plugins, or however many we started with

# Before pickle
>>> len(self.config.plugins.plugins)
5

# After pickle dump and load
>>> len(config.plugins.plugins)
4

Here is a simple case that uses a 3rd-party plugin (name test_main.py).

import pickle
import logging
import unittest
import nose
import sys
from nose.plugins import Plugin

class TestSimpleCase(unittest.TestCase):

    def test(self):
        return True

class NosePlugin(Plugin):

    name = 'noseplugin'

    def prepareTest(self, test):
        logging.warn('Plugin count (before pickle): {}'.format(len(test.config.plugins.plugins)))

        pickled_config = pickle.dumps(test.config)
        unpickled_config = pickle.loads(pickled_config)

        logging.warn('Plugin count (after pickle): {}'.format(len(unpickled_config.plugins.plugins)))

    def startTest(self, test):
        logging.warn('Plugin was successfully loaded!!!!!')

if __name__ == '__main__':
    argv = ['nosetests', '--with-noseplugin']

    processes = sys.argv[1]

    if int(processes) > 1:
        argv.append('--processes={}'.format(processes))

    logging.warn('Running tests with {} processes'.format(processes))

    nose.main(argv=argv, addplugins=[NosePlugin()])

Or you can just clone: https://github.com/justiniso/nose-multiprocess-repro.

If you run that file with arg 1 (i.e. do not use multiprocess), the custom plugin will be be correctly called in tests. If you run with arg 2 or higher (i.e. use multiprocess plugin), the custom plugin will not be called in tests.

[Kojoley]

Nose completely drops all plugins not at pickle.load, but here at pickle.dump