nose.run() won't notice script or test changes after its first invocation.

[gfixler]

Inside of any running instance of Python, calling nose.run() only works the first time, after which tests are run on in-memory copies of the original imports, with no apparent way to reload them. Reloading nose doesn't work. Deleting .pyc files or setting them to not be created in the first place does nothing. Even deleting test_ modules from sys.modules seems only to work sporadically, and seems to always stop working after awhile, requiring a restart. The only definite fix I've found so far is restarting the shell, which in my case isn't an option, as it runs inside of and is part of the program inside of which I'm trying to run frequent unit tests.

thing.py:

class Thing (object):
    def __init__ (self):
        self.whatever = 10

test_thing.py:

import unittest
import thing

class Test_Thing (unittest.TestCase):

    def setUp (self):
        self.thing = thing.Thing()

    def test_whateverInitValue (self):
        assert self.thing.whatever == 10

To see the error:

1. import nose (anywhere)
2. nose.run() --> test passes
3. don't close shell in which nose has been imported
4. change init of self.whatever to 11 in test.py and save
5. back in the instance of python: nose.run() --> test still passes

This may not be something worthy of a fix, but a workaround would definitely be appreciated.

[gfixler]

I'm not sure if I'm heading down a useful path or not, but I decided to dig into the nose source. I've been following calls, and I think the loaded classes and modules are all in TestProgram.testLoader.suiteClass.suites. I think the reason I can't seem to even delete these modules entirely (so they'll reload from scratch) is ref counts:

import nose
n = nose.core.TestProgram(argv=['', '--attr=!ui'], exit=False)
for i in n.testLoader.suiteClass.suites:
    print i, sys.getrefcount(i)

Some of my test modules/classes have ref counts near 80. Still, I don't know why even deleting and reloading nose doesn't update them. I guess they exist out in Python, separate from nose. I've also noticed that the ref counts get higher every time I run the above code snippet. It seems like it's creating more and more references to those modules and classes with each run. I'm not sure what that means.

[gfixler]

I've struck on something that appears for the moment to be working. It's finally noticing addition and removal of tests, assertion changes, etc. Instead of trying to remove the modules, which seems nearly impossible with all the interconnections (and potentially dangerous?), I'm simply reloading them from the suites dictionary:

import types
import sys
import nose

try:
    print 'reloading test modules...'
    for mySuite in mySuites.testLoader.suiteClass.suites:
        if type(mySuite) == types.ModuleType:
            print 'reloading...', mySuite, sys.getrefcount(mySuite)
            try:
                reload(mySuite)
            except:
                print 'could not reload', mySuite
except:
    print 'unable to reload test modules'
mySuites = nose.core.TestProgram(argv=['', '--attr=ui',
            '--where=/home/gfixler/code/py/work/test/'], exit=False)

Maybe there's already an equivalent of reload() in nose somewhere, but I haven't come across it yet. Maybe an explicit call to something from loader would do it?

[jpellerin]

There's nothing like this built in -- I'd be pretty wary of rerunning tests inside the same process. reload in general is ... hard. Maybe do like the web frameworks do and fork a 2nd process to do the work, then kill/restart it when you want to run tests again?

On Sun, Sep 2, 2012 at 10:38 PM, Gary Fixler notifications@github.comwrote:

I've struck on something that appears for the moment to be working. It's finally noticing addition and removal of tests, assertion changes, etc. Instead of trying to remove the modules, which seems nearly impossible with all the interconnections (and potentially dangerous?), I'm simply reloading them from the suites dictionary:

import types import sys import nose

try: print 'reloading test modules...' for mySuite in mySuites.testLoader.suiteClass.suites: if type(mySuite) == types.ModuleType: print 'reloading...', mySuite, sys.getrefcount(mySuite) try: reload(mySuite) except: print 'could not reload', mySuite except: print 'unable to reload test modules' mySuites = nose.core.TestProgram(argv=['', '--attr=ui', '--where=/home/gfixler/code/py/work/test/'], exit=False)

Maybe there's already an equivalent of reload() in nose somewhere, but I haven't come across it yet. Maybe an explicit call to something from loader would do it?

— Reply to this email directly or view it on GitHubhttps://github.com/nose-devs/nose/issues/546#issuecomment-8228587.

[gfixler]

I'm actually glad to finally hear definitively that nose doesn't handle this. It means I'm [possibly] not crazy. I've been going at this off and on without a SAN check for 5 days. I can appreciate the difficulty with reload.

My specific use case is the Autodesk Maya 3D package. It comes with a standalone python module, which allows the forked process you mention (I actually spawn it on every test from a Vim mapping, independent of Maya being open). Unfortunately, it doesn't allow window or UI creation, because it has no connection with the Qt framework that Maya uses. I could probably spawn a whole new Maya, but it can take more than 7 seconds to load, and then many more filling in the UI and setting things up, and firing off scripts automatically at the end of all that can get messy. It also wouldn't work well for TDD-style, continuous testing.

The new issue today has been that while what I wrote above does reload the modules, references to class instances seem to still hang around. E.g. the code I posted will notice a new test, or a deleted test in the UI class, but it won't let go of an instance of one class in a test of that class. In short, it's too messy to continue down this path.

I think I'll just have to leave off testing the UIs themselves for now and separate the concerns better between UI and what they connect to. UI tests should be manual anyway. I'm just in a nebulous region at the moment, wherein I'm designing a UI creation tool, which is somewhere between worlds. Anyway, thanks!

[jpellerin]

It just occurred to me that a technique like the one used by the isolation plugin might work for you -- before the test run, take a snapshot of sys.modules. After the test run, restore sys.modules to exactly what it was before. That should clear out anything loaded during the test run and allow it to be reloaded by the next one. Maybe!

JP

On Mon, Sep 3, 2012 at 9:03 PM, Gary Fixler notifications@github.comwrote:

I'm actually glad to finally hear definitively that nose doesn't handle this. It means I'm [possibly] not crazy. I've been going at this off and on without a SAN check for 5 days. I can appreciate the difficulty with reload.

My specific use case is the Autodesk Maya 3D package. It comes with a standalone python module, which allows the forked process you mention (I actually spawn it on every test from a Vim mapping, independent of Maya being open). Unfortunately, it doesn't allow window or UI creation, because it has no connection with the Qt framework that Maya uses. I could probably spawn a whole new Maya, but it can take more than 7 seconds to load, and then many more filling in the UI and setting things up, and firing off scripts automatically at the end of all that can get messy. It also wouldn't work well for TDD-style, continuous testing.

The new issue today has been that while what I wrote above does reload the modules, references to class instances seem to still hang around. E.g. the code I posted will notice a new test, or a deleted test in the UI class, but it won't let go of an instance of one class in a test of that class. In short, it's too messy to continue down this path.

I think I'll just have to leave off testing the UIs themselves for now and separate the concerns better between UI and what they connect to. UI tests should be manual anyway. I'm just in a nebulous region at the moment, wherein I'm designing a UI creation tool, which is somewhere between worlds. Anyway, thanks!

— Reply to this email directly or view it on GitHubhttps://github.com/nose-devs/nose/issues/546#issuecomment-8249990.

[gfixler]

Yes! I was at first excited, then worried that actually taking a snapshot of modules was very difficult, unless there's something other than a deep copy or a possibly difficult pickling process I could undertake. However, reading through the docs on the isolation plugin, it seemed like all I really needed to do was add '--with-isolation' to the arguments to nose.run(). I've been testing it for the last half hour, and it seems to be holding up really well to every kind of change. Fingers crossed! Thanks, jpellerin. This might finally be the solution.