Compilation errors immediately following changes to labscript_utils

[philipstarkey]

Original report (archived issue) by Shaun Johnstone (Bitbucket: shjohnst, GitHub: shjohnst).

---

It seems as though when changes are made to labscript_utils, runmanager notices, and reloads modules to ensure that everything still works. However, the first time you compile after this, there are often unusual errors, such as the following:

#!python

C:\labscript_suite\labscript_utils\qtwidgets\elide_label.py modified: all modules will be reloaded next run.
Traceback (most recent call last):
  File "C:\labscript_suite\userlib\labscriptlib\krb\flat_trap_rb_bottom_z_lev.py", line 1443, in <module>
    stop(t + 200e-6)
  File "C:\labscript_suite\labscript\labscript.py", line 2003, in stop
    generate_code()
  File "C:\labscript_suite\labscript\labscript.py", line 1914, in generate_code
    device.generate_code(hdf5_file)
  File "C:\labscript_suite\labscript_devices\PulseBlaster.py", line 550, in generate_code
    PseudoclockDevice.generate_code(self, hdf5_file)
  File "C:\labscript_suite\labscript\labscript.py", line 899, in generate_code
    Device.generate_code(self, hdf5_file)
  File "C:\labscript_suite\labscript\labscript.py", line 393, in generate_code
    device.generate_code(hdf5_file)
  File "C:\labscript_suite\labscript\labscript.py", line 785, in generate_code
    self.generate_clock()
  File "C:\labscript_suite\labscript\labscript.py", line 773, in generate_clock
    output.expand_timeseries(all_times[clock_line])
  File "C:\labscript_suite\labscript\labscript.py", line 1185, in expand_timeseries
    outarray = self.timeseries[i]['function'](midpoints-self.timeseries[i]['initial time'])
  File "C:\labscript_suite\userlib\labscriptlib\krb\switchablecoildriver.py", line 210, in <lambda>
    self.analogout.add_instruction(t, {'function': lambda x: abs(functions.sine_ramp(duration, initial, final)(x)), 'description':'sinusoidal ramp',
AttributeError: 'NoneType' object has no attribute 'sine_ramp'
Compilation aborted.

Clicking "Engage" again will result in a normal compilation.

Could this be a result of modules not being reloaded before runmanager starts the compilation? Can we make it wait until everything is ready to go so that this is avoided?

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

Huh. I'm not sure why this is happening. It's not a time based thing, ModuleWatcher doesn't reload modules, it only unloads them. They are re-loaded when your code runs again and imports them. And ModuleWatcher holds the Python interpreter's import lock whilst it's unloading modules, so your code will not see unloading half-complete.

the labscript module is whitelisted, batch_compiler.py uses it (we could move the import to be last-second to make it not need to be whitelisted though). So ModuleWatcher should not unload it. It looks like it isn't, however it looks like it is still unloading labscript.functions. I would have thought that should be whitelisted since it is imported by labscript, which is whitelisted. The whitelist is determined by what is imported already when ModuleWatcher is instantiated - it doesn't make much sense that labscript would be in the whitelist but not labscript.functions. So I'll add some debug printlines and see if I can work out why ModuleWatcher is mistakenly unloading labscript.functions.

Relatedly, it would be cool if ModuleWatcher, upon noticing changes in whitelisted modules still caused a message to be printed: "blah blah changed, but it can't be reloaded because this process relies on its state (or whatever). Changes won't be reflected until the analysis/compiler subprocess is restarted".

It would also be cool if ModuleWatcher knew which modules depended on which (which it could do by adding an import hook and tracing imports - we have profiling code out there that does this already), so that it didn't have to unload everything upon one thing changing. For performance reasons it would be nice to not have to reload a massive library like matplotlib or pandas just because some unrelated code changed. That's a separate issue though, and might be unworkable in practice.

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

So I'm not able to reproduce this, perhaps because I don't have the krb.switchablecoildriver module, which is where your traceback is happening. I have the rest of your experiment script though, courtesy of Phil from debugging another problem. If you send me krb.switchablecoildriver I can try reproducing.

Also, I've made a debug mode for ModuleWatcher (labscript_utils pull request #15). If you turn it on by adding debug=True to the instantiation of ModuleWatcher in batch_compiler.py, it will print out what it's unloading and what its whitelist is. That might give some hint as to what's going on.

[philipstarkey]

Original comment by Shaun Johnstone (Bitbucket: shjohnst, GitHub: shjohnst).

---

It's not even the same error each time, and doesn't seem to happen each time. Here's another one today:

#!python

C:\labscript_suite\labscript_utils\qtwidgets\ddsoutput.py modified: all modules will be reloaded next run.
Traceback (most recent call last):
  File "C:\labscript_suite\userlib\labscriptlib\krb\flat_trap_rb_bottom_z_lev.py", line 1443, in <module>
    stop(t + 200e-6)
  File "C:\labscript_suite\labscript\labscript.py", line 2003, in stop
    generate_code()
  File "C:\labscript_suite\labscript\labscript.py", line 1914, in generate_code
    device.generate_code(hdf5_file)
  File "C:\labscript_suite\labscript_devices\PulseBlaster.py", line 550, in generate_code
    PseudoclockDevice.generate_code(self, hdf5_file)
  File "C:\labscript_suite\labscript\labscript.py", line 899, in generate_code
    Device.generate_code(self, hdf5_file)
  File "C:\labscript_suite\labscript\labscript.py", line 393, in generate_code
    device.generate_code(hdf5_file)
  File "C:\labscript_suite\labscript\labscript.py", line 786, in generate_code
    Device.generate_code(self, hdf5_file)
  File "C:\labscript_suite\labscript\labscript.py", line 393, in generate_code
    device.generate_code(hdf5_file)
  File "C:\labscript_suite\labscript\labscript.py", line 393, in generate_code
    device.generate_code(hdf5_file)
  File "C:\labscript_suite\labscript_devices\NIBoard.py", line 44, in generate_code
    IntermediateDevice.generate_code(self, hdf5_file)
  File "C:\labscript_suite\labscript\labscript.py", line 393, in generate_code
    device.generate_code(hdf5_file)
  File "C:\labscript_suite\labscript\labscript.py", line 393, in generate_code
    device.generate_code(hdf5_file)
  File "C:\labscript_suite\labscript_devices\RFBlaster.py", line 189, in generate_code
    os.remove(temp_assembly_filepath)
AttributeError: 'NoneType' object has no attribute 'remove'
Compilation aborted.

I'll pull in your change to ModuleWatcher and see if that gives any further information next time it happens.

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

Ah, I see! ModuleWatcher is unloading modules whilst your shot is compiling!

That's no good. There's a lock that the batch compiler should be acquiring to tell ModuleWatcher to chill out whilst shots are compiling, but it's not being acquired. Will fix.

[philipstarkey]

Original comment by Shaun Johnstone (Bitbucket: shjohnst, GitHub: shjohnst).

---

• changed state from "new" to "resolved"

Fix issue #42, ModuleWatcher unloading modules mid-compilation.

→ \<\<cset e3ad1765a1b814931fdc008077784a3f385ab643>>

[philipstarkey]

Original comment by Shaun Johnstone (Bitbucket: shjohnst, GitHub: shjohnst).

---

Merged in cbillington/runmanager/bugfix (pull request #7)

Fix issue #42, ModuleWatcher unloading modules mid-compilation.

Approved-by: Philip Starkey philip.starkey@monash.edu Approved-by: Shaun Johnstone shaun.johnstone@monash.edu

→ \<\<cset a044bac207e02aed397bcee54aa307f37de977ab>>