[BUG] Python hangs in jupyter/ipython after mesmerize import when any exception is hit

[EricThomson]

Describe the bug In Jupyter/ipython (but not Python command line), after importing a function from mesmerize, if I hit an exception, Python hangs. (Indefinite asterisk in Jupyter. Booted from ipython nb). In python command line I see the standard error message.

To Reproduce Steps to reproduce the behavior, for example:

1. from mesmerize.viewer.modules.caiman_dfof import get_CNMF_obj
2. Raise some exception (either enter raise or enter something like 1+'a'

Expected behavior A brief exception message, but Python should continue to work.

Copy paste message from the error window (if any)

ERROR:root:Internal Python error in the inspect module. Below is the traceback from this internal error.

ERROR:root:Internal Python error in the inspect module. Below is the traceback from this internal error.

ERROR:root:Internal Python error in the inspect module. Below is the traceback from this internal error.

Traceback (most recent call last):

File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 3343, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 1, in 1+'a' TypeError: unsupported operand type(s) for +: 'int' and 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 2044, in showtraceback stb = value._rendertraceback() AttributeError: 'TypeError' object has no attribute '_rendertraceback'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 1169, in get_records return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 316, in wrapped return f(*args, **kwargs) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 350, in _fixed_getinnerframes records = fix_frame_records_filenames(inspect.getinnerframes(etb, context)) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\inspect.py", line 1490, in getinnerframes frameinfo = (tb.tb_frame,) + getframeinfo(tb, context) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\inspect.py", line 1448, in getframeinfo filename = getsourcefile(frame) or getfile(frame) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\inspect.py", line 696, in getsourcefile if getattr(getmodule(object, filename), 'loader', None) is not None: File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\inspect.py", line 733, in getmodule if ismodule(module) and hasattr(module, 'file'): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\tensorflow__init.py", line 50, in getattr module = self._load() File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\tensorflow__init__.py", line 44, in _load module = _importlib.import_module(self.name__) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\importlib__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 994, in _gcd_import File "", line 971, in _find_and_load File "", line 953, in _find_and_load_unlocked ModuleNotFoundError: No module named 'tensorflow_core.estimator' Traceback (most recent call last): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 3343, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 1, in 1+'a' TypeError: unsupported operand type(s) for +: 'int' and 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 2044, in showtraceback stb = value._rendertraceback() AttributeError: 'TypeError' object has no attribute '_rendertraceback'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 3263, in run_ast_nodes if (await self.runcode(code, result, async=asy)): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 3360, in run_code self.showtraceback(running_compiled_code=True) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 2047, in showtraceback value, tb, tb_offset=tb_offset) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 1436, in structured_traceback self, etype, value, tb, tb_offset, number_of_lines_of_context) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 1336, in structured_traceback self, etype, value, tb, tb_offset, number_of_lines_of_context File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 1193, in structured_traceback tb_offset) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 1150, in format_exception_as_a_whole last_unique, recursion_repeat = find_recursion(orig_etype, evalue, records) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 451, in find_recursion return len(records), 0 TypeError: object of type 'NoneType' has no len()

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 2044, in showtraceback stb = value._rendertraceback() AttributeError: 'TypeError' object has no attribute '_rendertraceback'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 1169, in get_records return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 316, in wrapped return f(*args, **kwargs) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 350, in _fixed_getinnerframes records = fix_frame_records_filenames(inspect.getinnerframes(etb, context)) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\inspect.py", line 1490, in getinnerframes frameinfo = (tb.tb_frame,) + getframeinfo(tb, context) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\inspect.py", line 1448, in getframeinfo filename = getsourcefile(frame) or getfile(frame) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\inspect.py", line 696, in getsourcefile if getattr(getmodule(object, filename), 'loader', None) is not None: File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\inspect.py", line 733, in getmodule if ismodule(module) and hasattr(module, 'file'): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\tensorflow__init.py", line 50, in getattr module = self._load() File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\tensorflow__init__.py", line 44, in _load module = _importlib.import_module(self.name__) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\importlib__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 994, in _gcd_import File "", line 971, in _find_and_load File "", line 953, in _find_and_load_unlocked ModuleNotFoundError: No module named 'tensorflow_core.estimator' Traceback (most recent call last): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 3343, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 1, in 1+'a' TypeError: unsupported operand type(s) for +: 'int' and 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 2044, in showtraceback stb = value._rendertraceback() AttributeError: 'TypeError' object has no attribute '_rendertraceback'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 3263, in run_ast_nodes if (await self.runcode(code, result, async=asy)): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 3360, in run_code self.showtraceback(running_compiled_code=True) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 2047, in showtraceback value, tb, tb_offset=tb_offset) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 1436, in structured_traceback self, etype, value, tb, tb_offset, number_of_lines_of_context) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 1336, in structured_traceback self, etype, value, tb, tb_offset, number_of_lines_of_context File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 1193, in structured_traceback tb_offset) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 1150, in format_exception_as_a_whole last_unique, recursion_repeat = find_recursion(orig_etype, evalue, records) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 451, in find_recursion return len(records), 0 TypeError: object of type 'NoneType' has no len()

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 2044, in showtraceback stb = value._rendertraceback() AttributeError: 'TypeError' object has no attribute '_rendertraceback'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 2895, in _run_cell return runner(coro) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\async_helpers.py", line 68, in _pseudo_sync_runner coro.send(None) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 3072, in run_cell_async interactivity=interactivity, compiler=compiler, result=result) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 3282, in run_ast_nodes self.showtraceback() File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 2047, in showtraceback value, tb, tb_offset=tb_offset) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 1436, in structured_traceback self, etype, value, tb, tb_offset, number_of_lines_of_context) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 1336, in structured_traceback self, etype, value, tb, tb_offset, number_of_lines_of_context File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 1211, in structured_traceback chained_exceptions_tb_offset) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 1150, in format_exception_as_a_whole last_unique, recursion_repeat = find_recursion(orig_etype, evalue, records) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 451, in find_recursion return len(records), 0 TypeError: object of type 'NoneType' has no len()

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\interactiveshell.py", line 2044, in showtraceback stb = value._rendertraceback() AttributeError: 'TypeError' object has no attribute '_rendertraceback'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 1169, in get_records return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 316, in wrapped return f(*args, **kwargs) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\IPython\core\ultratb.py", line 350, in _fixed_getinnerframes records = fix_frame_records_filenames(inspect.getinnerframes(etb, context)) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\inspect.py", line 1490, in getinnerframes frameinfo = (tb.tb_frame,) + getframeinfo(tb, context) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\inspect.py", line 1448, in getframeinfo filename = getsourcefile(frame) or getfile(frame) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\inspect.py", line 696, in getsourcefile if getattr(getmodule(object, filename), 'loader', None) is not None: File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\inspect.py", line 733, in getmodule if ismodule(module) and hasattr(module, 'file'): File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\tensorflow__init.py", line 50, in getattr module = self._load() File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\site-packages\tensorflow__init__.py", line 44, in _load module = _importlib.import_module(self.name__) File "C:\Users\lab_user\miniconda3\envs\mesmerize\lib\importlib__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 994, in _gcd_import File "", line 971, in _find_and_load File "", line 953, in _find_and_load_unlocked ModuleNotFoundError: No module named 'tensorflow_core.estimator'

(Note tensorflow 2.x is installed)

Trace from powershell It is similar so I won't copy/paste it here (but it isn't exactly the same: if you want it I'll drop it in in a comment).

Screenshots

Operating System & specs (CPU, RAM etc.). Windows 10, with caiman installed with mamba/mesmerize witih pip. 128 GB of RAM. Using main branch of mesmerize.

Additional context If I don't import mesmerize stuff it doesn't do this in the environment. E.g., numpy and matplotlib are fine. This creates very strong motivation to not raise exceptions in my code. 💯

[kushalkolar]

Thanks for providing all the details!

I'm not able to reproduce this (in either jupyter or ipython)

I'm not exactly sure what's causing this, however this function isn't really part of the API and isn't meant to be used outside of Mesmerize. A lot of GUI stuff is imported at the point of mesmerize.viewer.modules.caiman_dfof. Pretty much anything within mesmerize.viewer shouldn't be imported outside of the Viewer Console/Script Editor.

Moving this function to mesmerize.analysis.utils might fix it though. Just replace the first line of the function get_sys_config()['_MESMERIZE_N_THREADS'] with psutil.cpu_count() (too of a thread count isn't really important for this). Do you want to make a PR (since I can't produce this error).

[kushalkolar]

Set the base as dev-0.7.2 branch if you want to submit a PR :)

[EricThomson]

Thanks for the thoughts -- I was just wondering if I might be stretching mesmerize beyond its intended domain by using this outside the gui framework. Let me try these changes and see if they can fix the problem and will make a PR if so. The nice thing with just this one function from mesmerize I can seamlessly integrate it into the rest of my caiman workflow :)

Will look this over tonight thanks again!

[kushalkolar]

No worries, if the function is useful in a jupyter analysis workflow it should be outside the GUI modules. I do use quite a lot of mesmerize functions in notebooks, but mostly from mesmerize.analysis or mesmerize.plotting

[EricThomson]

No luck I'm still getting the same behavior when I drop it into analysis.utils. I tried downgrading tensorflow to 1.15 from 2.x and same thing. Even if I just cut out the code and create a new file foo.py and define a function with no dependencies that only prints, I'm getting the same thing. Seems to clearly be a bizarre ipython problem with my environment I will look into it a bit more tomorrow, but after a while tonight it seems like a really tough nut to crack probably not worth tracking down right now.

I'm thinking the smart thing to do is just absorb your useful function into my caiman environment for now, not try to force it into my ipython notebooks using the mesmerize envs. It's a natural interface from mesmerize back into caiman/jupyter for now I think.

[kushalkolar]

I could see if I can reproduce it on windows, you basically use micromamba and installed mesmerize according to the installation docs I guess?

On Thu, Jun 3, 2021, 06:54 Eric Thomson @.***> wrote:

No luck I'm still getting the same behavior when I drop it into analysis.utils. I tried downgrading tensorflow to 1.15 from 2.x and same thing. Seems to clearly be an ipython problem will look into it a bit more tomorrow.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/kushalkolar/MESmerize/issues/56#issuecomment-853560953, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACHXXRCUWELABA5VLEWIS73TQ4DHLANCNFSM457QR3OA .

[EricThomson]

I think I may have just fixed it. While I had downgraded tensorflow to 1.15 I somehow left tensorflow-estimator at 2.3. I just downgraded it, and the weird behavior is gone, I think. I will test this out some more but my test code is working I'm getting some good old-fashioned exceptions!

The last error in the trace was ModuleNotFoundError: No module named 'tensorflow_core.estimator, and last night I noticed that in your Linux install instructions it said to force an install of tf 1.15 (but not in the Windows instructions). So I figured I'd just downgrade all the tensorflow stuff but I missed the tensorflow-estimator somehow.

[EricThomson]

Follow up: Definitely seems to be fixed I'm running through my main code and hitting lots of standard errors without the system hanging (without the original strange behavior) and everything is good.

Upshot: the fix was to add the following to my install instructions (I used mamba instead of conda but ... same thing basically):

conda install tensorflow=1.15
conda install tensorflow-estimator=1.15

[kushalkolar]

Great that you figured it out! Sorry I'll make the installation instructions clearer.