Unable to get sample notebooks to run

[mmroden]

Hi,

I've been trying to get the sample notebooks to run.

I set my environments up with this script:

#!/bin/bash

virtualenv --python=python3.7 venv
source venv/bin/activate
pip3 install -r requirements.txt

My requirements.txt file:

numpy~=1.19.2
matplotlib<=3.1.0
six
h5py~=2.10.0
keras<2.3.0,>=2.2.4
tifffile>=2020.5.11
imagecodecs>=2020.2.18
pathlib2;python_version<'3'
backports.tempfile;python_version<'3.4'
ruamel.yaml>=0.16.10
tqdm
csbdeep<0.6.0,>=0.4.0
Pillow
n2v
jupyter
tensorflow==1.15

For grins, I've also used a smaller requirements file:

n2v
jupyter
tensorflow==1.15

When I run this code from the top of the 2D examples:

from n2v.models import N2VConfig, N2V
import numpy as np
from csbdeep.utils import plot_history
from n2v.utils.n2v_utils import manipulate_val_data
from n2v.internals.N2V_DataGenerator import N2V_DataGenerator
from matplotlib import pyplot as plt
import urllib
import os
import zipfile

I get this error:

Using TensorFlow backend.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-3f7351bee2cc> in <module>
      3 from csbdeep.utils import plot_history
      4 from n2v.utils.n2v_utils import manipulate_val_data
----> 5 from n2v.internals.N2V_DataGenerator import N2V_DataGenerator
      6 from matplotlib import pyplot as plt
      7 import urllib

~/github-src/n2v_mmr/venv/lib/python3.7/site-packages/n2v/internals/N2V_DataGenerator.py in <module>
      3 from glob import glob
      4 import tifffile
----> 5 from matplotlib import image
      6 from csbdeep.utils import _raise
      7 

~/github-src/n2v_mmr/venv/lib/python3.7/site-packages/matplotlib/__init__.py in <module>
    136 # cbook must import matplotlib only within function
    137 # definitions, so it is safe to import from it here.
--> 138 from . import cbook, rcsetup
    139 from matplotlib.cbook import (
    140     MatplotlibDeprecationWarning, dedent, get_label, sanitize_sequence)

~/github-src/n2v_mmr/venv/lib/python3.7/site-packages/matplotlib/rcsetup.py in <module>
     22 from matplotlib import cbook
     23 from matplotlib.cbook import ls_mapper
---> 24 from matplotlib.fontconfig_pattern import parse_fontconfig_pattern
     25 from matplotlib.colors import is_color_like
     26 

~/github-src/n2v_mmr/venv/lib/python3.7/site-packages/matplotlib/fontconfig_pattern.py in <module>
     16 import re
     17 
---> 18 from pyparsing import (Literal, ZeroOrMore, Optional, Regex, StringEnd,
     19                        ParseException, Suppress)
     20 

~/github-src/n2v_mmr/venv/lib/python3.7/site-packages/pyparsing.py in <module>
   5670 stringEnd   = StringEnd().setName("stringEnd")
   5671 
-> 5672 _escapedPunc = Word(_bslash, r"\[]-*.$+^?()~ ", exact=2).setParseAction(lambda s, l, t: t[0][1])
   5673 _escapedHexChar = Regex(r"\\0?[xX][0-9a-fA-F]+").setParseAction(lambda s, l, t: unichr(int(t[0].lstrip(r'\0x'), 16)))
   5674 _escapedOctChar = Regex(r"\\0[0-7]+").setParseAction(lambda s, l, t: unichr(int(t[0][1:], 8)))

~/github-src/n2v_mmr/venv/lib/python3.7/site-packages/pyparsing.py in setParseAction(self, *fns, **kwargs)
   1561             if not all(callable(fn) for fn in fns):
   1562                 raise TypeError("parse actions must be callable")
-> 1563             self.parseAction = list(map(_trim_arity, list(fns)))
   1564             self.callDuringTry = kwargs.get("callDuringTry", False)
   1565         return self

~/github-src/n2v_mmr/venv/lib/python3.7/site-packages/pyparsing.py in _trim_arity(func, maxargs)
   1308     # IF ANY CODE CHANGES, EVEN JUST COMMENTS OR BLANK LINES, BETWEEN THE NEXT LINE AND
   1309     # THE CALL TO FUNC INSIDE WRAPPER, LINE_DIFF MUST BE MODIFIED!!!!
-> 1310     this_line = extract_stack(limit=2)[-1]
   1311     pa_call_line_synth = (this_line[0], this_line[1] + LINE_DIFF)
   1312 

~/github-src/n2v_mmr/venv/lib/python3.7/site-packages/pyparsing.py in extract_stack(limit)
   1292             # special handling for Python 3.5.0 - extra deep call stack by 1
   1293             offset = -3 if system_version == (3, 5, 0) else -2
-> 1294             frame_summary = traceback.extract_stack(limit=-offset + limit - 1)[offset]
   1295             return [frame_summary[:2]]
   1296         def extract_tb(tb, limit=0):

/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/traceback.py in extract_stack(f, limit)
    209     if f is None:
    210         f = sys._getframe().f_back
--> 211     stack = StackSummary.extract(walk_stack(f), limit=limit)
    212     stack.reverse()
    213     return stack

/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/traceback.py in extract(klass, frame_gen, limit, lookup_lines, capture_locals)
    361         if lookup_lines:
    362             for f in result:
--> 363                 f.line
    364         return result
    365 

/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/traceback.py in line(self)
    283     def line(self):
    284         if self._line is None:
--> 285             self._line = linecache.getline(self.filename, self.lineno).strip()
    286         return self._line
    287 

/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/linecache.py in getline(filename, lineno, module_globals)
     14 
     15 def getline(filename, lineno, module_globals=None):
---> 16     lines = getlines(filename, module_globals)
     17     if 1 <= lineno <= len(lines):
     18         return lines[lineno-1]

/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/linecache.py in getlines(filename, module_globals)
     46     try:
     47         if module_globals is None:
---> 48             for mod in sys.modules.values():
     49                 if getattr(mod, '__file__', None) == filename:
     50                     module_globals = mod.__dict__

RuntimeError: dictionary changed size during iteration

My environment contains these libraries and versions (from my most recent attempt, when I've limited matplotlib<3):

absl-py==0.12.0
appnope==0.1.2
argon2-cffi==20.1.0
astor==0.8.1
async-generator==1.10
attrs==20.3.0
backcall==0.2.0
bleach==3.3.0
cached-property==1.5.2
cffi==1.14.5
csbdeep==0.5.2
cycler==0.10.0
decorator==5.0.7
defusedxml==0.7.1
entrypoints==0.3
gast==0.2.2
google-pasta==0.2.0
grpcio==1.37.0
h5py==3.2.1
imagecodecs==2021.3.31
importlib-metadata==4.0.0
ipykernel==5.5.3
ipython==7.22.0
ipython-genutils==0.2.0
ipywidgets==7.6.3
jedi==0.18.0
Jinja2==2.11.3
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.12
jupyter-console==6.4.0
jupyter-core==4.7.1
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.0
Keras==2.2.5
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.2
kiwisolver==1.3.1
Markdown==3.3.4
MarkupSafe==1.1.1
matplotlib==2.2.5
mistune==0.8.4
n2v==0.2.1
nbclient==0.5.3
nbconvert==6.0.7
nbformat==5.1.3
nest-asyncio==1.5.1
notebook==6.3.0
numpy==1.20.2
opt-einsum==3.3.0
packaging==20.9
pandocfilters==1.4.3
parso==0.8.2
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.2.0
prometheus-client==0.10.1
prompt-toolkit==3.0.18
protobuf==3.15.8
ptyprocess==0.7.0
pycparser==2.20
Pygments==2.8.1
pyparsing==2.4.7
pyrsistent==0.17.3
python-dateutil==2.8.1
pytz==2021.1
PyYAML==5.4.1
pyzmq==22.0.3
qtconsole==5.0.3
QtPy==1.9.0
ruamel.yaml==0.17.4
ruamel.yaml.clib==0.2.2
scipy==1.6.2
Send2Trash==1.5.0
six==1.15.0
tensorboard==1.15.0
tensorflow==1.15.0
tensorflow-estimator==1.15.1
termcolor==1.1.0
terminado==0.9.4
testpath==0.4.4
tifffile==2021.4.8
tornado==6.1
tqdm==4.60.0
traitlets==5.0.5
typing-extensions==3.7.4.3
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==1.0.1
widgetsnbextension==3.5.1
wrapt==1.12.1
zipp==3.4.1

[mmroden]

Happy to discuss all the various iterations I went through; Python 3.7 because 3.8 requires tensorflow>2, going to cbdeep==0.6.1 instead of 0.5.2 because of another error, etc etc. All ends in that matplotlib error, and I've tried various versions of that as well, going back to 2.2.5.

[mmroden]

OK, this looks like a problem with matplotlib, not n2v. I can get things to go by splitting up into three code blocks:

import numpy as np
from csbdeep.utils import plot_history
from n2v.utils.n2v_utils import manipulate_val_data

then

from matplotlib import pyplot as plt
import urllib
import os
import zipfile

then

import n2v
from n2v.internals.N2V_DataGenerator import N2V_DataGenerator
datagen = N2V_DataGenerator()

The matplotlib error will happen on the second chunk, if it happens, and then rerunning the cell repeatedly until the problem goes away.

I don't see this happening on 3.8 or 3.9, but since this is a heisenbug, I'm not sure if that was just luck.

[mmroden]

BTW eventually got things working when I downgraded to tensorflow 1.14, 1.15 caused other errors with csbdeep that might be fixed in version 0.61 (which, in turn, would require downgrading n2v...)

[tibuch]

Interesting, I just installed n2v with tf 1.15 last week and had no such problem.

Anyway, thank you for reporting and 'resolving' the issue. I think it is time to finally make the jump to tf 2. I will look into this.

[mmroden]

Thanks-- just ping me if you want an outside tester. I'm on a mac, too, not sure if that makes a difference here (probably, since I can't use tensorflow-gpu).

I started a tf2 conversion myself when I saw that the Adam optimizer is not the tf version, but the keras version, which yields a whole slew of other issues. I figured my time would be better spent trying to get what exists running before I tried to change anything.

Good luck!