figure windows not closing properly

[dschneiderch]

with the latest master and with roi_obj_holes, I'm seeing multiple colorbars again. this image here is the first image that is output from my script and it already has 4 colorbars. each subsequent figure has another colorbar. debug = 'None' so I'm not sure where the figures are coming from

Here is the part of my code that output's figures:

Fv = (imgmax_masked - imgmin_masked)
Fv_fig = pcv.visualize.pseudocolor(Fv, mask=mask, background='black', cmap='hsv', min_value=0, max_value=255)
Fv_fig.savefig(os.path.join(outdir, outfn + '_fv.png'), bbox_inches='tight')

Fmp_fig = pcv.visualize.pseudocolor(imgmax, mask=mask, background='black', cmap='hsv', min_value=0, max_value=255)
Fmp_fig.savefig(os.path.join(outdir, outfn + '_fmp.png'), bbox_inches='tight')

FvFm = np.divide(Fv, imgmax_masked, where = imgmax_masked!=0)
FvFm_fig = pcv.visualize.pseudocolor(FvFm, mask=mask, background='black', cmap='hsv', min_value=0, max_value=1)
FvFm_fig.savefig(os.path.join(outdir, outfn + '_yii.png'), bbox_inches='tight')

[nfahlgren]

@dschneiderch I can't replicate the multiple colorbar issue in either Jupyter or running a script on macOS or Linux. On macOS I did notice a window briefly pop up because matplotlib in the pseudocolor function tries to plot the image if there is a graphic backend available. We have to do this to make the plot, even if debug = None.

Here are the versions of things in my environments, let me know if anything stands out to you versus your environment. Are you on Windows?

I tested in Python 3.6 and 3.7.

python -c 'from plantcv import plantcv; print(plantcv.__version__)'

3.2.0+28.g513d2ad

python -c 'import matplotlib; print(matplotlib.get_backend())'

Jupyter: module://ipykernel.pylab.backend_inline
macOS: TkAgg
Linux: Agg

pip freeze

alabaster==0.7.10
anaconda-client==1.6.14
appnope==0.1.0
asn1crypto==0.24.0
attrs==18.1.0
Babel==2.5.3
backcall==0.1.0
backports.functools-lru-cache==1.5
bcrypt==3.1.4
bleach==2.1.3
botocore==1.10.13
certifi==2018.11.29
cffi==1.11.5
chardet==3.0.4
click==6.7
cloudpickle==0.5.3
clyent==1.2.2
CMRESHandler==1.0.0
colorama==0.3.7
coverage==4.5.1
coveralls==1.5.1
cryptography==2.4.2
cycler==0.10.0
Cython==0.28.3
cytoolz==0.9.0.1
dask==0.17.5
decorator==4.3.0
descartes==1.1.0
dill==0.2.8.2
distributed==1.21.8
docopt==0.6.2
docutils==0.14
elasticsearch==6.3.0
entrypoints==0.2.3
globus-sdk==1.5.0
heapdict==1.0.0
html5lib==1.0.1
idna==2.6
imageio==2.3.0
imagesize==1.0.0
ipykernel==4.8.2
ipyparallel==6.2.2
ipython==6.4.0
ipython-genutils==0.2.0
ipywidgets==7.2.1
jedi==0.12.0
Jinja2==2.10
jmespath==0.9.3
jsonschema==2.6.0
jupyter-client==5.2.3
jupyter-contrib-core==0.3.3
jupyter-core==4.4.0
jupyter-latex-envs==1.4.4
kiwisolver==1.0.1
libsubmit==0.4.0
livereload==2.5.2
locket==0.2.0
Markdown==2.6.11
MarkupSafe==1.0
matplotlib==3.0.2
mistune==0.8.3
mizani==0.5.2
mkdocs==0.17.2
mkl-fft==1.0.10
mkl-random==1.0.1
more-itertools==4.1.0
msgpack==0.5.6
msgpack-python==0.5.6
nb-anacondacloud==1.4.0
nb-conda==2.2.1
nb-conda-kernels==2.1.0
nbconvert==5.3.1
nbformat==4.4.0
nbpresent==3.0.2
nbsphinx==0.3.3
networkx==2.1
notebook==5.5.0
numpy==1.15.4
numpydoc==0.8.0
olefile==0.45.1
opencv-python==3.4.5.20
packaging==17.1
palettable==3.1.1
pandas==0.23.4
pandocfilters==1.4.2
paramiko==2.4.1
parsl==0.5.2
parso==0.2.0
partd==0.3.8
patsy==0.5.0
pexpect==4.5.0
pickleshare==0.7.4
Pillow==5.2.0
pkginfo==1.4.2
plantcv==3.2.0+28.g513d2ad
plantcv-hyperspectral==1.0
plotnine==0.5.1
pluggy==0.6.0
prompt-toolkit==1.0.15
psutil==5.4.5
ptyprocess==0.5.2
py==1.5.3
pyasn1==0.4.2
pycparser==2.18
Pygments==2.2.0
PyJWT==1.6.4
PyNaCl==1.2.1
pyOpenSSL==18.0.0
pyparsing==2.2.0
PySocks==1.6.8
pytest==3.5.1
python-dateutil==2.7.3
pytz==2018.4
PyWavelets==0.5.2
PyYAML==3.12
pyzmq==17.0.0
requests==2.18.4
requests-toolbelt==0.8.0
rsa==3.4.2
s3transfer==0.1.13
scikit-image==0.14.0
scikit-learn==0.19.1
scipy==1.1.0
Send2Trash==1.5.0
simplegeneric==0.8.1
six==1.11.0
snowballstemmer==1.2.1
sortedcontainers==1.5.10
spectral==0.19
Sphinx==1.7.4
sphinxcontrib-websupport==1.0.1
statsmodels==0.9.0
tblib==1.3.2
terminado==0.8.1
testpath==0.3.1
toolz==0.9.0
tornado==5.0.2
tqdm==4.23.0
traitlets==4.3.2
twine==1.11.0
typing==3.6.4
urllib3==1.22
versioneer==0.18
wcwidth==0.1.7
webencodings==0.5.1
widgetsnbextension==3.2.1
zict==0.1.3

[dschneiderch]

I’m on Windows 10 using conda. Looks like matplotlib is the same version. I’m not sure what else matters. I am running the script either interactively using the ipykernel or on the commandline with ipython.

Oddly, conda info says I’m on python 3.7 but my prompt says python 3.6.8.

(plantcv) C:\Users\dominikschneider\Documents\phenomics\Kiwamu\SILK>conda info

 active environment : plantcv
active env location : C:\Users\dominikschneider\Anaconda3\envs\plantcv
        shell level : 2
   user config file : C:\Users\dominikschneider\.condarc

populated config files : C:\Users\dominikschneider.condarc conda version : 4.6.4 conda-build version : 3.15.1 python version : 3.7.0.final.0 base environment : C:\Users\dominikschneider\Anaconda3 (writable) channel URLs : https://conda.anaconda.org/conda-forge/win-64 https://conda.anaconda.org/conda-forge/noarch https://repo.anaconda.com/pkgs/main/win-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/free/win-64 https://repo.anaconda.com/pkgs/free/noarch https://repo.anaconda.com/pkgs/r/win-64 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/msys2/win-64 https://repo.anaconda.com/pkgs/msys2/noarch package cache : C:\Users\dominikschneider\Anaconda3\pkgs C:\Users\dominikschneider.conda\pkgs C:\Users\dominikschneider\AppData\Local\conda\conda\pkgs envs directories : C:\Users\dominikschneider\Anaconda3\envs C:\Users\dominikschneider.conda\envs C:\Users\dominikschneider\AppData\Local\conda\conda\envs platform : win-64 user-agent : conda/4.6.4 requests/2.19.1 CPython/3.7.0 Windows/10 Windows/10.0.17134 administrator : False netrc file : None offline mode : False

(plantcv) C:\Users\dominikschneider\Documents\phenomics\Kiwamu\SILK>python Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 18:50:55) [MSC v.1915 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.

from plantcv import plantcv print(plantcv.version) 3.2.0+28.g513d2ad import matplotlib print(matplotlib.get_backend()) Qt5Agg

Pip freeze

(plantcv) C:\Users\dominikschneider\Documents\phenomics\Kiwamu\SILK>pip freeze alabaster==0.7.12 arrow==0.12.1 asn1crypto==0.24.0 astroid==2.1.0 atomicwrites==1.2.1 attrs==18.2.0 autopep8==1.4.3 Babel==2.6.0 backcall==0.1.0 bcrypt==3.1.4 binaryornot==0.4.4 bleach==3.0.2 certifi==2018.11.29 cffi==1.11.5 chardet==3.0.4 Click==7.0 cloudpickle==0.6.1 colorama==0.4.0 cookiecutter==1.6.0 cryptography==2.4.2 cycler==0.10.0 cytoolz==0.9.0.1 dask==1.0.0 decorator==4.3.0 defusedxml==0.5.0 descartes==1.1.0 docutils==0.14 entrypoints==0.2.3 future==0.17.1 idna==2.7 imageio==2.4.1 imagesize==1.1.0 ipykernel==5.1.0 ipython==7.2.0 ipython-genutils==0.2.0 ipywidgets==7.4.2 isort==4.3.4 jedi==0.13.1 Jinja2==2.10 jinja2-time==0.2.0 jsonschema==2.6.0 jupyter==1.0.0 jupyter-client==5.2.3 jupyter-console==6.0.0 jupyter-core==4.4.0 jupyterlab==0.35.3 jupyterlab-server==0.2.0 keyring==16.1.1 kiwisolver==1.0.1 lazy-object-proxy==1.3.1 MarkupSafe==1.1.0 matplotlib==3.0.2 mccabe==0.6.1 mistune==0.8.4 mizani==0.5.2 mkl-fft==1.0.6 mkl-random==1.0.1 more-itertools==4.3.0 nbconvert==5.4.0 nbformat==4.4.0 networkx==2.2 notebook==5.7.4 numpy==1.15.4 numpydoc==0.8.0 olefile==0.46 opencv-python==3.4.4.19 packaging==18.0 palettable==3.1.1 pandas==0.24.1 pandocfilters==1.4.2 paramiko==2.4.2 parso==0.3.1 patsy==0.5.1 pickleshare==0.7.5 Pillow==5.3.0 plantcv==3.2.0+28.g513d2ad plotnine==0.5.1 pluggy==0.8.0 poyo==0.4.2 prometheus-client==0.4.2 prompt-toolkit==2.0.7 psutil==5.4.8 psycopg2==2.7.6.1 py==1.7.0 pyasn1==0.4.4 pycodestyle==2.4.0 pycparser==2.19 pydocstyle==3.0.0 pyflakes==2.0.0 Pygments==2.2.0 pylint==2.2.2 PyNaCl==1.3.0 pyOpenSSL==18.0.0 pyparsing==2.3.0 PySocks==1.6.8 pytest==4.0.1 python-dateutil==2.7.5 python-dotenv==0.10.1 python-jsonrpc-server==0.1.2 python-language-server==0.22.0 pytz==2018.7 PyWavelets==1.0.1 pywin32==223 pywinpty==0.5.4 pyzmq==17.1.2 QtAwesome==0.5.3 qtconsole==4.4.3 QtPy==1.5.2 requests==2.20.1 rope==0.11.0 scikit-image==0.14.1 scipy==1.1.0 Send2Trash==1.5.0 six==1.11.0 snowballstemmer==1.2.1 Sphinx==1.8.2 sphinxcontrib-websupport==1.1.0 spyder==3.3.2 spyder-kernels==0.3.0 statsmodels==0.9.0 terminado==0.8.1 testpath==0.4.2 toolz==0.9.0 tornado==5.1.1 traitlets==4.3.2 typed-ast==1.1.0 urllib3==1.23 wcwidth==0.1.7 webencodings==0.5.1 whichcraft==0.5.2 widgetsnbextension==3.4.2 win-inet-pton==1.0.1 wincertstore==0.2 wrapt==1.10.11 yapf==0.26.0

[dschneiderch]

I guess I can mention that the graphics backend looks different between our setups.

I also notice with conda list

plantcv 3.1.0+71.g620125d pypi_0 pypi

not sure why that doesn’t match with print version For what its worth, I usually install plantcv directly from github repo with python setup.py install

[nfahlgren]

I also have some weird older version of plantcv when I do conda list, must be something from awhile back still in there. Not sure if it's hurting anything but either removing it or creating a new clean environment could help. I'm also installing from the GitHub repo with python setup.py.

What backend does matplotlib use on your Windows setup?

[dschneiderch]

Matplotlib.get_backend says Qt5Agg

---

From: Noah Fahlgren notifications@github.com Sent: Thursday, March 28, 2019 7:43:11 PM To: danforthcenter/plantcv Cc: Schneider, Dominik; Mention Subject: Re: [danforthcenter/plantcv] figure windows not closing properly (#358)

I also have some weird older version of plantcv when I do conda list, must be something from awhile back still in there. Not sure if it's hurting anything but either removing it or creating a new clean environment could help. I'm also installing from the GitHub repo with python setup.py.

What backend does matplotlib use on your Windows setup?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_danforthcenter_plantcv_issues_358-23issuecomment-2D477844701&d=DwMFaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=mfF42lNlc_eXjwWhfTYoU_T7y-aauTyI1-BCcVTO9VI&m=Df3-HxJVShtnrPqLclK4m21tMAHPpblpUk2tNodf9uI&s=xTGG5cXhI-Yroqq02ckUzJ8CFgfoWyWDH8LRmqTmUxs&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AHHZZQA-2DknFo8D1ou-5F8PGc7em3K1AViyks5vbX2-2DgaJpZM4cRF4e&d=DwMFaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=mfF42lNlc_eXjwWhfTYoU_T7y-aauTyI1-BCcVTO9VI&m=Df3-HxJVShtnrPqLclK4m21tMAHPpblpUk2tNodf9uI&s=5zaHkBMrYuBJ1vQuA6iiIyp_m1Ft1h-Lib0kto-GQvY&e=.

[nfahlgren]

Hmm, I just tested on Windows with Anaconda Prompt and I get Agg as the backend and I get an image with only one colorbar.

[dschneiderch]

Weird. I tried multiple new kernels but let me try an old fashioned computer restart.

---

From: Noah Fahlgren notifications@github.com Sent: Thursday, March 28, 2019 9:16:03 PM To: danforthcenter/plantcv Cc: Schneider, Dominik; Mention Subject: Re: [danforthcenter/plantcv] figure windows not closing properly (#358)

Hmm, I just tested on Windows with Anaconda Prompt and I get Agg as the backend and I get an image with only one colorbar.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_danforthcenter_plantcv_issues_358-23issuecomment-2D477860932&d=DwMCaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=mfF42lNlc_eXjwWhfTYoU_T7y-aauTyI1-BCcVTO9VI&m=Rn5XBzkYWeGC9k5PcCoXOsZRGWVTM9K5Q83xvG-3qu8&s=LuHFGO_5O1IpY4Flzbw5qOtTAu_PVbWkEIKgcLFp9dw&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AHHZZR3VJ1IvNusiu4CasLqxAWwv0VLDks5vbZODgaJpZM4cRF4e&d=DwMCaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=mfF42lNlc_eXjwWhfTYoU_T7y-aauTyI1-BCcVTO9VI&m=Rn5XBzkYWeGC9k5PcCoXOsZRGWVTM9K5Q83xvG-3qu8&s=37pZQMsqTM2p2sIeBJG2SzB8yt47paKOHuD0Hf5FUEQ&e=.

[dschneiderch]

@nfahlgren still the same issue even after restart :-(

Here is how I'm calling my code: My function is called image_avg. It basically reads in the image, creates a mask, outputs some pseudocolor images, computes mean for each frame and returns the mean for each frame as a column in the dataframe. I've tried three ways to call the function. There are some noticeable differences:

dfgrps = (df.query('jobdate=="2018-12-15" and sampleid=="A3" and experiment.str.contains("light")')
  .reset_index()
  .groupby(['sampleid','experiment','jobdate','datetime','parameter'])
)
df_avg = dfgrps.apply(image_avg)

This is how i'd like to do it. I basically call the rows of a dataframe that have the metadata for a pair of PSII images. The apply calls image_avg which returns a dataframe with new columns. After going through all groups the dataframe pieces are glued back together.

for grps, grpdf in dfgrps:
    print(grps)
    df_avg = image_avg(grpdf)

A more explicit version of groupby().apply() . this doesn't glue the pieces back together (it only keeps the last one) but I was curious if a simple for loop replicated the multi colorbar behavior I see with groupby().apply(). It does.

dt1 = df.query('jobdate=="2018-12-15" and sampleid=="A3" and experiment=="SILK_light" and parameter=="Y1_ALoff"').iloc[:2]
dt1
image_avg(dt2)
dt2 = df.query('jobdate=="2018-12-15" and sampleid=="A3" and experiment=="SILK_light" and parameter=="Y1_ALoff"').iloc[2:4]
dt2
image_avg(dt2)

lastly, the brute force approach. this one was interesting. within each call to image_avg() the colorbar problem persists. each call to pseudocolor adds another colorbar. But, the colorbar problem does not persist between image_avg(dt1) and image_avg(dt2)

does that offer any clues?

[dschneiderch]

small update: I installed a new conda environment with plantcv 3.2 from bioconda. The problem persists.

[nfahlgren]

@dschneiderch what environment are you using on Windows? Windows Subsystem for Linux, something else?

Could you try adding this to your code to see if the issue is Qt5Agg:

import matplotlib
matplotlib.use('Agg')
from plantcv import plantcv as pcv

[dschneiderch]

I’m using the anaconda prompt via cmd

Using ‘Agg’ didn’t change the output.

From: Noah Fahlgren notifications@github.com Sent: Friday, March 29, 2019 11:10 AM To: danforthcenter/plantcv plantcv@noreply.github.com Cc: Schneider, Dominik dominik.schneider@wsu.edu; Mention mention@noreply.github.com Subject: Re: [danforthcenter/plantcv] figure windows not closing properly (#358)

@dschneiderchhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dschneiderch&d=DwMFaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=mfF42lNlc_eXjwWhfTYoU_T7y-aauTyI1-BCcVTO9VI&m=uvSMDjdfIRqhwL33e9Jjfoy5ma6k2RknfWzE390GeYI&s=WdhG9MpvZhM6OGw8-0l7VmOwlNrPKCnT1sv-NRwKh8Y&e= what environment are you using on Windows? Windows Subsystem for Linux, something else?

Could you try adding this to your code to see if the issue is Qt5Agg:

import matplotlib

matplotlib.use('Agg')

from plantcv import plantcv as pcv

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_danforthcenter_plantcv_issues_358-23issuecomment-2D478097103&d=DwMFaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=mfF42lNlc_eXjwWhfTYoU_T7y-aauTyI1-BCcVTO9VI&m=uvSMDjdfIRqhwL33e9Jjfoy5ma6k2RknfWzE390GeYI&s=vX2YnzoYONPxc6ii_H1xAuHkNqZgUhVo-ZZARMFmVFA&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AHHZZRvrfublnhLNuOCYtwGJc-2D1u78lOks5vblbegaJpZM4cRF4e&d=DwMFaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=mfF42lNlc_eXjwWhfTYoU_T7y-aauTyI1-BCcVTO9VI&m=uvSMDjdfIRqhwL33e9Jjfoy5ma6k2RknfWzE390GeYI&s=4rfe6zuZw6byW__eEQKqFbipyWtAyTqIjSu6EprlL00&e=.

[nfahlgren]

@dschneiderch I was able to replicate the problem by running your program with your sample data. I don't have an explanation for why it's happening though. We tried multiple plots, including in a loop without replicating the issue. The only other difference I can think of is that it's inside a function in your code but I don't see why that would matter.

It is definitely some sort of matplotlib caching issue though. I was able to bypass the problem by modifying your code to add a clear figure step after each plot:

Fv = (imgmax_masked - imgmin_masked)
Fv_fig = pcv.visualize.pseudocolor(Fv, mask=mask, background='black', cmap='hsv', min_value=0, max_value=255)
Fv_fig.savefig(os.path.join(outdir, outfn + '_fv.png'), bbox_inches='tight')
Fv_fig.clf()

Fmp_fig = pcv.visualize.pseudocolor(imgmax, mask=mask, background='black', cmap='hsv', min_value=0, max_value=255)
Fmp_fig.savefig(os.path.join(outdir, outfn + '_fmp.png'), bbox_inches='tight')
Fmp_fig.clf()

FvFm = np.divide(Fv, imgmax_masked, where = imgmax_masked!=0)
FvFm_fig = pcv.visualize.pseudocolor(FvFm, mask=mask, background='black', cmap='hsv', min_value=0, max_value=1)
FvFm_fig.savefig(os.path.join(outdir, outfn + '_yii.png'), bbox_inches='tight')
FvFm_fig.clf()

Hopefully that's an okay workaround for now. We will need to keep poking around at this to see if there is anything we could change on our end to fix it. One solution could be to include the .clf() step in pcv.print_image but that's not fully guaranteed to fix the problem for everyone.

[dschneiderch]

strange error. Thanks for the work around. did you reproduce it on macos or linux too?

[nfahlgren]

I only tested it on Windows but I can test on the other systems next week.