Hanging indefinitely during plotting/visualizations

[ChaeYoungKim-snu]

Hi Dong, While analyzing my data using minian, I ran into a problem at a certain step of the analysis, that takes a lot of time than I expected. It was during 'first spatial update' in CNMF.

Specification of my computer is :

1. OS : Linux chaeyoungkim-Z68X-UD3H-B3 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

2. Hardware :
   -4 processors (Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz) -Memory : total used free shared buff/cache available Mem: 15964 11804 707 127 3452 3900 Swap: 9585 4820 4765

3. Packages:

   Name Version Build Channel

   _libgcc_mutex 0.1 main
   ansiwrap 0.8.4 py_0 conda-forge asciitree 0.3.3 py_2 conda-forge attrs 19.3.0 py_0 conda-forge av 6.2.0 py37h866369f_1 conda-forge backcall 0.1.0 py_0 conda-forge backports 1.0 py_2 conda-forge backports.tempfile 1.0 py_0 conda-forge backports.weakref 1.0.post1 py37_1000 conda-forge bleach 3.1.0 py_0 conda-forge bokeh 1.4.0 py37_0 conda-forge bottleneck 1.3.0 py37hc1659b7_0 conda-forge bzip2 1.0.8 h516909a_1 conda-forge ca-certificates 2019.9.11 hecc5488_0 conda-forge cairo 1.16.0 hfb77d84_1002 conda-forge certifi 2019.9.11 py37_0 conda-forge cffi 1.13.2 py37h8022711_0 conda-forge cftime 1.0.4.2 py37hc1659b7_0 conda-forge chardet 3.0.4 py37_1003 conda-forge click 7.0 py_0 conda-forge cloudpickle 1.2.2 py_0 conda-forge colorcet 2.0.1 py_0 conda-forge cryptography 2.8 py37h72c5cf5_0 conda-forge curl 7.65.3 hf8cf82a_0 conda-forge cvxpy 1.0.25 py37he1b5a44_2 conda-forge cvxpy-base 1.0.25 py37he1b5a44_2 conda-forge cycler 0.10.0 py_2 conda-forge cytoolz 0.10.1 py37h516909a_0 conda-forge dask 1.2.2 py_3 conda-forge dask-core 1.2.2 py_0 conda-forge datashader 0.8.0 py_0 conda-forge datashape 0.5.4 py_1 conda-forge dbus 1.13.6 he372182_0 conda-forge decorator 4.4.1 py_0 conda-forge defusedxml 0.6.0 py_0 conda-forge dill 0.3.1.1 py37_0 conda-forge distributed 1.28.1 py37_0 conda-forge ecos 2.0.7.post1 py37hc1659b7_0 conda-forge entrypoints 0.3 py37_1000 conda-forge expat 2.2.5 he1b5a44_1004 conda-forge fastcache 1.1.0 py37h516909a_0 conda-forge fasteners 0.14.1 py_3 conda-forge ffmpeg 4.1.3 h167e202_0 conda-forge ffmpeg-python 0.2.0 pypi_0 pypi fftw 3.3.8 nompi_h7f3a6c3_1110 conda-forge fontconfig 2.13.1 h86ecdb6_1001 conda-forge freetype 2.10.0 he983fc9_1 conda-forge future 0.18.2 py37_0 conda-forge gettext 0.19.8.1 hc5be6a0_1002 conda-forge giflib 5.1.9 h516909a_0 conda-forge glib 2.58.3 py37h6f030ca_1002 conda-forge gmp 6.1.2 hf484d3e_1000 conda-forge gnutls 3.6.5 hd3a4fd2_1002 conda-forge graphite2 1.3.13 hf484d3e_1000 conda-forge gst-plugins-base 1.14.5 h0935bb2_0 conda-forge gstreamer 1.14.5 h36ae1b5_0 conda-forge harfbuzz 2.4.0 h9f30f68_3 conda-forge hdf4 4.2.13 hf30be14_1003 conda-forge hdf5 1.10.5 nompi_h3c11f04_1104 conda-forge heapdict 1.0.1 py_0 conda-forge holoviews 1.12.6 py_0 conda-forge icu 64.2 he1b5a44_1 conda-forge idna 2.8 py37_1000 conda-forge imageio 2.6.1 py37_0 conda-forge importlib_metadata 0.23 py37_0 conda-forge ipykernel 5.1.3 py37h5ca1d4c_0 conda-forge ipython 7.9.0 py37h5ca1d4c_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.5.1 py_0 conda-forge jasper 1.900.1 h07fcdf6_1006 conda-forge jedi 0.15.1 py37_0 conda-forge jinja2 2.10.3 py_0 conda-forge joblib 0.14.0 py_0 conda-forge jpeg 9c h14c3975_1001 conda-forge jsonschema 3.1.1 py37_0 conda-forge jupyter 1.0.0 py_2 conda-forge jupyter_client 5.3.3 py37_1 conda-forge jupyter_console 6.0.0 py_0 conda-forge jupyter_core 4.5.0 py_0 conda-forge kiwisolver 1.1.0 py37hc9558a2_0 conda-forge krb5 1.16.3 h05b26f9_1001 conda-forge lame 3.100 h14c3975_1001 conda-forge libblas 3.8.0 14_openblas conda-forge libcblas 3.8.0 14_openblas conda-forge libclang 9.0.0 default_hde54327_4 conda-forge libcurl 7.65.3 hda55be3_0 conda-forge libedit 3.1.20170329 hf8c457e_1001 conda-forge libffi 3.2.1 he1b5a44_1006 conda-forge libgcc-ng 9.1.0 hdf63c60_0
   libgfortran-ng 7.3.0 hdf63c60_2 conda-forge libiconv 1.15 h516909a_1005 conda-forge liblapack 3.8.0 14_openblas conda-forge liblapacke 3.8.0 14_openblas conda-forge libllvm8 8.0.1 hc9558a2_0 conda-forge libllvm9 9.0.0 hc9558a2_3 conda-forge libnetcdf 4.7.1 nompi_h94020b1_101 conda-forge libopenblas 0.3.7 h6e990d7_3 conda-forge libpng 1.6.37 hed695b0_0 conda-forge libsodium 1.0.17 h516909a_0 conda-forge libssh2 1.8.2 h22169c7_2 conda-forge libstdcxx-ng 9.1.0 hdf63c60_0
   libtiff 4.1.0 hfc65ed5_0 conda-forge libuuid 2.32.1 h14c3975_1000 conda-forge libuv 1.33.1 h516909a_0 conda-forge libwebp 1.0.2 h576950b_1 conda-forge libxcb 1.13 h14c3975_1002 conda-forge libxkbcommon 0.9.1 hebb1f50_0 conda-forge libxml2 2.9.10 hee79883_0 conda-forge llvmlite 0.30.0 py37h8b12597_1 conda-forge locket 0.2.0 py_2 conda-forge lz4-c 1.8.3 he1b5a44_1001 conda-forge markdown 3.1.1 py_0 conda-forge markupsafe 1.1.1 py37h516909a_0 conda-forge matplotlib 3.1.2 py37_0 conda-forge matplotlib-base 3.1.2 py37h250f245_0 conda-forge medpy 0.4.0 pypi_0 pypi mistune 0.8.4 py37h516909a_1000 conda-forge monotonic 1.5 py_0 conda-forge more-itertools 7.2.0 py_0 conda-forge msgpack-python 0.6.2 py37hc9558a2_0 conda-forge multipledispatch 0.6.0 py_0 conda-forge multiprocess 0.70.9 py37h516909a_0 conda-forge natsort 6.1.0 py_0 conda-forge nbconvert 5.6.1 py37_0 conda-forge nbformat 4.4.0 py_1 conda-forge ncurses 6.1 hf484d3e_1002 conda-forge netcdf4 1.5.3 py37hf3213d6_0 conda-forge nettle 3.4.1 h1bed415_1002 conda-forge networkx 2.4 py_0 conda-forge nodejs 12.13.0 h10a4023_0 conda-forge notebook 6.0.1 py37_0 conda-forge nspr 4.23 he1b5a44_0 conda-forge nss 3.47 he751ad9_0 conda-forge numba 0.46.0 py37hb3f55d8_1 conda-forge numcodecs 0.6.4 py37he1b5a44_0 conda-forge numpy 1.17.3 py37h95a1406_0 conda-forge olefile 0.46 py_0 conda-forge opencv 4.1.1 py37h0cc45ee_2 conda-forge openh264 1.8.0 hdbcaa40_1000 conda-forge openssl 1.1.1d h516909a_0 conda-forge osqp 0.6.1 py37hb3f55d8_1 conda-forge packaging 19.2 py_0 conda-forge pandas 0.25.3 py37hb3f55d8_0 conda-forge pandoc 2.7.3 0 conda-forge pandocfilters 1.4.2 py_1 conda-forge panel 0.6.4 0 conda-forge papermill 1.2.1 py37_0 conda-forge param 1.9.2 py_0 conda-forge parso 0.5.1 py_0 conda-forge partd 1.0.0 py_0 conda-forge patsy 0.5.1 py_0 conda-forge pcre 8.43 he1b5a44_0 conda-forge pexpect 4.7.0 py37_0 conda-forge pickleshare 0.7.5 py37_1000 conda-forge pillow 6.2.1 py37h6b7be26_0 conda-forge pip 19.3.1 py37_0 conda-forge pixman 0.38.0 h516909a_1003 conda-forge prometheus_client 0.7.1 py_0 conda-forge prompt_toolkit 2.0.10 py_0 conda-forge psutil 5.6.5 py37h516909a_0 conda-forge pthread-stubs 0.4 h14c3975_1001 conda-forge ptyprocess 0.6.0 py_1001 conda-forge pycparser 2.19 py37_1 conda-forge pyct 0.4.6 py_0 conda-forge pyct-core 0.4.6 py_0 conda-forge pyfftw 0.11.1 py37h240ffea_1002 conda-forge pygments 2.4.2 py_0 conda-forge pyopenssl 19.0.0 py37_0 conda-forge pyparsing 2.4.5 py_0 conda-forge pyqt 5.12.3 py37hcca6a23_1 conda-forge pyqt5-sip 4.19.18 pypi_0 pypi pyqtwebengine 5.12.1 pypi_0 pypi pyrsistent 0.15.5 py37h516909a_0 conda-forge pysocks 1.7.1 py37_0 conda-forge python 3.7.3 h33d41f4_1 conda-forge python-dateutil 2.8.1 py_0 conda-forge pytz 2019.3 py_0 conda-forge pyviz_comms 0.7.2 py_0 conda-forge pywavelets 1.1.1 py37hc1659b7_0 conda-forge pyyaml 5.1.2 py37h516909a_0 conda-forge pyzmq 18.1.0 py37h1768529_0 conda-forge qt 5.12.5 hd8c4c69_1 conda-forge qtconsole 4.5.5 py_0 conda-forge readline 8.0 hf8c457e_0 conda-forge requests 2.22.0 py37_1 conda-forge scikit-image 0.16.2 py37hb3f55d8_0 conda-forge scikit-learn 0.21.3 py37hcdab131_0 conda-forge scipy 1.3.2 py37h921218d_0 conda-forge scs 2.1.1.2 py37h4ff444d_3 conda-forge send2trash 1.5.0 py_0 conda-forge setuptools 41.6.0 py37_1 conda-forge simpleitk 1.2.3 pypi_0 pypi six 1.13.0 py37_0 conda-forge sk-video 1.1.10 pypi_0 pypi sortedcontainers 2.1.0 py_0 conda-forge sqlite 3.30.1 hcee41ef_0 conda-forge statsmodels 0.10.1 py37hc1659b7_2 conda-forge tblib 1.4.0 py_0 conda-forge tenacity 6.0.0 py37_0 conda-forge terminado 0.8.3 py37_0 conda-forge testpath 0.4.4 py_0 conda-forge textwrap3 0.9.2 py_0 conda-forge tifffile 0.15.1 py37hc1659b7_1001 conda-forge tk 8.6.9 hed695b0_1003 conda-forge toolz 0.10.0 py_0 conda-forge tornado 6.0.3 py37h516909a_0 conda-forge tqdm 4.38.0 py_0 conda-forge traitlets 4.3.3 py37_0 conda-forge urllib3 1.25.7 py37_0 conda-forge wcwidth 0.1.7 py_1 conda-forge webencodings 0.5.1 py_1 conda-forge wheel 0.33.6 py37_0 conda-forge widgetsnbextension 3.5.1 py37_0 conda-forge x264 1!152.20180806 h14c3975_0 conda-forge xarray 0.14.0 py_0 conda-forge xorg-kbproto 1.0.7 h14c3975_1002 conda-forge xorg-libice 1.0.10 h516909a_0 conda-forge xorg-libsm 1.2.3 h84519dc_1000 conda-forge xorg-libx11 1.6.9 h516909a_0 conda-forge xorg-libxau 1.0.9 h14c3975_0 conda-forge xorg-libxdmcp 1.1.3 h516909a_0 conda-forge xorg-libxext 1.3.4 h516909a_0 conda-forge xorg-libxrender 0.9.10 h516909a_1002 conda-forge xorg-renderproto 0.11.1 h14c3975_1002 conda-forge xorg-xextproto 7.3.0 h14c3975_1002 conda-forge xorg-xproto 7.0.31 h14c3975_1007 conda-forge xz 5.2.4 h14c3975_1001 conda-forge yaml 0.1.7 h14c3975_1001 conda-forge zarr 2.3.2 py37_0 conda-forge zeromq 4.3.2 he1b5a44_2 conda-forge zict 1.0.0 py_0 conda-forge zipp 0.6.0 py_0 conda-forge zlib 1.2.11 h516909a_1006 conda-forge zstd 1.4.3 h3b9ef0a_0 conda-forge

Here I attach a screenshot of my monitor. Thank you so much!

[phildong]

Thank you for the report @ChaeYoungKim-snu ! This is definitely not the time it should take. In fact, it's just plotting the results and should take a few seconds. I believe this is a bug related to the interaction between dask and holoviews, and interestingly was only reproducible on certain Linux machines so far. A quick fix would be to call the compute method on every single variable you are plotting. For example in your screenshot, you would replace b_init, f_init, b_spatial, f_spatial with b_init.compute(), f_init.compute(), b_spatial.compute(), f_spatial.compute() respectively. This looks stupid but should work. Sadly you have to do this for every plotting cell that has this issue. I'm still hesitant to push this as a fix and hoping this will just go away in future updates of dask or holoviews, but we will see.

[phildong]

There has been some speculation that this is related to over-scheduling with dask. See some ref here: https://github.com/dask/dask/issues/5229 It seems the easiest way to avoid this is to set environment variables according to dask documentation: https://docs.dask.org/en/latest/array-best-practices.html?highlight=OMP_NUM_THREADS#avoid-oversubscribing-threads Specifically adding the following lines at the beginning of the notebook (right after imports) should work without the need to add those compute()s:

os.environ["OMP_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["NUMBA_NUM_THREADS"] = "1"

If you don't mind could you give this a try and let me know if it works? (I'm having trouble reproducing this on my side)

[ForoughHa]

Hi @phildong

So I was running the pipeline with no problems until I got to the "first spatial update" section of the code. The analysis and plotting in the last cell of this section is taking a very long time being stuck at 95% usually! I tried running it several times but didn't help. Then I added

os.environ["OMP_NUM_THREADS"] = "1" os.environ["MKL_NUM_THREADS"] = "1" os.environ["OPENBLAS_NUM_THREADS"] = "1" os.environ["NUMBA_NUM_THREADS"] = "1"

to the beginning of the pipeline, restarted my kernel, ran everything from the start but faced the same issue when I got to this cell again.

I also tried adding .compute() but no luck. The process doesn't even seem to be starting this time. Not even 1% progress after a long wait time:

[ForoughHa]

Just as more evidence of how long it takes:

[phildong]

@ForoughHa I believe once it hangs and you interrupted it won't work for that session. Have you tried restarting the kernel and running them with compute() without anything hanging/interrupted before?

[phildong]

FYI I have reported this to holoviews.

[ForoughHa]

@phildong Thanks a lot for following up so quickly. So I did as you said and restarted and ran the kernel again after adding "compute()" and this time, it did run the previous problematic block but got stuck in a later step at "first temporal update" I am attaching below the two latest blocks which have been running for a while now:

(So cell number 52 took 5hrs to complete and now it's been 3hrs that the next cell is running and I haven't seen progress from 79% for a long time now)

[ForoughHa]

@phildong I am now going to add .compute() to this block (and probably all the next blocks as well) to see how much it can help. Just a quick question, is 5hrs for cell 52 above a normal run time?

[ForoughHa]

This time got stuck in an even earlier step! Seems to be an issue I might not be thinking about. This step used to run rather smoothly. It has been stuck on 59% for more than 55 minutes now. Am I missing something here?

[nshukla5]

Hi @phildong and @ForoughHa were you able to resolve the hanging at that cell? I was able to make it to that point by adding the .compute() to the variables but now I'm running into the same issue at the same step as ForoughHa.

[ForoughHa]

Hi @nshukla5, No, unfortunately, I have been facing the issue ever since and haven't been able to fix it yet. I also tried to removing everything and cloning the latest version of the package from the "dev" branch once again and doing it all over but didn't help. Also since this is not an "error" per see, I cannot really dig dip into it and I am hoping @phildong might have an answer to this sometime which could help us as I haven't' really been able to use the package because of this issue.

[nshukla5]

Hi @ForoughHa,

Thanks for the quick response! That is disappointing, have you tried any other packages to analyze miniscope calcium signaling data?

[wmau]

Hi @ForoughHa and @nshukla5 . That looks like the visualization step right before the first temporal update. Try replacing the cell with the code below, which adds .compute()s to a few plotted variables. Unfortunately this is a buggy holoviews/dask interaction which is out of our ability to fix. Adding .compute()s to every plotting variable should be a temporary fix.

%%time if interactive: p_ls = [1] sprs_ls = [0.01, 0.05, 0.1, 2] add_ls = [20] noise_ls = [0.06] YA_dict, C_dict, S_dict, g_dict, sig_dict, Adict = [dict() for in range(6)] YrA = compute_trace(Y, A_sub, b_spatial, C_sub, f_spatial).persist() for cur_p, cur_sprs, cur_add, cur_noise in itt.product(p_ls, sprs_ls, add_ls, noise_ls): ks = (cur_p, cur_sprs, cur_add, cur_noise) print("p:{}, sparse penalty:{}, additional lag:{}, noise frequency:{}" .format(cur_p, cur_sprs, cur_add, cur_noise)) YrA, cur_C, cur_S, cur_B, cur_C0, cur_sig, cur_g, cur_scal = update_temporal( Y, A_sub, b_spatial, C_sub, f_spatial, sn_spatial, YrA=YrA, sparse_penal=cur_sprs, p=cur_p, use_spatial=False, use_smooth=True, add_lag = cur_add, noise_freq=cur_noise) YA_dict[ks], C_dict[ks], S_dict[ks], g_dict[ks], sig_dict[ks], A_dict[ks] = ( YrA.compute(), cur_C.compute(), cur_S.compute(), cur_g.compute(), cur_sig.compute(), A_sub.compute()) hv_res = visualize_temporal_update( YA_dict, C_dict, S_dict, g_dict, sig_dict, A_dict, kdims=['p', 'sparse penalty', 'additional lag', 'noise frequency'])

[ForoughHa]

Dear @wmau,

Thank you very much for getting back to us. I did change the mentioned cell with the one you provided and also included ".comute()" in a number of more cells after this where we needed to draw an object with holoviews. Things now run to a further point but I still face the same issue this time in the cell right before "merge units" before "test parameters for spatial update" for the second spatial update.

What I have changed in the cell preceding this one is copied below:

Code below

hv.output(size=output_size) if interactive: h, w = A_spatial.sizes['height'], A_spatial.sizes['width'] im_opts = dict(aspect=w/h, frame_width=500, cmap='Viridis') cr_opts = dict(aspect=3, frame_width=1000) bad_units = list(set(A_spatial.coords['unit_id'].values) - set(A_temporal.coords['unit_id'].values)) bad_units.sort() if len(bad_units)>0: hv_res = (hv.NdLayout({ "Spatial Footprin": regrid(hv.Dataset(A_spatial.sel(unit_id=bad_units).compute().rename('A')) .to(hv.Image, kdims=['width', 'height'])).opts(*im_opts), "Spatial Footprints of Accepted Units": regrid(hv.Image(A_temporal.sum('unit_id').compute().rename('A'), kdims=['width', 'height'])).opts(**im_opts) })

• datashade(hv.Dataset(YrA.sel(unit_id=bad_units).compute().rename('raw')) .to(hv.Curve, kdims=['frame'])).opts(*cr_opts).relabel("Temporal Trace")).cols(1) display(hv_res) else: print("No rejected units to display")

And this cell itself gives me the desired output but then the next cell which can be seen in the attached screenshot still hangs indefinitely while running. Have I missed something or misinterpreted where ".compute()" must be added?

[wmau]

The inputs to visualize_temporal_update also all need .compute(): YrA.compute(), C_temporal.compute(), S_temporal.compute(), g_temporal.compute(), sig_temporal.compute(), A_temporal.compute()

[ForoughHa]

Hi @wmau ,

Of course! I had missed that visualization function there. Thank you so much for your help. The pipeline is now up and running for me. @nshukla5 I think your problem would also be solved if you would add ".compute()" to all the inputs of the visualization functions in different cells.

[wmau]

Glad I could help! :) @nshukla5 let us know if you have any questions or if this doesn't resolve your issue.

[nshukla5]

Hi @wmau, the whole pipeline is working for me as well. Thank you for all of your help!

[ghost]

Hi guys, could you share the cells where you added 'compute()' ? I am still facing the same problem and I dont know where I am missing something

[ForoughHa]

Hi guys, could you share the cells where you added 'compute()' ? I am still facing the same problem and I dont know where I am missing something

Hi there,

I think mentioning the cells one by one would be a bit frustrating for all of us and we might even miss a couple. So I thought I would just attach the notebook I am running right now but this comment box doesn't support attaching .ipynb files. Would you be willing to share your email address with me so that I can email the file to you? When you receive this file, I would suggest you only replace the cells for the final parts of the pipeline (Spatial and Temporal updates) as I have also made some more changes on other cells (updates to some parameters) that you might not want to use. Most of these changes are at the beginning of the pipeline and in parameter exploration steps so you can just skip those and use the updated cells from after the point your code is stuck which would be the spatial/temporal updates I assume. Those parts are exactly like the original pipeline in my file as well but just with .compute() added to fix our runtime issue.

[ghost]

awesome! that would be very helpful. I had it all running smoothly until numpy was upgraded then I cannot make the pipeline to run. my email is richardson.leao at neuro.ufrn.br

[wmau]

@cineguerrilha can you show me where the notebook hangs for you? Also numpy should not cause the hangs described in this issue. Can you open a separate issue for this error?

@ForoughHa thank you for offering your notebook. We are holding off on pushing the .compute() changes in hopes that dask/holoviews will fix their issue. And we are also working on a more permanent with fix with a team of software engineers. If none of those prove fruitful, we will roll out the fix with the .compute()s.

[ghost]

@wmau

%%time YrA, C_temporal, S_temporal, B_temporal, C0_temporal, sig_temporal, g_temporal, scale = update_temporal( Y, A_spatial, b_spatial, C_spatial, f_spatial, sn_spatial, **param_first_temporal) A_temporal = A_spatial.sel(unit_id = C_temporal.coords['unit_id'])

first temporal:

grouping overlaping units

C:\Users\richardson\Anaconda3\envs\minian\lib\site-packages\dask\array\routines.py:264: PerformanceWarning: Increasing number of chunks by factor of 76 intermediate = blockwise( C:\Users\richardson\Anaconda3\envs\minian\lib\site-packages\dask\array\routines.py:264: PerformanceWarning: Increasing number of chunks by factor of 76 intermediate = blockwise(

[ghost]

sorry, i did not read properly the title (but in my case, the script hangs in other instances). I tried now to interrupt it and save:

%%time A_spatial = save_minian(A_spatial.rename('A_spatial'), param_save_minian) b_spatial = save_minian(b_spatial.rename('b_spatial'), param_save_minian) C_spatial = save_minian(C_spatial.rename('C_spatial'), param_save_minian) f_spatial = save_minian(f_spatial.rename('f_spatial'), param_save_minian) sn_spatial = save_minian(sn_spatial.rename('sn_spatial'), **param_save_minian)

but saving also makes the notebook to hang

[EricThomson]

Just including what wmau did above, but with code formatting so hopefully a little easier to read:

%%time
if interactive:
    p_ls = [1]
    sprs_ls = [0.01, 0.05, 0.1, 2]
    add_ls = [20]
    noise_ls = [0.06]
    YA_dict, C_dict, S_dict, g_dict, sig_dict, A_dict = [dict() for _ in range(6)]
    YrA = compute_trace(Y, A_sub, b_spatial, C_sub, f_spatial).persist()
    for cur_p, cur_sprs, cur_add, cur_noise in itt.product(p_ls, sprs_ls, add_ls, noise_ls):
        ks = (cur_p, cur_sprs, cur_add, cur_noise)
        print("p:{}, sparse penalty:{}, additional lag:{}, noise frequency:{}"
          .format(cur_p, cur_sprs, cur_add, cur_noise))
        YrA, cur_C, cur_S, cur_B, cur_C0, cur_sig, cur_g, cur_scal = update_temporal(
            Y, A_sub, b_spatial, C_sub, f_spatial, sn_spatial, YrA=YrA,
            sparse_penal=cur_sprs, p=cur_p, use_spatial=False, use_smooth=True,
            add_lag = cur_add, noise_freq=cur_noise)
        YA_dict[ks], C_dict[ks], S_dict[ks], g_dict[ks], sig_dict[ks], A_dict[ks] = (
            YrA.compute(), cur_C.compute(), cur_S.compute(), cur_g.compute(), cur_sig.compute(), A_sub.compute())
    hv_res = visualize_temporal_update(
    YA_dict, C_dict, S_dict, g_dict, sig_dict, A_dict,
    kdims=['p', 'sparse penalty', 'additional lag', 'noise frequency'])

[EricThomson]

@cineguerrilha

sorry, i did not read properly the title (but in my case, the script hangs in other instances). I tried now to interrupt it and save:

%%time A_spatial = save_minian(A_spatial.rename('A_spatial'), param_save_minian) b_spatial = save_minian(b_spatial.rename('b_spatial'), param_save_minian) C_spatial = save_minian(C_spatial.rename('C_spatial'), param_save_minian) f_spatial = save_minian(f_spatial.rename('f_spatial'), param_save_minian) sn_spatial = save_minian(sn_spatial.rename('sn_spatial'), **param_save_minian)

but saving also makes the notebook to hang

I have hit similar issues, but I think in general with Jupyter notebooks interrupting is just not very reliable. Often when I interrupt and it seems to work, it really hasn't. I often need to restart the kernel and start from the beginning (or an intermediate save point). After correcting all the spots where holoview and dask were clashing, I was able to run save_minian at a later point (after the second temporal update) and it only took a couple of seconds.

[cjgabrie]

Hi all,

I have been following along with the discussion and have been trying to troubleshoot Minian for a Windows system on the demo dataset. I have added *.compute() to variables as described above, but have now stalled at a new step during the first temporal update, and don't see any variables that could be modified to fix it. Can you advise if there are any needed changes to my code?

Thanks!

[cjgabrie]

(This ended up stalling for 40+ minutes)

[wmau]

Hi everyone,

Sorry again for everyone running into this issue. At this time we are still working on a fix. For now, @cjgabrie , there are C_temporal and S_temporal that are arguments for hv.Image that you can add .compute() to. Try that and please report back if that fixes the issue.

[cjgabrie]

Thanks for the assistance @wmau

I tried adding the .compute() lines to the code and still got the same stall as shown below. Did I modify the code correctly?

Thanks!

[wmau]

@cjgabrie oops sorry try also with 'C_init'.

[ghost]

I have set interactive to false, so I presume it bypasses all holloviews stuff. Nevertheless, the program still hangs, just after the 1st apatial. Here is what I get after 2h waiting (it is just 6 files msCam files).

hv.output(size=output_size) opts_im = dict(plot=dict(height=b_init.sizes['height'], width=b_init.sizes['width'], colorbar=True), style=dict(cmap='Viridis')) opts_cr = dict(plot=dict(height=b_init.sizes['height'], width=b_init.sizes['height'] * 2)) (regrid(hv.Image(b_init, kdims=['width', 'height'])).opts(**opts_im).relabel('Background Spatial Initial')

• datashade(hv.Curve(f_init, kdims=['frame'])).opts(**opts_cr).relabel('Background Temporal Initial')
• regrid(hv.Image(b_spatial, kdims=['width', 'height'])).opts(**opts_im).relabel('Background Spatial First Update')
• datashade(hv.Curve(f_spatial, kdims=['frame'])).opts(**opts_cr).relabel('Background Temporal First Update') ).cols(2)

here is where the interruption happen

---

KeyboardInterrupt Traceback (most recent call last)

in 2 opts_im = dict(plot=dict(height=b_init.sizes['height'], width=b_init.sizes['width'], colorbar=True), style=dict(cmap='Viridis')) 3 opts_cr = dict(plot=dict(height=b_init.sizes['height'], width=b_init.sizes['height'] * 2)) ----> 4 (regrid(hv.Image(b_init, kdims=['width', 'height'])).opts(**opts_im).relabel('Background Spatial Initial') 5 + datashade(hv.Curve(f_init, kdims=['frame'])).opts(**opts_cr).relabel('Background Temporal Initial') 6 + regrid(hv.Image(b_spatial, kdims=['width', 'height'])).opts(**opts_im).relabel('Background Spatial First Update') ~\.conda\envs\minian\lib\site-packages\IPython\core\displayhook.py in __call__(self, result) 260 self.start_displayhook() 261 self.write_output_prompt() --> 262 format_dict, md_dict = self.compute_format_data(result) 263 self.update_user_ns(result) 264 self.fill_exec_result(result) ~\.conda\envs\minian\lib\site-packages\IPython\core\displayhook.py in compute_format_data(self, result) 149 150 """ --> 151 return self.shell.display_formatter.format(result) 152 153 # This can be set to True by the write_output_prompt method in a subclass ~\.conda\envs\minian\lib\site-packages\IPython\core\formatters.py in format(self, obj, include, exclude) 148 return {}, {} 149 --> 150 format_dict, md_dict = self.mimebundle_formatter(obj, include=include, exclude=exclude) 151 152 if format_dict or md_dict: in __call__(self, obj, include, exclude) ~\.conda\envs\minian\lib\site-packages\IPython\core\formatters.py in catch_format_error(method, self, *args, **kwargs) 222 """show traceback on failed format call""" 223 try: --> 224 r = method(self, *args, **kwargs) 225 except NotImplementedError: 226 # don't warn on NotImplementedErrors ~\.conda\envs\minian\lib\site-packages\IPython\core\formatters.py in __call__(self, obj, include, exclude) 968 969 if method is not None: --> 970 return method(include=include, exclude=exclude) 971 return None 972 else: ~\.conda\envs\minian\lib\site-packages\holoviews\core\dimension.py in _repr_mimebundle_(self, include, exclude) 1292 combined and returned. 1293 """ -> 1294 return Store.render(self) 1295 1296 ~\.conda\envs\minian\lib\site-packages\holoviews\core\options.py in render(cls, obj) 1366 data, metadata = {}, {} 1367 for hook in hooks: -> 1368 ret = hook(obj) 1369 if ret is None: 1370 continue ~\.conda\envs\minian\lib\site-packages\holoviews\ipython\display_hooks.py in pprint_display(obj) 279 if not ip.display_formatter.formatters['text/plain'].pprint: 280 return None --> 281 return display(obj, raw_output=True) 282 283 ~\.conda\envs\minian\lib\site-packages\holoviews\ipython\display_hooks.py in display(obj, raw_output, **kwargs) 252 elif isinstance(obj, (Layout, NdLayout, AdjointLayout)): 253 with option_state(obj): --> 254 output = layout_display(obj) 255 elif isinstance(obj, (HoloMap, DynamicMap)): 256 with option_state(obj): ~\.conda\envs\minian\lib\site-packages\holoviews\ipython\display_hooks.py in wrapped(element) 144 try: 145 max_frames = OutputSettings.options['max_frames'] --> 146 mimebundle = fn(element, max_frames=max_frames) 147 if mimebundle is None: 148 return {}, {} ~\.conda\envs\minian\lib\site-packages\holoviews\ipython\display_hooks.py in layout_display(layout, max_frames) 217 return None 218 --> 219 return render(layout) 220 221 ~\.conda\envs\minian\lib\site-packages\holoviews\ipython\display_hooks.py in render(obj, **kwargs) 66 renderer = renderer.instance(fig='png') 67 ---> 68 return renderer.components(obj, **kwargs) 69 70 ~\.conda\envs\minian\lib\site-packages\holoviews\plotting\renderer.py in components(self, obj, fmt, comm, **kwargs) 319 plot = obj 320 else: --> 321 plot, fmt = self._validate(obj, fmt) 322 323 data, metadata = {}, {} ~\.conda\envs\minian\lib\site-packages\holoviews\plotting\renderer.py in _validate(self, obj, fmt, **kwargs) 218 if isinstance(obj, tuple(self.widgets.values())): 219 return obj, 'html' --> 220 plot = self.get_plot(obj, renderer=self, **kwargs) 221 222 fig_formats = self.mode_formats['fig'][self.mode] ~\.conda\envs\minian\lib\site-packages\holoviews\plotting\renderer.py in get_plot(self_or_cls, obj, renderer, **kwargs) 205 init_key = tuple(v if d is None else d for v, d in 206 zip(plot.keys[0], defaults)) --> 207 plot.update(init_key) 208 else: 209 plot = obj ~\.conda\envs\minian\lib\site-packages\holoviews\plotting\mpl\plot.py in update(self, key) 247 def update(self, key): 248 if len(self) == 1 and ((key == 0) or (key == self.keys[0])) and not self.drawn: --> 249 return self.initialize_plot() 250 return self.__getitem__(key) 251 ~\.conda\envs\minian\lib\site-packages\holoviews\plotting\mpl\plot.py in wrapper(self, *args, **kwargs) 52 def wrapper(self, *args, **kwargs): 53 with _rc_context(self.fig_rcparams): ---> 54 return f(self, *args, **kwargs) 55 return wrapper 56 ~\.conda\envs\minian\lib\site-packages\holoviews\plotting\mpl\plot.py in initialize_plot(self) 1097 def initialize_plot(self): 1098 key = self.keys[-1] -> 1099 ranges = self.compute_ranges(self.layout, key, None) 1100 for subplot in self.subplots.values(): 1101 subplot.initialize_plot(ranges=ranges) ~\.conda\envs\minian\lib\site-packages\holoviews\plotting\plot.py in compute_ranges(self, obj, key, ranges) 392 if (not (axiswise and not isinstance(obj, HoloMap)) or 393 (not framewise and isinstance(obj, HoloMap))): --> 394 self._compute_group_range(group, elements, ranges) 395 self.ranges.update(ranges) 396 return ranges ~\.conda\envs\minian\lib\site-packages\holoviews\plotting\plot.py in _compute_group_range(cls, group, elements, ranges) 503 data_range = el.nodes.range(2, dimension_range=False) 504 else: --> 505 data_range = el.range(el_dim, dimension_range=False) 506 507 if el_dim.name not in group_ranges: ~\.conda\envs\minian\lib\site-packages\holoviews\element\raster.py in range(self, dim, data_range, dimension_range) 594 return (low-halfd, high+halfd) 595 else: --> 596 return super(Image, self).range(dim, data_range, dimension_range) 597 598 ~\.conda\envs\minian\lib\site-packages\holoviews\core\data\__init__.py in range(self, dim, data_range, dimension_range) 287 return dim.range 288 elif dim in self.dimensions() and data_range and bool(self): --> 289 lower, upper = self.interface.range(self, dim) 290 else: 291 lower, upper = (np.NaN, np.NaN) ~\.conda\envs\minian\lib\site-packages\holoviews\core\data\xarray.py in range(cls, dataset, dimension) 219 da = dask_array_module() 220 if da and isinstance(dmin, da.Array): --> 221 dmin, dmax = da.compute(dmin, dmax) 222 if isinstance(dmin, np.ndarray) and dmin.shape == (): 223 dmin = dmin[()] ~\.conda\envs\minian\lib\site-packages\dask\base.py in compute(*args, **kwargs) 435 keys = [x.__dask_keys__() for x in collections] 436 postcomputes = [x.__dask_postcompute__() for x in collections] --> 437 results = schedule(dsk, keys, **kwargs) 438 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)]) 439 ~\.conda\envs\minian\lib\site-packages\dask\threaded.py in get(dsk, result, cache, num_workers, pool, **kwargs) 74 pools[thread][num_workers] = pool 75 ---> 76 results = get_async( 77 pool.apply_async, 78 len(pool._pool), ~\.conda\envs\minian\lib\site-packages\dask\local.py in get_async(apply_async, num_workers, dsk, result, cache, get_id, rerun_exceptions_locally, pack_exception, raise_exception, callbacks, dumps, loads, **kwargs) 473 # Main loop, wait on tasks to finish, insert new ones 474 while state["waiting"] or state["ready"] or state["running"]: --> 475 key, res_info, failed = queue_get(queue) 476 if failed: 477 exc, tb = loads(res_info) ~\.conda\envs\minian\lib\site-packages\dask\local.py in queue_get(q) 123 while True: 124 try: --> 125 return q.get(block=True, timeout=0.1) 126 except Empty: 127 pass ~\.conda\envs\minian\lib\queue.py in get(self, block, timeout) 177 if remaining <= 0.0: 178 raise Empty --> 179 self.not_empty.wait(remaining) 180 item = self._get() 181 self.not_full.notify() ~\.conda\envs\minian\lib\threading.py in wait(self, timeout) 304 else: 305 if timeout > 0: --> 306 gotit = waiter.acquire(True, timeout) 307 else: 308 gotit = waiter.acquire(False) KeyboardInterrupt: i like this program so much and i need to say that it all happen after I upgrade some python libs. The version i installed in nov 2019 was flawless

[ghost]

after that, i cannot run the rest of the script. a.spatial.sel does not run (no error, just it never finishes). It outputs this but i aint shure it is important:

C:\Users\DLC_Leao.conda\envs\minian\lib\site-packages\dask\array\routines.py:264: PerformanceWarning: Increasing number of chunks by factor of 16 intermediate = blockwise( C:\Users\DLC_Leao.conda\envs\minian\lib\site-packages\dask\array\routines.py:264: PerformanceWarning: Increasing number of chunks by factor of 16 intermediate = blockwise(

[eforbes24]

@ForoughHa are you still offering the notebook with the added .compute() changes? If so could you please send it to eforbes24@gmail.com, I'm having trouble piecing all the changes together and would appreciate the help. Thanks!

[Suntail]

@ForoughHa could you be so kind to share the notebook with the added .compute() changes? I've spent several days editing initial notebook, but it still does not work. Please send your version to marina.zots@gmail.com. Thanks a lot!

[ForoughHa]

@Suntail @eforbes24 , I just shared the notebook with both of you. Hope this helps :)

[kacmssm]

Hey @ForoughHa could you please share your notebook with edits with me as well? I would like to try. My address is kirstie.cummings at Gmail.com

[wmau]

Hi all, this should be fixed with a recent patch: https://github.com/DeniseCaiLab/minian/commit/086ed66993b5b34c18d7efc046296a2a25454cbb. Please pull from the repository and if you still have this issue, we'll reopen.