Batch error unsuccessful

[zhouyi0812]

Hello!

I wonder after I run cnmf-e but wasn't able to continue evaluation because of this error, but I am not sure I understand to fix it. Thank you!!

BatchItemUnsuccessfulError                Traceback (most recent call last)
Cell In[65], line 7
      4 cnmf_movie = df.iloc[index].caiman.get_input_movie()
      6 # we can get the contours of the spatial components
----> 7 contours, coms = df.iloc[index].cnmf.get_contours("all", swap_dim=False)
      9 # and temporal components
     10 temporal = df.iloc[index].cnmf.get_temporal("all")

File C:\Users\Public\anaconda3\envs\mescore\lib\site-packages\mesmerize_core\caiman_extensions\_utils.py:24, in validate.<locals>.dec.<locals>.wrapper(self, *args, **kwargs)
     22 if not self._series["outputs"]["success"]:
     23     tb = self._series["outputs"]["traceback"]
---> 24     raise BatchItemUnsuccessfulError(f"Batch item was unsuccessful, traceback from subprocess:\n{tb}")
     25 return func(self, *args, **kwargs)

BatchItemUnsuccessfulError: Batch item was unsuccessful, traceback from subprocess:
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "C:\Users\Public\anaconda3\envs\mescore\lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "C:\Users\Public\anaconda3\envs\mescore\lib\multiprocessing\pool.py", line 48, in mapstar
    return list(map(*args))
  File "C:\Users\Public\anaconda3\envs\mescore\lib\site-packages\caiman\source_extraction\cnmf\map_reduce.py", line 130, in cnmf_patches
    cnm = cnm.fit(images)
  File "C:\Users\Public\anaconda3\envs\mescore\lib\site-packages\caiman\source_extraction\cnmf\cnmf.py", line 500, in fit
    self.initialize(Y)
  File "C:\Users\Public\anaconda3\envs\mescore\lib\site-packages\caiman\source_extraction\cnmf\cnmf.py", line 959, in initialize
    extra_1p = initialize_components(
  File "C:\Users\Public\anaconda3\envs\mescore\lib\site-packages\caiman\source_extraction\cnmf\initialization.py", line 353, in initialize_components
    Ain, Cin, _, b_in, f_in, extra_1p = greedyROI_corr(
  File "C:\Users\Public\anaconda3\envs\mescore\lib\site-packages\caiman\source_extraction\cnmf\initialization.py", line 1300, in greedyROI_corr
    A, C = caiman.source_extraction.cnmf.merging.merge_components(
  File "C:\Users\Public\anaconda3\envs\mescore\lib\site-packages\caiman\source_extraction\cnmf\merging.py", line 164, in merge_components
    FF1 = (C_corr + C_corr.T) > thr
  File "C:\Users\Public\anaconda3\envs\mescore\lib\site-packages\scipy\sparse\_compressed.py", line 325, in __gt__
    return self._inequality(other, operator.gt, '_gt_',
  File "C:\Users\Public\anaconda3\envs\mescore\lib\site-packages\scipy\sparse\_compressed.py", line 316, in _inequality
    raise ValueError("Operands could not be compared.")
ValueError: Operands could not be compared.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\Public\anaconda3\envs\mescore\lib\site-packages\mesmerize_core\algorithms\cnmfe.py", line 89, in run_algo
    cnm = cnm.fit(images)
  File "C:\Users\Public\anaconda3\envs\mescore\lib\site-packages\caiman\source_extraction\cnmf\cnmf.py", line 594, in fit
    self.estimates.sn, self.estimates.optional_outputs = run_CNMF_patches(
  File "C:\Users\Public\anaconda3\envs\mescore\lib\site-packages\caiman\source_extraction\cnmf\map_reduce.py", line 250, in run_CNMF_patches
    file_res = dview.map_async(cnmf_patches, args_in).get(4294967)
  File "C:\Users\Public\anaconda3\envs\mescore\lib\multiprocessing\pool.py", line 774, in get
    raise self._value
ValueError: Operands could not be compared.

[kushalkolar]

This is from inside the algorithm so it's probably your params, I would post a screenshot of your imaging data with the dimensions and the parameters you used.

[zhouyi0812]

params_cnmfe =\ { "main": { 'method_init': 'corr_pnr', # use this for 1 photon 'K': None, 'gSig': (gSig, gSig), 'gSiz': (4 gSig + 1, 4 gSig + 1), 'merge_thr': 0.7, 'p': 1, 'tsub': 2, 'ssub': 1, 'rf': 40, 'stride': 20, 'only_init': True, # set it to True to run CNMF-E 'nb': 0, 'nb_patch': 0, 'method_deconvolution': 'oasis', # could use 'cvxpy' alternatively 'low_rank_background': None, 'update_background_components': True, # sometimes setting to False improve the results 'normalize_init': False, # just leave as is 'center_psf': True, # leave as is for 1 photon 'ssub_B': 2, 'ring_size_factor': 1.4, 'del_duplicates': True, # whether to remove duplicates from initialization **corr_pnr # unpack corr_pnr vals into here } }

variants of a single parameter merge_thr_variants = [0.75, 0.8, 0.85]

always use deepcopy like before new_params_cnmf = deepcopy(params_cnmfe)

for merge_thr in merge_thr_variants: new_params_cnmf = deepcopy(new_params_cnmf)

new_params_cnmf["main"]["merge_thr"] = merge_thr

df.caiman.add_item(
    algo="cnmfe",
    item_name=df.iloc[0]["item_name"],
    input_movie_path=df.iloc[0]["input_movie_path"],  # Make sure this is the correct key for the movie path
    params=new_params_cnmf
)

Here is my parameters, and it took me 2days to run this which is much longer than directly with caiman Thank you!!

[kushalkolar]

What does your movie look like, can you post the correlation image and mean projection?

Mesmerize-core passes params directly to caiman, if the processing time is different the params you used must've been different. My guess is you were subsampling when using Caiman directly.

On Mon, Nov 6, 2023, 15:51 Yi Zhou @.***> wrote:

params_cnmfe = { "main": { 'method_init': 'corr_pnr', # use this for 1 photon 'K': None, 'gSig': (gSig, gSig), 'gSiz': (4 gSig + 1, 4 gSig + 1), 'merge_thr': 0.7, 'p': 1, 'tsub': 2, 'ssub': 1, 'rf': 40, 'stride': 20, 'only_init': True, # set it to True to run CNMF-E 'nb': 0, 'nb_patch': 0, 'method_deconvolution': 'oasis', # could use 'cvxpy' alternatively 'low_rank_background': None, 'update_background_components': True, # sometimes setting to False improve the results 'normalize_init': False, # just leave as is 'center_psf': True, # leave as is for 1 photon 'ssub_B': 2, 'ring_size_factor': 1.4, 'del_duplicates': True, # whether to remove duplicates from initialization **corr_pnr # unpack corr_pnr vals into here } }

variants of a single parameter merge_thr_variants = [0.75, 0.8, 0.85]

always use deepcopy like before new_params_cnmf = deepcopy(params_cnmfe)

for merge_thr in merge_thr_variants: new_params_cnmf = deepcopy(new_params_cnmf)

new_params_cnmf["main"]["merge_thr"] = merge_thr

df.caiman.add_item( algo="cnmfe", item_name=df.iloc[0]["item_name"], input_movie_path=df.iloc[0]["input_movie_path"], # Make sure this is the correct key for the movie path params=new_params_cnmf )

Here is my parameters, and it took me 2days to run this which is much longer than directly with caiman Thank you!!

— Reply to this email directly, view it on GitHub https://github.com/nel-lab/mesmerize-core/issues/244#issuecomment-1796413347, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACHXXRD2ZCTCTRVJVSQELMTYDFETRAVCNFSM6AAAAAA634EZNOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJWGQYTGMZUG4 . You are receiving this because you commented.Message ID: @.***>

[zhouyi0812]

This is my mean projection and correlation image, thank you very much for your guidance!!

https://github.com/nel-lab/mesmerize-core/assets/48576330/b9d292f2-01f0-494b-941a-45372f228be5

This is my parameter in Caiman, but it looks like there isn't much different...

p = 1
K = None
gSig = (7, 7) gaussian width of a 2D gaussian kernel, which approximates a neuron gSiz = (29, 29) average diameter of a neuron, in general 4gSig+1 Ain = None possibility to seed with predetermined binary masks merge_thr = .75 merging threshold, max correlation allowed rf = 30 half-size of the patches in pixels. e.g., if rf=40, patches are 80x80 stride_cnmf = 15 amount of overlap between the patches in pixels tsub = 2 downsampling factor in time for initialization. Increase if you have memory problems ssub = 1 downsampling factor in space for initialization low_rank_background = None # None leaves background of each patch intact, True performs global low-rank approximation if gnb>0 gnb = 0 number of background components (rank) if positive, nb_patch = 0 number of background components (rank) per patch if gnb>0, else it is set automatically min_corr = .8 min peak value from correlation image min_pnr = 10 min peak to noise ration from PNR image ssub_B = 1 additional downsampling factor in space for background ring_size_factor = 1.4 radius of ring is gSizring_size_factor bord_px = 0 bas_nonneg=True shift the raw trace up to avoid negative

opts.change_params(params_dict={'method_init': 'corr_pnr', use this for 1 photon 'K': K, 'gSig': gSig, 'gSiz': gSiz, 'merge_thr': merge_thr, 'p': p, 'tsub': tsub, 'ssub': ssub, 'rf': rf, 'stride': stride_cnmf, 'only_init': True, set it to True to run CNMF-E 'nb': gnb, 'nb_patch': nb_patch, 'method_deconvolution': 'oasis', could use 'cvxpy' alternatively 'low_rank_background': low_rank_background, 'update_background_components': True, sometimes setting to False improve the results 'min_corr': min_corr, 'min_pnr': min_pnr, 'normalize_init': False, just leave as is 'center_psf': True, leave as is for 1 photon 'ssub_B': ssub_B, 'ring_size_factor': ring_size_factor, 'del_duplicates': True, whether to remove duplicates from initialization 'bas_nonneg': True,
'border_pix': bord_px })

[kushalkolar]

We're pretty busy prepping for the workshop this week, @EricThomson and I will get back to you mid-next week, don't hesitate to remind us!

[zhouyi0812]

Sure! Thank you and I will remind you! And here is the df after cnmfe

[kushalkolar]

That activity looks really weird, are frames duplicated?

@EricThomson

[zhouyi0812]

No, it is not duplicated, I also tried my other data, and it shows a similar situation.

[zhouyi0812]

Hello! Just want to follow up with the issue a bit! I also tried my other file, it is the same situation. Thank you!

[kushalkolar]

Can you send the movie that's causing this? How large is it?

[zhouyi0812]

Sure! It is 911mb. Thank you very much!! And here is the link: https://drive.google.com/file/d/1MeJIDnFDdgRO3ahbYiDHSS-Fkphk6r-U/view?usp=sharing

And here is my parameters:

params_cnmfe =\ { "main": { 'method_init': 'corr_pnr', # use this for 1 photon 'K': None, 'gSig': (gSig, gSig), 'gSiz': (4 gSig + 1, 4 gSig + 1), 'merge_thr': 0.7, 'p': 1, 'tsub': 2, 'ssub': 1, 'rf': 40, 'stride': 20, 'only_init': True, # set it to True to run CNMF-E 'nb': 0, 'nb_patch': 0, 'method_deconvolution': 'oasis', # could use 'cvxpy' alternatively 'low_rank_background': None, 'update_background_components': True, # sometimes setting to False improve the results 'normalize_init': False, # just leave as is 'center_psf': True, # leave as is for 1 photon 'ssub_B': 2, 'ring_size_factor': 1.4, 'del_duplicates': True, # whether to remove duplicates from initialization 'bas_nonneg':True, **corr_pnr # unpack corr_pnr vals into here } }

from itertools import product

merge_thr_variants = [0.75,0.8,0.85]

new_params_cnmf = deepcopy(params_cnmfe)

for merge_thr in merge_thr_variants: new_params_cnmf = deepcopy(new_params_cnmf)

new_params_cnmf["main"]["merge_thr"] = merge_thr

add param combination variant to batch
df.caiman.add_item(
    algo="cnmfe",
    item_name=df.iloc[0]["item_name"],
    input_movie_path=df.iloc[0],
    params=new_params_cnmf
)

[kushalkolar]

What is your gSig? Also your cells seem to be moving out of plane over the course of the movie which is a problem.

[zhouyi0812]

gSig is 7

I saw that cells were moving....and when I use caiman it will identify the whole trajectory as an ROI, so I wonder maybe I could filter out the cells that are too big?

Thank you!

[zhouyi0812]

Hello! I tried the demo data for cnmfe. It is the same error, and I am not sure how to fix it. Thank you!

[kushalkolar]

You can print the traceback using df.iloc[index].cnmf.get_output()

[zhouyi0812]

it has an error, and do you think it is because of the package problem? Thank you!

BatchItemUnsuccessfulError Traceback (most recent call last) Cell In[42], line 2 1 index = 5 ----> 2 df.iloc[index].cnmf.get_output()

File ~\anaconda3\envs\mescore\lib\site-packages\mesmerize_core\caiman_extensions_utils.py:24, in validate..dec..wrapper(self, *args, *kwargs) 22 if not self._series["outputs"]["success"]: 23 tb = self._series["outputs"]["traceback"] ---> 24 raise BatchItemUnsuccessfulError(f"Batch item was unsuccessful, traceback from subprocess:\n{tb}") 25 return func(self, args, **kwargs)

BatchItemUnsuccessfulError: Batch item was unsuccessful, traceback from subprocess: Traceback (most recent call last): File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\mesmerize_core\algorithms\cnmfe.py", line 97, in run_algo d["cnmf-hdf5-path"] = cnmf_hdf5_path.relative_to(output_dir.parent) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\pathlib.py", line 818, in relative_to raise ValueError("{!r} is not in the subpath of {!r}" ValueError: 'C:\Users\yyizhou\caiman_data\mesmerize-cnmfe\6971d600-8570-45ff-bd9e-450ca6090fb0\6971d600-8570-45ff-bd9e-450ca6090fb0.hdf5' is not in the subpath of '\Users\yyizhou\caiman_data\mesmerize-cnmfe' OR one path is relative and the other is absolute.

[kushalkolar]

You probably forgot "C:" in your parent raw data path

On Thu, Nov 30, 2023, 03:36 Yi Zhou @.***> wrote:

it has an error, and do you think it is because of the package problem? Thank you!

BatchItemUnsuccessfulError Traceback (most recent call last) Cell In[42], line 2 1 index = 5 ----> 2 df.iloc[index].cnmf.get_output()

File ~\anaconda3\envs\mescore\lib\site-packages\mesmerize_core\caiman_extensions_utils.py:24, in validate..dec..wrapper(self, *args, *kwargs) 22 if not self._series["outputs"]["success"]: 23 tb = self._series["outputs"]["traceback"] ---> 24 raise BatchItemUnsuccessfulError(f"Batch item was unsuccessful, traceback from subprocess:\n{tb}") 25 return func(self, args, **kwargs)

BatchItemUnsuccessfulError: Batch item was unsuccessful, traceback from subprocess: Traceback (most recent call last): File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\mesmerize_core\algorithms\cnmfe.py", line 97, in run_algo d["cnmf-hdf5-path"] = cnmf_hdf5_path.relative_to(output_dir.parent) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\pathlib.py", line 818, in relative_to raise ValueError("{!r} is not in the subpath of {!r}" ValueError: 'C:\Users\yyizhou\caiman_data\mesmerize-cnmfe\6971d600-8570-45ff-bd9e-450ca6090fb0\6971d600-8570-45ff-bd9e-450ca6090fb0.hdf5' is not in the subpath of '\Users\yyizhou\caiman_data\mesmerize-cnmfe' OR one path is relative and the other is absolute.

— Reply to this email directly, view it on GitHub https://github.com/nel-lab/mesmerize-core/issues/244#issuecomment-1833319112, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACHXXRHIQPAX6SYKBAAFHSTYHBAQXAVCNFSM6AAAAAA634EZNOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZTGMYTSMJRGI . You are receiving this because you commented.Message ID: @.***>

[zhouyi0812]

I see! Thank you so much and I successfully run the cnmfe!! However, the dimension of the data looks like this: <caiman.source_extraction.cnmf.cnmf.CNMF at 0x19d38993eb0>

viz_cnmf = df.cnmf.viz( image_data_options=["input", "rcm"], # cnmfe does not support rcb and residuals yet ) ValueError: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 140 and the array at index 1 has size 141

[kushalkolar]

this is a fastplotlib bug that was fixed recently, I just did a new release, should be out on pypi in a few minutes.

[zhouyi0812]

I just update the fastplotlib and rerun the cnmfe, but still have the same error. Or should I do other step? Thanks! Successfully uninstalled fastplotlib-0.1.0a15 Successfully installed fastplotlib-0.1.0a16

[kushalkolar]

You don't have to rerun cnmfe, just the viz.

That's weird, can you do the viz with only input and then only rcm to see which one is causing the issue?

[zhouyi0812]

so for rcm viz_cnmf = df.cnmf.viz( image_data_options=["rcm"], # cnmfe does not support rcb and residuals yet ) ValueError: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 140 and the array at index 1 has size 141

for input, there are some warning message viz_cnmf = df.cnmf.viz( image_data_options=["input"], # cnmfe does not support rcb and residuals yet )

it has some warning messaging C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\mesmerize_core\caiman_extensions\cnmf.py:309: RuntimeWarning: Mean of empty slice. com = coors.mean(axis=0) C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\numpy\core_methods.py:121: RuntimeWarning: invalid value encountered in divide ret = um.true_divide( C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\fastplotlib\graphics_features_base.py:34: UserWarning: converting float64 array to float32 warn(f"converting {array.dtype} array to float32") C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_client\session.py:721: UserWarning: Message serialization failed with: Out of range float values are not JSON compliant Supporting this message is deprecated in jupyter-client 7, please make sure your message is JSON-compliant content = self.pack(content)

and when I try to visualize input, the kernal die for twice

[kushalkolar]

If your kernel is dieing that's either insufficient hardware or bad drivers. What gpu do you have?

On Thu, Nov 30, 2023, 16:04 Yi Zhou @.***> wrote:

so for rcm viz_cnmf = df.cnmf.viz( image_data_options=["rcm"], # cnmfe does not support rcb and residuals yet ) ValueError: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 140 and the array at index 1 has size 141

for input, there are some warning message viz_cnmf = df.cnmf.viz( image_data_options=["input"], # cnmfe does not support rcb and residuals yet )

it has some warning messaging C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\mesmerize_core\caiman_extensions\cnmf.py:309: RuntimeWarning: Mean of empty slice. com = coors.mean(axis=0) C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\numpy\core_methods.py:121: RuntimeWarning: invalid value encountered in divide ret = um.true_divide( C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\fastplotlib\graphics_features_base.py:34: UserWarning: converting float64 array to float32 warn(f"converting {array.dtype} array to float32") C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_client\session.py:721: UserWarning: Message serialization failed with: Out of range float values are not JSON compliant Supporting this message is deprecated in jupyter-client 7, please make sure your message is JSON-compliant content = self.pack(content)

and when I try to visualize input, the kernal die for twice

— Reply to this email directly, view it on GitHub https://github.com/nel-lab/mesmerize-core/issues/244#issuecomment-1834556964, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACHXXRH3HYIGMTBEXA7SNM3YHDYEDAVCNFSM6AAAAAA634EZNOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZUGU2TMOJWGQ . You are receiving this because you commented.Message ID: @.***>

[zhouyi0812]

I think it is 63GB

[kushalkolar]

Are you able to run this example and pan-zoom etc? https://pygfx.readthedocs.io/en/stable/_gallery/other/collection_line.html#sphx-glr-gallery-other-collection-line-py

[zhouyi0812]

yes! this is the result and there is a graph

EGL surface: -srgb

No config found! EGL says it can present to the window but not natively 30000000 vertices in total Time to first draw: 8.37

[kushalkolar]

you can see the lines? What's the VRAM usage? and how many components do you have in your cnmfe results?

[zhouyi0812]

https://github.com/nel-lab/mesmerize-core/assets/48576330/192b0570-deb8-4ffc-847b-d8fcf76db519

Here is the result with a video! So far i am using the demo video, not my data. I checked dedicated GPU memory it doesn't change when I run the code you provided.

[kushalkolar]

Is it using the Intel GPU? I would find it strange that the intel GPU has access to 64GB of VRAM. When the kernel crashes do you get something like this in the terminal that jupyterlab is running in? Or in the jupyterlab console log.

amdgpu: Not enough memory for command submission.
thread '<unnamed>' panicked at src/lib.rs:495:5:
Error in wgpuQueueSubmit: Validation Error

Caused by:
    Parent device is lost

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
Aborted

[zhouyi0812]

The info from terminal here:

thread '' panicked at src\lib.rs:1670:38: invalid size note: run with RUST_BACKTRACE=1 environment variable to display a backtrace [I 2023-11-30 14:05:56.290 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), keep random ports [W 2023-11-30 14:05:56.290 ServerApp] kernel 9de01b4f-799d-45b4-b670-d45fabbc887b restarted [I 2023-11-30 14:05:56.299 ServerApp] Starting buffering for 9de01b4f-799d-45b4-b670-d45fabbc887b:07c0c334-58a6-4021-a1b2-f8d0045c70fb [I 2023-11-30 14:05:56.326 ServerApp] Connecting to kernel 9de01b4f-799d-45b4-b670-d45fabbc887b. [I 2023-11-30 14:05:56.326 ServerApp] Restoring connection for 9de01b4f-799d-45b4-b670-d45fabbc887b:07c0c334-58a6-4021-a1b2-f8d0045c70fb [W 2023-11-30 14:05:57.390 ServerApp] Got events for closed stream <zmq.eventloop.zmqstream.ZMQStream object at 0x00000218095B06A0> [W 2023-11-30 14:05:57.390 ServerApp] Got events for closed stream <zmq.eventloop.zmqstream.ZMQStream object at 0x00000218095B31F0>

[kushalkolar]

can you run jupyter lab with RUST_BACKTRACE=1? Looks like it's something else.

[zhouyi0812]

it is still the same kernal death. And there is nothing happen after run RUST_BACKTRACE=1

[kushalkolar]

yes but that env variable should then make it print more stuff to the terminal jupyter is running in

[zhouyi0812]

i see what you mean. I tried this: RUST_BACKTRACE=1 viz_cnmf.show(sidecar=True)

and terminal looks nothing different

thread '' panicked at src\lib.rs:1670:38: invalid size note: run with RUST_BACKTRACE=1 environment variable to display a backtrace [I 2023-11-30 14:17:08.393 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), keep random ports [W 2023-11-30 14:17:08.394 ServerApp] kernel 9de01b4f-799d-45b4-b670-d45fabbc887b restarted [I 2023-11-30 14:17:08.400 ServerApp] Starting buffering for 9de01b4f-799d-45b4-b670-d45fabbc887b:07c0c334-58a6-4021-a1b2-f8d0045c70fb [I 2023-11-30 14:17:08.413 ServerApp] Connecting to kernel 9de01b4f-799d-45b4-b670-d45fabbc887b. [I 2023-11-30 14:17:08.413 ServerApp] Restoring connection for 9de01b4f-799d-45b4-b670-d45fabbc887b:07c0c334-58a6-4021-a1b2-f8d0045c70fb [I 2023-11-30 14:17:12.423 ServerApp] Saving file at /cnmfe_demo.ipynb

[kushalkolar]

You need to set that environment variable in the terminal and then run jupyter lab in that terminal

[zhouyi0812]

ok~ so i run this in terminal, the result like this (mescore) C:\Users\yyizhou>RUST_BACKTRACE=1 'RUST_BACKTRACE' is not recognized as an internal or external command, operable program or batch file.

[kushalkolar]

I think on windows this is how you set env variables:

set RUST_BACKTRACE=1

and then run jupyter lab

[zhouyi0812]

so rust_backtrace=1 is this and I run =full thread '' panicked at src\lib.rs:1670:38: invalid size stack backtrace: note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace. [I 2023-11-30 14:24:45.177 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), keep random ports [W 2023-11-30 14:24:45.177 ServerApp] kernel 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d restarted [I 2023-11-30 14:24:45.179 ServerApp] Starting buffering for 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d:36a619de-16c9-4b33-bad7-d70cf80dd4e3 [I 2023-11-30 14:24:45.190 ServerApp] Connecting to kernel 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d. [I 2023-11-30 14:24:45.190 ServerApp] Restoring connection for 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d:36a619de-16c9-4b33-bad7-d70cf80dd4e3 [W 2023-11-30 14:24:56.711 ServerApp] 404 GET /api/kernels/e2c017b7-5c3c-4bbb-b7e6-3628c7bc59a0/channels?session_id=599f92e0-7720-4cae-8700-2bc5ab2d84ed (::1): Kernel does not exist: e2c017b7-5c3c-4bbb-b7e6-3628c7bc59a0 [W 2023-11-30 14:24:56.714 ServerApp] 404 GET /api/kernels/e2c017b7-5c3c-4bbb-b7e6-3628c7bc59a0/channels?session_id=599f92e0-7720-4cae-8700-2bc5ab2d84ed (3e46ca4e87ed40ca93a4822f738293e5@::1) 4.68ms referer=None [W 2023-11-30 14:24:56.719 ServerApp] 404 GET /api/kernels/e2c017b7-5c3c-4bbb-b7e6-3628c7bc59a0?1701383096715 (::1): Kernel does not exist: e2c017b7-5c3c-4bbb-b7e6-3628c7bc59a0 [W 2023-11-30 14:24:56.720 ServerApp] wrote error: 'Kernel does not exist: e2c017b7-5c3c-4bbb-b7e6-3628c7bc59a0' Traceback (most recent call last): File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\tornado\web.py", line 1786, in _execute result = await result File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\handlers.py", line 75, in get model = await ensure_async(km.kernel_model(kernel_id)) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\kernelmanager.py", line 500, in kernel_model self._check_kernel_id(kernel_id) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\kernelmanager.py", line 531, in _check_kernel_id raise web.HTTPError(404, "Kernel does not exist: %s" % kernel_id) tornado.web.HTTPError: HTTP 404: Not Found (Kernel does not exist: e2c017b7-5c3c-4bbb-b7e6-3628c7bc59a0) [W 2023-11-30 14:24:56.721 ServerApp] 404 GET /api/kernels/e2c017b7-5c3c-4bbb-b7e6-3628c7bc59a0?1701383096715 (3e46ca4e87ed40ca93a4822f738293e5@::1) 2.00ms referer=http://localhost:8888/lab/tree/cnmfe_demo.ipynb [W 2023-11-30 14:25:02.865 ServerApp] 404 GET /api/kernels/360fe68b-8e14-4ee7-abd3-7e7ab57eee5d/channels?session_id=d12893f2-a90b-4c05-a072-df5d31438f6d (::1): Kernel does not exist: 360fe68b-8e14-4ee7-abd3-7e7ab57eee5d [W 2023-11-30 14:25:02.868 ServerApp] 404 GET /api/kernels/360fe68b-8e14-4ee7-abd3-7e7ab57eee5d/channels?session_id=d12893f2-a90b-4c05-a072-df5d31438f6d (3e46ca4e87ed40ca93a4822f738293e5@::1) 3.37ms referer=None [W 2023-11-30 14:25:02.873 ServerApp] 404 GET /api/kernels/360fe68b-8e14-4ee7-abd3-7e7ab57eee5d?1701383102869 (::1): Kernel does not exist: 360fe68b-8e14-4ee7-abd3-7e7ab57eee5d [W 2023-11-30 14:25:02.873 ServerApp] wrote error: 'Kernel does not exist: 360fe68b-8e14-4ee7-abd3-7e7ab57eee5d' Traceback (most recent call last): File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\tornado\web.py", line 1786, in _execute result = await result File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\handlers.py", line 75, in get model = await ensure_async(km.kernel_model(kernel_id)) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\kernelmanager.py", line 500, in kernel_model self._check_kernel_id(kernel_id) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\kernelmanager.py", line 531, in _check_kernel_id raise web.HTTPError(404, "Kernel does not exist: %s" % kernel_id) tornado.web.HTTPError: HTTP 404: Not Found (Kernel does not exist: 360fe68b-8e14-4ee7-abd3-7e7ab57eee5d) [W 2023-11-30 14:25:02.875 ServerApp] 404 GET /api/kernels/360fe68b-8e14-4ee7-abd3-7e7ab57eee5d?1701383102869 (3e46ca4e87ed40ca93a4822f738293e5@::1) 3.00ms referer=http://localhost:8888/lab/tree/cnmfe_demo.ipynb

run=full

[W 2023-11-30 14:26:46.111 LabApp] Could not determine jupyterlab build status without nodejs [I 2023-11-30 14:26:50.681 ServerApp] Kernel started: 82d8f2be-497a-434b-beb2-c483419f345d [I 2023-11-30 14:26:51.726 ServerApp] Connecting to kernel 82d8f2be-497a-434b-beb2-c483419f345d. [I 2023-11-30 14:26:51.735 ServerApp] Connecting to kernel 82d8f2be-497a-434b-beb2-c483419f345d. [I 2023-11-30 14:26:51.742 ServerApp] Connecting to kernel 82d8f2be-497a-434b-beb2-c483419f345d. [W 2023-11-30 14:26:51.773 ServerApp] Got events for closed stream <zmq.eventloop.zmqstream.ZMQStream object at 0x000001B3C91312A0> [W 2023-11-30 14:26:53.374 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d?1701383213374 (::1): Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d [W 2023-11-30 14:26:53.374 ServerApp] wrote error: 'Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d' Traceback (most recent call last): File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\tornado\web.py", line 1786, in _execute result = await result File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\handlers.py", line 75, in get model = await ensure_async(km.kernel_model(kernel_id)) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\kernelmanager.py", line 500, in kernel_model self._check_kernel_id(kernel_id) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\kernelmanager.py", line 531, in _check_kernel_id raise web.HTTPError(404, "Kernel does not exist: %s" % kernel_id) tornado.web.HTTPError: HTTP 404: Not Found (Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d) [W 2023-11-30 14:26:53.374 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d?1701383213374 (3e46ca4e87ed40ca93a4822f738293e5@::1) 0.00ms referer=http://localhost:8888/lab/workspaces/auto-O/tree/cnmfe_demo.ipynb [W 2023-11-30 14:26:54.384 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d/channels?session_id=36a619de-16c9-4b33-bad7-d70cf80dd4e3 (::1): Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d [W 2023-11-30 14:26:54.400 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d/channels?session_id=36a619de-16c9-4b33-bad7-d70cf80dd4e3 (3e46ca4e87ed40ca93a4822f738293e5@::1) 15.64ms referer=None [W 2023-11-30 14:26:54.415 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d?1701383214410 (::1): Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d [W 2023-11-30 14:26:54.415 ServerApp] wrote error: 'Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d' Traceback (most recent call last): File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\tornado\web.py", line 1786, in _execute result = await result File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\handlers.py", line 75, in get model = await ensure_async(km.kernel_model(kernel_id)) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\kernelmanager.py", line 500, in kernel_model self._check_kernel_id(kernel_id) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\kernelmanager.py", line 531, in _check_kernel_id raise web.HTTPError(404, "Kernel does not exist: %s" % kernel_id) tornado.web.HTTPError: HTTP 404: Not Found (Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d) [W 2023-11-30 14:26:54.431 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d?1701383214410 (3e46ca4e87ed40ca93a4822f738293e5@::1) 15.62ms referer=http://localhost:8888/lab/workspaces/auto-O/tree/cnmfe_demo.ipynb [W 2023-11-30 14:26:57.375 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d/channels?session_id=36a619de-16c9-4b33-bad7-d70cf80dd4e3 (::1): Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d [W 2023-11-30 14:26:57.375 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d/channels?session_id=36a619de-16c9-4b33-bad7-d70cf80dd4e3 (3e46ca4e87ed40ca93a4822f738293e5@::1) 0.00ms referer=None [W 2023-11-30 14:26:57.375 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d?1701383217384 (::1): Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d [W 2023-11-30 14:26:57.375 ServerApp] wrote error: 'Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d' Traceback (most recent call last): File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\tornado\web.py", line 1786, in _execute result = await result File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\handlers.py", line 75, in get model = await ensure_async(km.kernel_model(kernel_id)) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\kernelmanager.py", line 500, in kernel_model self._check_kernel_id(kernel_id) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\kernelmanager.py", line 531, in _check_kernel_id raise web.HTTPError(404, "Kernel does not exist: %s" % kernel_id) tornado.web.HTTPError: HTTP 404: Not Found (Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d) [W 2023-11-30 14:26:57.375 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d?1701383217384 (3e46ca4e87ed40ca93a4822f738293e5@::1) 0.00ms referer=http://localhost:8888/lab/workspaces/auto-O/tree/cnmfe_demo.ipynb [W 2023-11-30 14:27:01.383 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d/channels?session_id=36a619de-16c9-4b33-bad7-d70cf80dd4e3 (::1): Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d [W 2023-11-30 14:27:01.398 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d/channels?session_id=36a619de-16c9-4b33-bad7-d70cf80dd4e3 (3e46ca4e87ed40ca93a4822f738293e5@::1) 15.71ms referer=None [W 2023-11-30 14:27:01.398 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d?1701383221400 (::1): Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d [W 2023-11-30 14:27:01.398 ServerApp] wrote error: 'Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d' Traceback (most recent call last): File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\tornado\web.py", line 1786, in _execute result = await result File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\handlers.py", line 75, in get model = await ensure_async(km.kernel_model(kernel_id)) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\kernelmanager.py", line 500, in kernel_model self._check_kernel_id(kernel_id) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\kernelmanager.py", line 531, in _check_kernel_id raise web.HTTPError(404, "Kernel does not exist: %s" % kernel_id) tornado.web.HTTPError: HTTP 404: Not Found (Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d) [W 2023-11-30 14:27:01.398 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d?1701383221400 (3e46ca4e87ed40ca93a4822f738293e5@::1) 0.00ms referer=http://localhost:8888/lab/workspaces/auto-O/tree/cnmfe_demo.ipynb [W 2023-11-30 14:27:04.383 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d/channels?session_id=36a619de-16c9-4b33-bad7-d70cf80dd4e3 (::1): Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d [W 2023-11-30 14:27:04.383 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d/channels?session_id=36a619de-16c9-4b33-bad7-d70cf80dd4e3 (3e46ca4e87ed40ca93a4822f738293e5@::1) 0.00ms referer=None [W 2023-11-30 14:27:04.383 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d?1701383224387 (::1): Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d [W 2023-11-30 14:27:04.383 ServerApp] wrote error: 'Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d' Traceback (most recent call last): File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\tornado\web.py", line 1786, in _execute result = await result File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\handlers.py", line 75, in get model = await ensure_async(km.kernel_model(kernel_id)) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\kernelmanager.py", line 500, in kernel_model self._check_kernel_id(kernel_id) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\kernelmanager.py", line 531, in _check_kernel_id raise web.HTTPError(404, "Kernel does not exist: %s" % kernel_id) tornado.web.HTTPError: HTTP 404: Not Found (Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d) [W 2023-11-30 14:27:04.383 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d?1701383224387 (3e46ca4e87ed40ca93a4822f738293e5@::1) 0.00ms referer=http://localhost:8888/lab/workspaces/auto-O/tree/cnmfe_demo.ipynb [W 2023-11-30 14:27:07.377 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d/channels?session_id=36a619de-16c9-4b33-bad7-d70cf80dd4e3 (::1): Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d [W 2023-11-30 14:27:07.378 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d/channels?session_id=36a619de-16c9-4b33-bad7-d70cf80dd4e3 (3e46ca4e87ed40ca93a4822f738293e5@::1) 1.01ms referer=None [W 2023-11-30 14:27:07.379 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d?1701383227378 (::1): Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d [W 2023-11-30 14:27:07.379 ServerApp] wrote error: 'Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d' Traceback (most recent call last): File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\tornado\web.py", line 1786, in _execute result = await result File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\handlers.py", line 75, in get model = await ensure_async(km.kernel_model(kernel_id)) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\kernelmanager.py", line 500, in kernel_model self._check_kernel_id(kernel_id) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\kernelmanager.py", line 531, in _check_kernel_id raise web.HTTPError(404, "Kernel does not exist: %s" % kernel_id) tornado.web.HTTPError: HTTP 404: Not Found (Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d) [W 2023-11-30 14:27:07.380 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d?1701383227378 (3e46ca4e87ed40ca93a4822f738293e5@::1) 1.01ms referer=http://localhost:8888/lab/workspaces/auto-O/tree/cnmfe_demo.ipynb thread '' panicked at src\lib.rs:1670:38: invalid size stack backtrace: 0: 0x7ffbb2064cea - wgpu_render_pass_execute_bundles 1: 0x7ffbb207bb1b - wgpu_render_pass_execute_bundles 2: 0x7ffbb20620c1 - wgpu_render_pass_execute_bundles 3: 0x7ffbb2064a6a - wgpu_render_pass_execute_bundles 4: 0x7ffbb2066e1a - wgpu_render_pass_execute_bundles 5: 0x7ffbb2066a88 - wgpu_render_pass_execute_bundles 6: 0x7ffbb20674ce - wgpu_render_pass_execute_bundles 7: 0x7ffbb206737a - wgpu_render_pass_execute_bundles 8: 0x7ffbb20656d9 - wgpu_render_pass_execute_bundles 9: 0x7ffbb20670c0 - wgpu_render_pass_execute_bundles 10: 0x7ffbb20a7f15 - wgpu_render_pass_execute_bundles 11: 0x7ffbb1c84f30 - wgpuDeviceDestroy 12: 0x7ffbb1bf9203 - wgpuDeviceDestroy 13: 0x7ffbb1d25d89 - wgpuDeviceCreateBindGroup 14: 0x7ffbc0ca10f3 - 15: 0x7ffbc0cbac90 - PyInit__cffi_backend 16: 0x7ffbc0ca74af - 17: 0x7ffc3557c2b7 - PyObject_Call 18: 0x7ffc35690602 - PyEval_GetFuncDesc 19: 0x7ffc3568b560 - PyEval_EvalFrameDefault 20: 0x7ffc3568eadf - PyEval_EvalFrameDefault 21: 0x7ffc3557c3be - PyFunction_Vectorcall 22: 0x7ffc35687e91 - PyOS_URandomNonblock 23: 0x7ffc35690402 - PyEval_GetFuncDesc 24: 0x7ffc3568c74d - PyEval_EvalFrameDefault 25: 0x7ffc3568eadf - PyEval_EvalFrameDefault 26: 0x7ffc3557c3be - PyFunction_Vectorcall 27: 0x7ffc3557e6a1 - PyCell_Set 28: 0x7ffc3557e84d - PyMethod_Self 29: 0x7ffc35687e91 - PyOS_URandomNonblock 30: 0x7ffc35690402 - PyEval_GetFuncDesc 31: 0x7ffc3568c7be - PyEval_EvalFrameDefault 32: 0x7ffc3568eadf - PyEval_EvalFrameDefault 33: 0x7ffc3557c3be - PyFunction_Vectorcall 34: 0x7ffc35687e91 - PyOS_URandomNonblock 35: 0x7ffc35690402 - PyEval_GetFuncDesc 36: 0x7ffc3568c1bd - PyEval_EvalFrameDefault 37: 0x7ffc3568eadf - PyEval_EvalFrameDefault 38: 0x7ffc3557c3be - PyFunction_Vectorcall 39: 0x7ffc35687e91 - PyOS_URandomNonblock 40: 0x7ffc35690402 - PyEval_GetFuncDesc 41: 0x7ffc3568c763 - PyEval_EvalFrameDefault 42: 0x7ffc3568eadf - PyEval_EvalFrameDefault 43: 0x7ffc3557c3be - PyFunction_Vectorcall 44: 0x7ffc35687e91 - PyOS_URandomNonblock 45: 0x7ffc35690402 - PyEval_GetFuncDesc 46: 0x7ffc3568c763 - PyEval_EvalFrameDefault 47: 0x7ffc3568eadf - PyEval_EvalFrameDefault 48: 0x7ffc3557c3be - PyFunction_Vectorcall 49: 0x7ffc35687e91 - PyOS_URandomNonblock 50: 0x7ffc35690402 - PyEval_GetFuncDesc 51: 0x7ffc3568c763 - PyEval_EvalFrameDefault 52: 0x7ffc3568eadf - PyEval_EvalFrameDefault 53: 0x7ffc3557c3be - PyFunction_Vectorcall 54: 0x7ffc35687e91 - PyOS_URandomNonblock 55: 0x7ffc35690402 - PyEval_GetFuncDesc 56: 0x7ffc3568c763 - PyEval_EvalFrameDefault 57: 0x7ffc3568eadf - PyEval_EvalFrameDefault 58: 0x7ffc3557c3be - PyFunction_Vectorcall 59: 0x7ffc35687e91 - PyOS_URandomNonblock 60: 0x7ffc35690402 - PyEval_GetFuncDesc 61: 0x7ffc3568c1bd - PyEval_EvalFrameDefault 62: 0x7ffc3568eadf - PyEval_EvalFrameDefault 63: 0x7ffc3557c3be - PyFunction_Vectorcall 64: 0x7ffc3557e6a1 - PyCell_Set 65: 0x7ffc3557e84d - PyMethod_Self 66: 0x7ffc35687e91 - PyOS_URandomNonblock 67: 0x7ffc35690402 - PyEval_GetFuncDesc 68: 0x7ffc3568c7be - PyEval_EvalFrameDefault 69: 0x7ffc3568eadf - PyEval_EvalFrameDefault 70: 0x7ffc3557c3be - PyFunction_Vectorcall 71: 0x7ffc35687e91 - PyOS_URandomNonblock 72: 0x7ffc35690402 - PyEval_GetFuncDesc 73: 0x7ffc3568c763 - PyEval_EvalFrameDefault 74: 0x7ffc3568eadf - PyEval_EvalFrameDefault 75: 0x7ffc3557c3be - PyFunction_Vectorcall 76: 0x7ffc35687e91 - PyOS_URandomNonblock 77: 0x7ffc35690402 - PyEval_GetFuncDesc 78: 0x7ffc3568c763 - PyEval_EvalFrameDefault 79: 0x7ffc3568eadf - PyEval_EvalFrameDefault 80: 0x7ffc3557c3be - PyFunction_Vectorcall 81: 0x7ffc3557e6a1 - PyCell_Set 82: 0x7ffc3557e84d - PyMethod_Self 83: 0x7ffc35687e91 - PyOS_URandomNonblock 84: 0x7ffc35690402 - PyEval_GetFuncDesc 85: 0x7ffc3568c74d - PyEval_EvalFrameDefault 86: 0x7ffc3568eadf - PyEval_EvalFrameDefault 87: 0x7ffc3557c3be - PyFunction_Vectorcall 88: 0x7ffc35687e91 - PyOS_URandomNonblock 89: 0x7ffc35690402 - PyEval_GetFuncDesc 90: 0x7ffc3568c763 - PyEval_EvalFrameDefault 91: 0x7ffc3568eadf - PyEval_EvalFrameDefault 92: 0x7ffc3557c3be - PyFunction_Vectorcall 93: 0x7ffc35687e91 - PyOS_URandomNonblock 94: 0x7ffc35690402 - PyEval_GetFuncDesc 95: 0x7ffc3568c763 - PyEval_EvalFrameDefault 96: 0x7ffc3568eadf - PyEval_EvalFrameDefault 97: 0x7ffc3557c3be - PyFunction_Vectorcall 98: 0x7ffc3557e6a1 - PyCell_Set 99: 0x7ffc3557e884 - PyMethod_Self 100: 0x7ffc356a933c - PyContextVar_Reset 101: 0x7ffc355c0fbb - PyCFunction_GetFlags 102: 0x7ffc356906dc - PyEval_GetFuncDesc 103: 0x7ffc3568b560 - PyEval_EvalFrameDefault 104: 0x7ffc3568eadf - PyEval_EvalFrameDefault 105: 0x7ffc3557c3be - PyFunction_Vectorcall 106: 0x7ffc35687e91 - PyOS_URandomNonblock 107: 0x7ffc35690402 - PyEval_GetFuncDesc 108: 0x7ffc3568c763 - PyEval_EvalFrameDefault 109: 0x7ffc3568eadf - PyEval_EvalFrameDefault 110: 0x7ffc3557c3be - PyFunction_Vectorcall 111: 0x7ffc35687e91 - PyOS_URandomNonblock 112: 0x7ffc35690402 - PyEval_GetFuncDesc 113: 0x7ffc3568c763 - PyEval_EvalFrameDefault 114: 0x7ffc3568eadf - PyEval_EvalFrameDefault 115: 0x7ffc3557c3be - PyFunction_Vectorcall 116: 0x7ffc35687e91 - PyOS_URandomNonblock 117: 0x7ffc35690402 - PyEval_GetFuncDesc 118: 0x7ffc3568c763 - PyEval_EvalFrameDefault 119: 0x7ffc3568eadf - PyEval_EvalFrameDefault 120: 0x7ffc3557c3be - PyFunction_Vectorcall 121: 0x7ffc35687e91 - PyOS_URandomNonblock 122: 0x7ffc35690402 - PyEval_GetFuncDesc 123: 0x7ffc3568c763 - PyEval_EvalFrameDefault 124: 0x7ffc3568eadf - PyEval_EvalFrameDefault 125: 0x7ffc3557c3be - PyFunction_Vectorcall 126: 0x7ffc35687e91 - PyOS_URandomNonblock 127: 0x7ffc35690402 - PyEval_GetFuncDesc 128: 0x7ffc3568c763 - PyEval_EvalFrameDefault 129: 0x7ffc3568eadf - PyEval_EvalFrameDefault 130: 0x7ffc3557c3be - PyFunction_Vectorcall 131: 0x7ffc3557e6a1 - PyCell_Set 132: 0x7ffc3557e84d - PyMethod_Self 133: 0x7ffc35687e91 - PyOS_URandomNonblock 134: 0x7ffc35690402 - PyEval_GetFuncDesc 135: 0x7ffc3568c74d - PyEval_EvalFrameDefault 136: 0x7ffc3568eadf - PyEval_EvalFrameDefault 137: 0x7ffc35689782 - PyEval_EvalCode 138: 0x7ffc35684aeb - PyWarnings_Init 139: 0x7ffc355c0f29 - PyCFunction_GetFlags 140: 0x7ffc35687e91 - PyOS_URandomNonblock 141: 0x7ffc35690402 - PyEval_GetFuncDesc 142: 0x7ffc3568c1bd - PyEval_EvalFrameDefault 143: 0x7ffc3568eadf - PyEval_EvalFrameDefault 144: 0x7ffc3557c3be - PyFunction_Vectorcall 145: 0x7ffc35687e91 - PyOS_URandomNonblock 146: 0x7ffc35690402 - PyEval_GetFuncDesc 147: 0x7ffc3568c1bd - PyEval_EvalFrameDefault 148: 0x7ffc3568eadf - PyEval_EvalFrameDefault 149: 0x7ffc3557c3be - PyFunction_Vectorcall 150: 0x7ffc354fb601 - PyObject_GC_IsFinalized 151: 0x7ffc354fc3a1 - PyObject_GC_IsFinalized 152: 0x7ffc354fd2f3 - Py_RunMain 153: 0x7ffc354fd366 - Py_Main 154: 0x7ff7aec01490 - OPENSSL_Applink 155: 0x7ffc8f2e257d - BaseThreadInitThunk 156: 0x7ffc9064aa58 - RtlUserThreadStart [I 2023-11-30 14:27:44.677 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), keep random ports [W 2023-11-30 14:27:44.678 ServerApp] kernel 82d8f2be-497a-434b-beb2-c483419f345d restarted [I 2023-11-30 14:27:44.685 ServerApp] Starting buffering for 82d8f2be-497a-434b-beb2-c483419f345d:ae15c4ea-d529-46d5-a7e3-6d2fe0a920b6 [I 2023-11-30 14:27:44.698 ServerApp] Connecting to kernel 82d8f2be-497a-434b-beb2-c483419f345d. [I 2023-11-30 14:27:44.698 ServerApp] Restoring connection for 82d8f2be-497a-434b-beb2-c483419f345d:ae15c4ea-d529-46d5-a7e3-6d2fe0a920b6 [W 2023-11-30 14:28:03.390 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d/channels?session_id=36a619de-16c9-4b33-bad7-d70cf80dd4e3 (::1): Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d [W 2023-11-30 14:28:03.391 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d/channels?session_id=36a619de-16c9-4b33-bad7-d70cf80dd4e3 (3e46ca4e87ed40ca93a4822f738293e5@::1) 1.00ms referer=None [W 2023-11-30 14:28:03.394 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d?1701383283392 (::1): Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d [W 2023-11-30 14:28:03.394 ServerApp] wrote error: 'Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d' Traceback (most recent call last): File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\tornado\web.py", line 1786, in _execute result = await result File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\handlers.py", line 75, in get model = await ensure_async(km.kernel_model(kernel_id)) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\kernelmanager.py", line 500, in kernel_model self._check_kernel_id(kernel_id) File "C:\Users\yyizhou\anaconda3\envs\mescore\lib\site-packages\jupyter_server\services\kernels\kernelmanager.py", line 531, in _check_kernel_id raise web.HTTPError(404, "Kernel does not exist: %s" % kernel_id) tornado.web.HTTPError: HTTP 404: Not Found (Kernel does not exist: 03c77a51-a2f0-415b-ab0b-5a7e9a155c1d) [W 2023-11-30 14:28:03.394 ServerApp] 404 GET /api/kernels/03c77a51-a2f0-415b-ab0b-5a7e9a155c1d?1701383283392 (3e46ca4e87ed40ca93a4822f738293e5@::1) 1.01ms referer=http://localhost:8888/lab/workspaces/auto-O/tree/cnmfe_demo.ipynb [I 2023-11-30 14:28:50.519 ServerApp] Saving file at /cnmfe_demo.ipynb

[kushalkolar]

I wonder if the Intel integrated GPUs have some limits for large visualizations. To confirm that it's binding to the Intel GPU can you run this:

import numpy as np
import fastplotlib as fpl

a = np.random.rand(100, 100)

plot = fpl.Plot()
plot.add_image(a)
plot.show()

In the next cell run this and past the output:

plot.renderer.device.adapter.request_adapter_info()

We have plans to make it possible to choose GPU for fastplotlib, so if it's binding to the Intel one by default you can force it to use the dedicated GPU instead. I can look into it over the next few days.

[zhouyi0812]

Here is the result

[kushalkolar]

Ah it's binding the dedicated gpu, that's weird (in that stuff crashes). I have an older AMD gpu that I use for testing and haven't run into issues.

Are the AMD graphics drivers installed?

[kushalkolar]

Are you free to get on a quick call tomorrow? Making fastplotlib robust and reliable is a priority. I'm in US Eastern time.

On Thu, Nov 30, 2023, 18:06 Yi Zhou @.***> wrote:

Screenshot.2023-11-30.150507.png (view on web) https://github.com/nel-lab/mesmerize-core/assets/48576330/9b2b9cf1-701a-4e5d-9c0d-196348572d74 Here is the result

— Reply to this email directly, view it on GitHub https://github.com/nel-lab/mesmerize-core/issues/244#issuecomment-1834697008, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACHXXRAQVUGQTW4B7IXG5JLYHEGN7AVCNFSM6AAAAAA634EZNOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZUGY4TOMBQHA . You are receiving this because you commented.Message ID: @.***>

[zhouyi0812]

sure!

[zhouyi0812]

i can even do now if you are available

[kushalkolar]

What's your email? I'll delete the comment soon after so it's not on the open internet

[zhouyi0812]

Ah it's binding the dedicated gpu, that's weird (in that stuff crashes). I have an older AMD gpu that I use for testing and haven't run into issues.

Are the AMD graphics drivers installed?

I have installed this after you told me: Install AMD Software: Adrenalin Edition And the kernel still dead

[zhouyi0812]

Hello! Also I found there is an error with the pnr imagine which doesn't have before Thank you!

TypeError Traceback (most recent call last) Cell In[27], line 18 12 # sync the threshold image widget with the corr-pnr plot 13 threshold_grid_plot_kwargs = { 14 "controllers": [[iw_corr_pnr.gridplot["corr"].controller]2]2, 15 "size": (650, 600) 16 } ---> 18 iw_thres_movie = fpl.ImageWidget( 19 mcorr_vids, 20 names=["over corr threshold", "over pnr threshold", "under corr threshold", "under pnr threshold"], 21 # sync this with the corr-pnr plot 22 grid_plot_kwargs=threshold_grid_plot_kwargs, 23 cmap="gnuplot2" 24 ) 26 # display threshold of the spatially filtered movie 27 def spatial_filter(frame):

File ~\anaconda3\envs\mescore\lib\site-packages\fastplotlib\widgets\image.py:561, in ImageWidget.init(self, data, dims_order, slider_dims, window_funcs, frame_apply, grid_shape, names, grid_plot_kwargs, histogram_widget, kwargs) 557 # update the default kwargs with any user-specified kwargs 558 # user specified kwargs will overwrite the defaults 559 grid_plot_kwargs_default.update(grid_plot_kwargs) --> 561 self._gridplot: GridPlot = GridPlot(shape=grid_shape, grid_plot_kwargs_default) 563 for data_ix, (d, subplot) in enumerate(zip(self.data, self.gridplot)): 564 if self._names is not None:

TypeError: GridPlot.init() got an unexpected keyword argument 'controllers'

[zhouyi0812]

Ah it's binding the dedicated gpu, that's weird (in that stuff crashes). I have an older AMD gpu that I use for testing and haven't run into issues. Are the AMD graphics drivers installed?

I have installed this after you told me: Install AMD Software: Adrenalin Edition And the kernel still dead

Hello, just want to give more error message when I rerun the code, this time it took a while for kernal to dead. And this is the error on eval parameters: Thank you!!!!

2.5 [Open Browser Console for more detailed log - Double click to close this message] Failed to create view for 'FloatSliderView' from module '@jupyter-widgets/controls' with model 'FloatSliderModel' from module '@jupyter-widgets/controls' Error: noUiSlider: 'step' is not numeric. at Object.N [as t] (http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/345.17494fea1ff555b26294.js?v=17494fea1ff555b26294:1:17628) at http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/345.17494fea1ff555b26294.js?v=17494fea1ff555b26294:1:23416 at Array.forEach () at W (http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/345.17494fea1ff555b26294.js?v=17494fea1ff555b26294:1:23364) at Object.G [as create] (http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/345.17494fea1ff555b26294.js?v=17494fea1ff555b26294:1:38320) at Re.createSlider (http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/495.79062b4ce5ec7920dcb1.js?v=79062b4ce5ec7920dcb1:1:38924) at Re.render (http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/495.79062b4ce5ec7920dcb1.js?v=79062b4ce5ec7920dcb1:1:37372) at http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/336.ebc7a55ea1768712771f.js?v=ebc7a55ea1768712771f:1:2959 at async Promise.all (index 1)

[kushalkolar]

closing