Kernel restarting

[siljeci]

When I try to run a code by using JupyterHub I get the message:

"Kernel restarting: The kernel for Untitled1.ipynb appears to have died. It will restart automatically."

It seems like this happens when I use large arrays. Is this a "normal" message, such that the only fix is to slice every dataset before using them?

[annefou]

It depends how you manipulate your xarray and what kind of data processing you perform. We can increase resources but before can we look at what you do? Do you want to look at here now? Where are you?

[siljeci]

It happens when I calculate weighted averages by using groupby().sum() on arrays which are large(!). We don't have to fix this tonight - I figured that the code works if I slice the datasets. So if I run into this problem again (and don't figure out what to do), I'll ask again!

[annefou]

Yes. Slicing is the right approach and actually you can use dask cluster to run your computation in parallel.

[daliagachc]

Hi! i'll write an example notebook on using xarray with dask so we can circumvent this problem. @annefou can you point me how to profile how much ram im using in the notebook?

[jrieksta]

Hei! Has this been fixed? I cannot plot any of my data as kernel crashes.

[annefou]

Do you use dask cluster?

[jrieksta]

No! @daliagachc earlier in the comments mentioned he will make a notebook for that, can we find it somewhere?

[annefou]

I put a very simple example at https://nordicesmhub.github.io/NEGI-Abisko-2019/training/CMIP6_dask.html Will add some more on how to profile and optimize.

[daliagachc]

Hi! @jrieksta here is another example notebook location

[sarambl]

Hey, @daliagachc! Maybe you can add the client thing here? https://nordicesmhub.github.io/NEGI-Abisko-2019/training/gridded_model_data_dask_rolling_mean.html Might be misleading if people go here

[daliagachc]

@sarambl agreed! how? :)

[sarambl]

Are you telling me I should do it, @daliagachc ?!

[daliagachc]

lets see.... i forked, modified and pull requested.

[sarambl]

Ahhh, now I get the question! Yes, if you don't already have a fork (in which case u sync it ) https://nordicesmhub.github.io/NEGI-Abisko-2019/training/github/synchronise_github.html

[jrieksta]

Is this error also related to dask? Worker exceeded 95% memory budget. Restarting I have chunks 10 and kernel still sometimes dies, would a suggestion would be to decrease chunks?

[annefou]

@jrieksta can you point to your notebook?

[daliagachc]

Hi @jrieksta! can you post the path to your notebook? (right click on the notebook in the file browser of jupyterhub and select 'copy shareable link')

[annefou]

@siljeci got it work. my visio finishes in few minutes.