pangeo.esipfed.org interface very slow

[dbuscombe-usgs]

Over the past few days the web interface http://pangeo.esipfed.org/ has been slow. Today it is really slow. It often takes several minutes to start server, and load an .ipynb file. Having to close and restart several times.

[dbuscombe-usgs]

So slow right now its unusable. 15 minutes to load a page. My internet connection is fast. Giving up for a while

[rsignell-usgs]

I just started my server and it came up in 30 seconds or so, and notebook came up and ran normally. I wonder if you've used up all your disk space or something. Can you start a new terminal and do a df -h to see?

[dbuscombe-usgs]

I'm using 30% disk space. I've tried again and it is still slow, just loading one notebook. I'll try another network tomorrow

[rsignell-usgs]

Did you do “top”? Did you try restarting your server?

-Rich

On Jul 1, 2018, at 17:20, Daniel Buscombe notifications@github.com wrote:

I'm using 30% disk space. I've tried again and it is still slow, just loading one notebook. I'll try another network tomorrow

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[dbuscombe-usgs]

Thanks - yes I have restarted a few times. Top shows almost no resources used. But after loading a file for a few mins, it freezes completely. No idea, its weird

[rsignell-usgs]

Can you share your notebooks with me so I can try?

[dbuscombe-usgs]

I think I figured it out. Basically, some of my notebooks are too large because tensorflow produces TONS of outputs and they remain in the notebook when executed. To try to counter this, I issue:

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'

which filters out INFO logs, warnings, and errors, but still produces thousands of lines of stdout. I could resort to using

%%capture

at the start of the cell but there is some useful info in all the output. I may just have to use a series of very small notebooks

[dbuscombe-usgs]

I guess I could use

%%capture capt
%run myscript.py

then, if we want to view output

capt.stdout

?

[rsignell-usgs]

@ocefpaf, have you encountered this kind of problem with stdout in Jupyter and/or have any good ideas?

[dbuscombe-usgs]

Looks like

capt.stdout

only captures the last line of the output

[ocefpaf]

@ocefpaf, have you encountered this kind of problem with stdout in Jupyter and/or have any good ideas?

No, but I never tried anything like that. I'll give it a try and report back.

[dbuscombe-usgs]

Just to clarify, one particular script I have (that I run using the %run magic command) generates tens of thousands of lines of output. I have reduced it as much as possible but most of it comes from low-level c++ routines.

Using %%capture is working well, except that I would like to be able to see the last "N" lines of output. I thought I'd be able to pass capt.stdout to a file then use tail to view, but it appears capt.stdout only captures the last line, I guess because most of it wasn't being generated by the python script, by by lower-level calls

[dbuscombe-usgs]

Update: this works well, using the bash magic command

%%bash -s "$var1" "$var2"
python myscript.py --flag1 $1 --flag2 $2 &> output.txt

[ocefpaf]

I was going to suggest the bash magic, if the problem can be platform dependent, or a thin wrapper that would save the stdout to a file.

[rsignell-usgs]

Hurrah for bash cell magic! Okay to close @dbuscombe-usgs ?

[dbuscombe-usgs]

Yep, case closed