Login sessions partially load then fail - no error in logs

[IanButterworth]

On an up-to-date fresh install, sessions aren't completing their loading after google auth. The page hangs until they timeout after 1 min.

During this hanging period the content on the iJulia pane doesn't load, however all other tabs are functional, (the console is accessible and can load julia, the file pane shows the tutorial directory present, and the settings are as usual)

There are no errors sent to the enginedaemon, engineinteractive, or engineapi logs.

Are there any other locations I can search for error logs?

Example content from enginedaemon.log:

2016-10-22 23:23:32,933 - DEBUG - juliabox.srvr_jboxd.JBoxd - JBox daemon waiting for commands...
2016-10-22 23:23:33,148 - DEBUG - juliabox.srvr_jboxd.JBoxd - processing offline...
2016-10-22 23:23:33,150 - INFO - juliabox.plugins.vol_loopback.loopback.JBoxLoopbackVol - Loopback Disk free: 8/8
2016-10-22 23:23:33,151 - INFO - juliabox.vol.jbox_volume.JBoxVol - Packages in use: set([])
2016-10-22 23:23:33,155 - INFO - juliabox.plugins.compute_singlenode.impl_singlenode.CompSingleNode - Stats: JuliaBox.localhost.NumActiveContainers=0(Count)
2016-10-22 23:23:33,155 - INFO - juliabox.plugins.compute_singlenode.impl_singlenode.CompSingleNode - Stats: JuliaBox.localhost.NumActiveAPIContainers=0(Count)
2016-10-22 23:23:33,155 - INFO - juliabox.plugins.compute_singlenode.impl_singlenode.CompSingleNode - Stats: JuliaBox.localhost.CPUUsed=1(Percent)
2016-10-22 23:23:33,155 - INFO - juliabox.plugins.compute_singlenode.impl_singlenode.CompSingleNode - Stats: JuliaBox.localhost.MemUsed=14.3(Percent)
2016-10-22 23:23:33,155 - INFO - juliabox.plugins.compute_singlenode.impl_singlenode.CompSingleNode - Stats: JuliaBox.localhost.DiskUsed=3.0(Percent)
2016-10-22 23:23:33,155 - INFO - juliabox.plugins.compute_singlenode.impl_singlenode.CompSingleNode - Stats: JuliaBox.localhost.ContainersUsed=0(Percent)
2016-10-22 23:23:33,155 - INFO - juliabox.plugins.compute_singlenode.impl_singlenode.CompSingleNode - Stats: JuliaBox.localhost.APIContainersUsed=0(Percent)
2016-10-22 23:23:33,156 - INFO - juliabox.plugins.compute_singlenode.impl_singlenode.CompSingleNode - Stats: JuliaBox.localhost.DiskIdsUsed=0.0(Percent)
2016-10-22 23:23:33,156 - INFO - juliabox.plugins.compute_singlenode.impl_singlenode.CompSingleNode - Stats: JuliaBox.localhost.Load=14.3(Percent)
2016-10-22 23:23:33,159 - DEBUG - juliabox.plugins.compute_singlenode.impl_singlenode.CompSingleNode - Using cached self_stats. Load=14.3
2016-10-22 23:23:33,159 - DEBUG - juliabox.plugins.compute_singlenode.impl_singlenode.CompSingleNode - Using cached self_stats. Load=14.3
2016-10-22 23:23:33,159 - DEBUG - juliabox.plugins.compute_singlenode.impl_singlenode.CompSingleNode - Using cached self_stats. Load=14.3
2016-10-22 23:23:33,159 - DEBUG - juliabox.plugins.compute_singlenode.impl_singlenode.CompSingleNode - Self load: 14.3
2016-10-22 23:23:33,167 - DEBUG - juliabox.srvr_jboxd.JBoxd - JBox daemon waiting for commands...

[IanButterworth]

The contents of /opt/juliabox/.juliabox/ijulia_err.log located within the session volume are:

[TerminalIPythonApp] WARNING | Subcommand `ipython notebook` is deprecated and will be removed in future versions.
[TerminalIPythonApp] WARNING | You likely want to use `jupyter notebook` in the future                                                       
[W 00:42:14.456 NotebookApp] Widgets are unavailable. Please install widgetsnbextension or ipywidgets 4.0                                    
[I 00:42:14.466 NotebookApp] Serving notebooks from local directory: /home/juser                                                             
[I 00:42:14.466 NotebookApp] 0 active kernels                                                                                                
[I 00:42:14.466 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/                                                      
[I 00:42:14.466 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).                       
[W 00:42:14.467 NotebookApp] No web browser found: could not locate runnable browser.

[tanmaykm]

Browser console or webserver logs may give some clues.

Is the juliabox.plugins.vol_defcfg plugin configured? You may need to delete the old configs at /jboxengine/data/configs/*?

[IanButterworth]

The browser console isn't showing any errors (apart from typekit, which i never get around to removing)

/jboxengine/data/configs/ is empty. Should there be something in there, or were you referring to remnants of old setup approaches?

I wondered if it's a port issue that's changed recently? I didn't use to open port 8998, but I have now after seeing it used for ijulia, but that hasn't changed anything.

I'll look into webserver logs

Thanks!

[IanButterworth]

When is the juliabox.plugins.vol_defcfg plugin configured during the install? I can't seem to find it

[tanmaykm]

It's part of the usual plugins section of jbox.user (see: INSTALL.MD) and uses config parameter cfg_location to locate folder to use. Default configuration is built by mk_user_home.sh.

The new Dockerfile installs newer Jupyter which may have changed the port number, but the port number is supposed to be configured by ipython_notebook_config.py

[IanButterworth]

Still not working.

Indeed, I have juliabox.plugins.vol_defcfg in my jbox.user and I see where cfg_location is defined in tornado.conf.tpl but I can't see it used anywhere else, or how it's used in mk_user_home.sh

My issue with ports I was referring to is because I'm running on AWS, so have to adjust the security group. I've now got 22, 80, 443, and 8998 open.

I'm stumped.. going to try another re-install

[nkottary]

Port numbers 8887 through 8890 must be open for ZMQ communication.

[tanmaykm]

@nkottary ZMQ ports should be opened only between machines in the cluster, not to the internet, right?

@ianshmean is this a single machine or cluster deployment?

[nkottary]

Yes, only between machines.