Struggling to get make 'make run' and 'make run-tmpnb' to run on linux 14.04

[Analect]

Thanks for your efforts on this repo ... I'm watching it with interest ... however:

I ran make run ... which appeared to succeed in building images and then launching containers ... I'm not sure if the bower install step didn't get run properly or something ... the terminal logs would suggest maybe it's missing express (see logs lower down)

The screen that is meant to list notebooks was blank ... and this is what shows up for the simple notebook.

-- Starting proxy container
npm info it worked if it ends with ok
npm info using npm@3.3.12
npm info using node@v5.5.0
npm info lifecycle node-deployed-dashboard@0.0.1~prestart: node-deployed-dashboard@0.0.1
npm info lifecycle node-deployed-dashboard@0.0.1~start: node-deployed-dashboard@0.0.1

> node-deployed-dashboard@0.0.1 start /home/node/app
> node ./bin/www

GET / 302 49.128 ms - 64
GET /notebooks 200 240.792 ms - 281
STACK: Error: Not Found
    at /home/node/app/app.js:106:15
    at Layer.handle [as handle_request] (/home/node/app/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/node/app/node_modules/express/lib/router/index.js:312:13)
    at /home/node/app/node_modules/express/lib/router/index.js:280:7
    at Function.process_params (/home/node/app/node_modules/express/lib/router/index.js:330:12)
    at next (/home/node/app/node_modules/express/lib/router/index.js:271:10)
    at /home/node/app/node_modules/express/lib/router/index.js:618:15
    at next (/home/node/app/node_modules/express/lib/router/index.js:256:14)
    at Function.handle (/home/node/app/node_modules/express/lib/router/index.js:176:3)
    at router (/home/node/app/node_modules/express/lib/router/index.js:46:12)
GET /sw-import.js?baseURI=http%3A%2F%2Flocalhost%3A3000%2Fbower_components%2Fplatinum-sw%2Fplatinum-sw-register.html&clientsClaim=true&defaultCacheStrategy=networkFirst&importscript=http%3A%2F%2Flocalhost%3A3000%2Fbower_components%2Fplatinum-sw%2Fbootstrap%2Fsw-toolbox-setup.js&precache=&skipWaiting=true&version=1.0 404 13.974 ms - 878
STACK: Error: Not Found

I tried to stop and restart the dashboard-proxy container, so that I might troubleshoot things manually inside the container ... but ctrl-c of the make run process appears to kill and remove that container.

I don't have that much experience with make, but each time a run that command, it appears to build a whole new image ... which obviously isn't as slow as the first time, as it uses caches etc ... but is there scope for getting this working using docker-compose, especially once the images are built?

When running make run-tmpnb I also hit a hurdle on it complaining that port 8000 is already being used ... Although I have other docker containers running ... none of them is touching port 8000 ... does this 1005/node on port 8000 look like anything related to this set-up?

Step 21 : CMD start
 ---> Using cache
 ---> 0e8b480a3420
Successfully built 0e8b480a3420
4eb7912c7e6fd15e096e18a8036bd476fd1c27c2dc831c084037d6286a6d3662
Error response from daemon: Cannot start container 4eb7912c7e6fd15e096e18a8036bd476fd1c27c2dc831c084037d6286a6d3662: failed to create endpoint tmpnb-proxy on network bridge: Error starting userland proxy: listen tcp 0.0.0.0:8000: bind: address already in use
make: *** [run-tmpnb-proxy] Error 1
mccoole@mccoole-ubuntu-W520:~/Development/Tools/jupyter/dashboards_nodejs_app$ netstat -tulpn | grep 8000
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      1005/node

[jtyberg]

FYI, the make run-tmpnb is kind of experimental. We've been experimenting with different ways of running kernel gateway instances that the dashboard app can talk to. So make run-tmpnb really means, run the dashboard app and have it talk to tmpnb to get kernels from kernel gateway instances. Unfortunately, the dashboard app code base does not even handle this scenario yet.

Bottom line, for now, use make run (or make run-logging/make run-debug) for additional output. This will start a single docker container running a single kernel gateway instance that the dashboard app can talk to.

Also, the project is still very early stages, so it's really only setup for dev only. That's one reason we rebuild the dashboard app image on a make run - we want to include any code changes. If you don't touch any files, the image build step is essentially a no-op.

As for the port conflict, your output does indicate that there is a node app running on port 8000, so it makes sense that Docker would complain. If you switch to make run, the kernel gateway container exposes port 8888 instead of 8000 (tmpnb).

As for the stack trace, it does seem like the node environment (within the container) is missing something. I can't reproduce. Any ideas @parente @jhpedemonte @dalogsdon ?

[jtyberg]

Just double-checking, using current master code base, I removed my existing dashboard app Docker image and rebuilt from scratch. I did not have an issue with the build or running the dashboard app.

docker rmi jupyter-incubator/dashboard-proxy
make build
make run
...
Successfully built 86942aeb3a76
-- kernel-gateway is already running.
-- Starting proxy container
npm info it worked if it ends with ok
npm info using npm@3.3.12
npm info using node@v5.5.0
npm info lifecycle node-deployed-dashboard@0.0.1~prestart: node-deployed-dashboard@0.0.1
npm info lifecycle node-deployed-dashboard@0.0.1~start: node-deployed-dashboard@0.0.1

> node-deployed-dashboard@0.0.1 start /home/node/app
> node ./bin/www

Attempting to load notebook file: /home/node/app/data/simple.ipynb
GET /notebooks/simple 304 68.039 ms - -
GET /components/jquery-ui/jquery-ui.min.css 200 8.645 ms - 30021
GET /components/widgets.min.css 200 11.315 ms - 6493
GET /components/gridstack.min.css 200 12.163 ms - 7044
GET /css/style.css 200 12.750 ms - 2950
GET /components/bootstrap/css/bootstrap.min.css 200 6.493 ms - 121260
GET /components/require.js 200 13.502 ms - 85921
GET /js/dashboard.js 304 0.719 ms - -
GET /js/gridstack-custom.js 304 2.115 ms - -
GET /js/widget-manager.js 304 6.792 ms - -
GET /js/kernel.js 304 0.393 ms - -
GET /components/jquery.min.js 200 11.074 ms - 85589
GET /components/gridstack.min.js 200 8.583 ms - 17506
GET /components/jupyter-js-services.js 200 6.797 ms - 135504
GET /components/jupyter-js-output-area.js 200 10.890 ms - 763195
GET /components/jupyter-js-widgets.js 200 11.701 ms - 1118612
GET /components/lodash.min.js 200 6.685 ms - 50543
GET /components/jquery-ui/widget.min.js 200 7.118 ms - 6829
GET /components/jquery-ui/mouse.min.js 200 7.393 ms - 3066
GET /components/jquery-ui/draggable.min.js 200 8.164 ms - 18831
GET /components/jquery-ui/resizable.min.js 200 8.939 ms - 18335
GET /components/jquery-ui/core.min.js 200 7.692 ms - 3902
POST /api/kernels?1454432652589 201 47.312 ms - 65
GET /notebooks 304 7.857 ms - -
GET / 302 3.777 ms - 64
GET /notebooks 304 4.647 ms - -

[Analect]

@jtyberg Thanks for your responses. I'll try to remove and rebuild ... maybe some network issues were behind the last image building properly. I'll let you know how I get on.

[parente]

The screen that is meant to list notebooks was blank ... and this is what shows up for the simple notebook.

The list notebooks screen is basically blank at the moment. PR #35 is addressing that. Once you do reach the simple dashboard, what you see looks correct to me.

The weirdness in the logs might simply be #37. We appear to be logging stack traces even for simple 404s on requests the browser makes to things like *.js.map and whatnot. That sw-import.js in your trace looks suspiciously like something a browser extension is requesting, not necessarily part of this project.

[Analect]

@jtyberg Thanks for your assistance. Closing this out, as Peter seems to think it looks right. @parente Yes, that sw-imports.js stuff relates to some service-worker stuff that must have got installed in the browser as part of a polymer starter kit, from what I've googled elsewhere.

[parente]

@Analect Thanks for giving the project a shot. I know the few samples in this repo aren't much to look at yet. Hopefully the ipywidget based demos from https://github.com/jupyter-incubator/dashboards/tree/master/etc/notebooks will start to work here reasonably soon. Maybe a little automation to vendor those into the dev environment here would start to push the envelope and making things a bit more attractive.

[Analect]

@parente Thanks for the extra colour above. I try to piece together (often from the monthly jupyter.hackpad.com monthly sessions) the various initiatives that are happening under the jupyter project, which seem too numerous to count! I'm really interested in what your IBM group are doing vis a vis some of this incubator stuff, but it's not clear to me if/when this will get folded back into Jupyter core (or if indeed that's even the plan), since there are now a whole bunch of ipywidgets and services that are intended as plugins ...

There's obviously also the work happening on phosphorjs, but as an outsider looking in, it's still hard to see how that is tying back in ... maybe all will be revealed in version 5.0 of the notebook, which seems like it's pushing to leverage javascript as the front-end on top of the various kernels that Jupyter exposes.

In the meantime, I install repos like this to try to learn a little more! It would be great to have a diagram of how everything in the eco-system is planned to hang together ... a view from 30,000ft, if you like! I know that's probably not your responsibility ... but as one of the more active collaborators on the project, you have a perspective better than most ... and you write well, as per your blogs.

[parente]

I'm really interested in what your IBM group are doing vis a vis some of this incubator stuff, but it's not clear to me if/when this will get folded back into Jupyter core (or if indeed that's even the plan), since there are now a whole bunch of ipywidgets and services that are intended as plugins ...

:warning: Warning, warning, warning: My opinion/perspective below. May not match what others think! :warning:

I look at the work we're doing here in the incubator as "it works today" implementations of dashboard layout (jupyter-incubator/dashboards), with cross-kernel widgets (jupyter-incubator/declarativewidgets), able to be deployed (jupyter-incubator/dashboards_bundlers), and run in a standalone web app (jupyter-incubator/dashboards_nodejs_app). The dashboards roadmap tries to capture how these come together with respect to the current Jupyter Notebook 4.x implementation.

Now, in the not-so-distant future, when Jupyter Lab (Notebook 5.x and beyond) matures to the point where there's a notebook experience and APIs for extending it equivalent to what exists today, some pieces of what we've built here will need re-design and/or re-implementation to work against that new platform (e.g., the dashboard layout extension). Other pieces will need to keep pace with the platform as it matures (e.g., the code here in the dashboard_nodejs_app already uses the bleeding edge packages like jupyter-js-services, jupyter-js-widgets, jupyter-js-output-area, etc.). Still others might migrate out of incubator and become top-level projects as-is (e.g., I hope to get the bundler API proposed as a Jupyter enhancement soonish; the declarative widgets can largely standalone as long as they keep pace with ipywidgets). These are not unique cases: most of the Jupyter extension ecosystem (e.g., nbgrader, nbpresent, RISE, ...) is going to need some amount of work to fit into the new Jupyter Lab platform.

At the moment, I think working here in the incubator and getting people to try out the functionality is the right thing to do. It's more important at this stage that we learn what works and what doesn't, and see if these concepts even have merit. If they do, then we can figure out how to move the implementations forward with the Jupyter platform. I believe the required proposal for incorporation will be the place to capture how the concepts, if not the entire projects, will leave the incubator when "the time is right". My gut tells me that time will be when Jupyter Lab has at least one stable release, when we've gained some experience trying to extend it, and we can make informed decisions about how the various incubating features and/or code can (or cannot) be folded into it.

[Analect]

Very clear. Thanks for taking the time to set this out.

You had a series demos pertaining to the kernel gateway. Would it be possible to make these binder-friendly ... I know some of them already have dockerfiles, but from recollection, I think binker likes to have a dockerfile at the top-level within a repo, so I'm not sure if that will work, where the dockerfile associated with a given demo is in a sub-folder.

On your dashboards bundler repo, you say:

You must refer to external, kernel-side resources in a portable manner (e.g., put it in an external data store, use absolute file paths if your only concern is File → Deploy as → Local Dashboard). You must also ensure your kernel environment has all the same libraries installed as your notebook authoring environment.

So is there no practical way for a bundler to do what something like binder is doing .. in terms of installing dependencies ahead of viewing a dashboard ... or is that way too heavy-weight an infrastructure to be able to support bundles?

Just one final thing. I notice in some of your recent sample notebooks, you're using *.dataframe files from box.ibm.com, which are obviously more portable for demos.

However, is there anything evolving in the kernel gateway ecosystem that manages a gateway of sorts into a backend datastore such as postgres (or even some of the no-sql stores).. where perhaps there's an ability for a user, if required, to supply credentials when in one of the notebook shells ... such that you could have a pre-structured query that might drop the data into a widget like the polestar one below. http://nbviewer.jupyter.org/github/uwdata/ipython-vega/blob/master/Example.ipynb

Thanks again.

[ellisonbg]

@parente thanks for the great summary of the state of things on this front. It is encouraging to see the progress that the incubation projects are making (the sparkmagic one too). We definitely need to continue to develop the main project roadmap as well though:

https://github.com/jupyter/roadmap

I expect we will be able to make progress on that at the in person core-dev meeting in March/April.

[fperez]

@parente, thanks for pitching in, I couldn't have said it better.

Just a few days ago, I was talking to @minrk and we were discussing the need for a good set of descriptions that is a "map of the territory" for the entire project (Jupyter + IPython). While at this point we have more moving parts than it's reasonable for anyone to keep in their own head, we're sort of in a situation that feels like living in a city without maps: you know your neighborhood and the one around your job, but you have no idea what else is there, how to get somewhere you don't know, etc. Some of us do have a full mental map in our heads (albeit a high-level one), but for newcomers this makes the project incredibly hard to approach.

The value of a map is that it lets you discover and navigate the territory you don't know intimately only as needed: you survey the landscape, you zoom into the part you need to go to, and you find your way around.

We lack that, and it's a real problem. With the great recent work spearheaded by @willingc, @captainsafia and others, we're make progress here, but we still have a long way to go, especially when it comes to that coherent high-level view of the architecture and project. We'll keep working on it!

[parente]

@Analect Sorry that I lost track of your last set of questions. Maybe we should take them to the Jupyter group so they're not squirreled away in this issue. Would you mind reposting your last set there?

[Analect]

@parente ... no problem ... you've been more than helpful.