More informative message on "Launch attempt failed retrying"

[tonyyang-svail]

Hi binder team,

Firstly, thank you for creating this project.

I was trying it out on https://github.com/tonyyang-svail/sqlflow branch mybinder. The Docker container was successfully built. However, when launching the corresponding site, it complains

Found built image, launching...
Launching server...
Launch attempt 1 failed, retrying...
Launch attempt 2 failed, retrying...
Launch attempt 3 failed, retrying...
Internal Server Error

Is there a way to get more detailed logs? How can I reproduce the error locally?

Thanks.

[choldgraf]

I'd recommend trying the build yourself locally using repo2docker - it's a tool that Binder is using under the hood, and if you run it locally then you might have more control over the Docker images that are created etc.

That said, looking at your repository, you have a custom Dockerfile in there - try following the instructions in the Binder guide for using a Dockerfile which provides info on how to make your code work with BinderHub

[tonyyang-svail]

Hi @choldgraf, thank you for your suggestion. I have tested the repo locally with repo2docker and it works fine.

$ jupyter-repo2docker --ref mybinder  https://github.com/tonyyang-svail/sqlflow/
Picked Git content provider.
Cloning into '/var/folders/bx/5_28cd7n3qv72r3t0q96nhzr0000gp/T/repo2dockerv1p3q4oi'...
remote: Enumerating objects: 33, done.
remote: Counting objects: 100% (33/33), done.
remote: Compressing objects: 100% (22/22), done.
remote: Total 5931 (delta 14), reused 22 (delta 11), pack-reused 5898
Receiving objects: 100% (5931/5931), 4.88 MiB | 1.16 MiB/s, done.
Resolving deltas: 100% (3768/3768), done.
HEAD is now at fce19d4 change NB_USER to hard coded jovyan
Using DockerBuildPack builder
Step 1/16 : FROM sqlflow/sqlflow:latest
 ---> 8920ea8802e5
Step 2/16 : ARG NB_USER=jovyan
 ---> Using cache
 ---> 9cc1917fa2aa
Step 3/16 : ARG NB_UID=1000
 ---> Using cache
 ---> 4fb9419d0abf
Step 4/16 : ENV USER ${NB_USER}
 ---> Using cache
 ---> ff9fbd4a985d
Step 5/16 : ENV NB_UID ${NB_UID}
 ---> Using cache
 ---> a6d018d0e786
Step 6/16 : ENV HOME /home/${NB_USER}
 ---> Using cache
 ---> 79b483f2bf7c
Step 7/16 : RUN adduser --disabled-password     --gecos "Default user"     --uid 1000 jovyan
 ---> Using cache
 ---> 583f777a1099
Step 8/16 : ENV IPYTHON_STARTUP ${HOME}/.ipython/profile_default/startup/
 ---> Using cache
 ---> e14ac893bf8f
Step 9/16 : RUN mkdir -p $IPYTHON_STARTUP
 ---> Using cache
 ---> 8aad30e264ae
Step 10/16 : RUN echo 'get_ipython().magic(u"%reload_ext sqlflow.magic")' >> $IPYTHON_STARTUP/00-first.py
 ---> Using cache
 ---> 78bfd8fcc52c
Step 11/16 : RUN echo 'get_ipython().magic(u"%autoreload 2")' >> $IPYTHON_STARTUP/00-first.py
 ---> Using cache
 ---> b54618c0f643
Step 12/16 : COPY . ${HOME}
 ---> 8388f457579b
Step 13/16 : RUN chown -R ${NB_UID} ${HOME}
 ---> Running in add71f632064
Removing intermediate container add71f632064
 ---> 88ad41ea983b
Step 14/16 : COPY scripts/entrypoint.sh /entrypoint.sh
 ---> 05bc968064bd
Step 15/16 : RUN chmod +x /entrypoint.sh
 ---> Running in 0e1da52acba4
Removing intermediate container 0e1da52acba4
 ---> 9b5bd1553324
Step 16/16 : ENTRYPOINT ["bash", "/entrypoint.sh"]
 ---> Running in a26512d023f1
Removing intermediate container a26512d023f1
 ---> 916d90079349
{"aux": {"ID": "sha256:916d90079349a68c8a51a0413537bdf3fa1114684595ca343c6da8f766eacaf4"}}Successfully built 916d90079349
Successfully tagged r2dhttps-3a-2f-2fgithub-2ecom-2ftonyyang-2dsvail-2fsqlflow-2ffce19d4:latest
setup all-in-one
 * Starting MySQL database server mysqld
   ...done.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
Connect to the datasource mysql://root:root@tcp(127.0.0.1:3306)/?maxAllowedPacket=0
running command jupyter notebook --ip 0.0.0.0 --port 65045 --NotebookApp.custom_display_url=http://127.0.0.1:65045
2019/10/08 00:40:37 Launch server with insecure mode.
2019/10/08 00:40:37 Server Started at :50051
[I 00:40:38.310 NotebookApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
[I 00:40:38.492 NotebookApp] Serving notebooks from local directory: /home/jovyan
[I 00:40:38.492 NotebookApp] The Jupyter Notebook is running at:
[I 00:40:38.492 NotebookApp] http://127.0.0.1:65045/?token=80feb44e64e1e5fa5c750bab9e442c33d5e33ad9981f5053
[I 00:40:38.492 NotebookApp]  or http://127.0.0.1:65045/?token=80feb44e64e1e5fa5c750bab9e442c33d5e33ad9981f5053
[I 00:40:38.492 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 00:40:38.498 NotebookApp] No web browser found: could not locate runnable browser.
[C 00:40:38.499 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///home/jovyan/.local/share/jupyter/runtime/nbserver-479-open.html
    Or copy and paste one of these URLs:
        http://127.0.0.1:65045/?token=80feb44e64e1e5fa5c750bab9e442c33d5e33ad9981f5053
     or http://127.0.0.1:65045/?token=80feb44e64e1e5fa5c750bab9e442c33d5e33ad9981f5053

Is there any other reason that could cause the problem? For example, I am testing repo2docker on Mac, should I try it on another platform? The built Docker image about 2GB, would it be too large?

[choldgraf]

Hmmm - I wonder if it's an error with the CMD command - I believe that a specific entrypoint needs to exist in order for the thing to run.

Though if this works for you locally, another possibility is that you are trying to run a something that isn't allowed on mybinder.org - the free public service prevents a lot of extra ports/services/etc from being run due to security reasons :-/

[tonyyang-svail]

Hi @choldgraf, thanks for your reply. I am pretty sure I followed the suggestion in the documentation by adding exec $@ at the end of the entrypoint.

Since I am not able to reproduce the error using repo2docker, could you please help on finding more debug info other than Internal Server Error? I really appreciate it.

[choldgraf]

I can try to put some time into it as some point, but I don't have much bandwidth to really dig into to any debugging issues. You might have luck getting input from people by posting in the Jupyter community forum as well (https://discourse.jupyter.org/c/binder)

Briefly looking through your Dockerfile - I'd also play around with removing the tornado pinning and such, I seem to remember that being an issue in the past, and one difference between mybinder.org and your local repo2docker instance could be the default version of jupyter lab / notebook that's on there, and it might be a dependency version mismatch problem

[betatim]

Pulling out the logs from launching your repository I see:

setup all-in-one
sed: couldn't open temporary file /etc/mysql/mysql.conf.d/sed7ukwoO: Permission denied

Being able to stream (or otherwise make available) this log output to users would be a cool feature to have. Unfortunately it is a bit tricky to implement and not requested very often. If you have time and interest to add this it would be welcome!

There are differences between how repo2docker launches a docker container and how BinderHub launches a kubernetes Pod. The best way to debug launch failures when you've replaced the start commands (or try to modify them) is to follow https://github.com/jupyterhub/binderhub/blob/master/CONTRIBUTING.md#kubernetes-integration-changes to setup a local instance. This way you get faster turnarounds and full access to the logs.

---

As a general principle we can't offer much user support for repos that use their own Dockerfile. The recommended way to get a repo to run is to use the config files provided by repo2docker. The reasoning for this is that we assume that if people use a dockerfile they are experts and need to do "weird stuff" that can't be achieved with repo2docker's standard config files. If this isn't the case the recommendation is to use the repo2docker config files. The good thing about using your own Dockerfile is that you can do arbitrarily complex things, the bad thing about using your own Dockerfile is that you can do arbitrarily complex things ;)

This means each Dockerfile case is unique, complex and "one off". Which means offering user support doesn't scale :-/

[manics]

https://mybinder.readthedocs.io/en/latest/tutorials/dockerfile.html

It is bad practice to run processes in containers as root, and on binder we do not allow root container processes.

It looks like your container assumes it's run as root.

[stevengj]

Even without a custom dockerfile, just using the default repo2docker, I'm getting repeated "Launch attempt X failed, retrying..." messages for https://github.com/mitmath/binder-env, which I've been using for years (periodically bumping the version) with no problem. When I tried it this week I could no longer get it to work, but these non-informative error messages make it hard to know what to do differently.