Understanding repo2docker's build context

[jhamman]

Suppose a repository structure like this:

.
├── appendix.txt
├── base-notebook
│   └── binder
│       └── environment.yml
├── config.yaml
└── full-notebook
    └── binder
        ├── environment.yml

where appendix.txt is:

USER root
RUN mkdir -p /etc/config/
ADD ./config.yaml /etc/config/
USER jovyan

From the root of this repository, I call repo2docker as:

repo2docker --user-name=jovyan --appendix="`cat appendix.txt`" ./full-notebook

This fails in the appendix step because the config.yaml is not in Docker's build context.

Step 41/43 : ADD ./config.yaml /etc/config/
ADD failed: stat /var/lib/docker/tmp/docker-builder898774711/config.yaml: no such file or directory%

My questions:

1) what is the root of the docker build in this case? 1) Is it possible to tell repo2docker where the build root should be? 1) The goal here is to have a common appendix/config for all the images to be built from this repo. Is there another way to share assets between images?

xref: https://github.com/pangeo-data/pangeo-stacks/issues/1

[jhamman]

So, I think I've figured out the problem here. There is a chdir in app.py. I'm curious if this is needed? I can certainly see how it makes the path handling logic easier but it does block certain use cases.

https://github.com/jupyter/repo2docker/blob/8f0fc918fc0f3ca06769a86b36f60c8a3f192b18/repo2docker/app.py#L606

[yuvipanda]

@jhamman build context is set here: https://github.com/jupyter/repo2docker/blob/8f0fc918fc0f3ca06769a86b36f60c8a3f192b18/repo2docker/buildpacks/base.py#L493

It looks like stuff is being copied into a 'src' directory in the context - so your ADD should be ADD src/config.yaml /etc/config. That's how it is set up in https://github.com/jupyter/repo2docker/blob/8f0fc918fc0f3ca06769a86b36f60c8a3f192b18/repo2docker/buildpacks/base.py#L118 for example.

I'd recommend finding some other way of doing the 'common' stuff other than Appendix though, if possible.

[jhamman]

I'd recommend finding some other way of doing the 'common' stuff other than Appendix though, if possible.

Do you have thoughts on other ways to do the common stuff? I want each image to be able to use the postBuild script so I can't co-opt that. I also tried using symbolic links but they seem to be broken when the directory is mounted by docker.

[yuvipanda]

@jhamman I think one way is to have a generalPostBuild of sorts that's symlinked in, and require all user postBuilds to source that at the beginning or end. This doesn't work if symlinks are broken - can you open an issue about that?

In the meantime, doing COPY src/config.yaml /etc/dask should work

[yuvipanda]

@jhamman where did you land on this?

[jhamman]

I found that it wasn't possible to have common files in a repository that could be used by multiple repo2docker configurations. I also never managed to get the symlinks to work with docker (the link is copied, not the file itself).