Adding custom XBlocks

[natea]

Where to make changes to repo or add additional XBlocks.

Before I was editing /openedx/Dockerfile and adding one of these two lines:

RUN git fetch origin && git checkout markdown && pip install -r requirements/appsembler.txt
RUN pip install -e git+https://github.com/hastexo/markdown-xblock.git@master#egg=markdown-xblock

Is this the best place to make these changes, or is it enough to have set this as an environment variable:

export EDX_PLATFORM_PATH=/Users/nateaune/Dropbox/code/natea-edx-platform-ginkgo

Which has already has the markdown branch checked out.

It seems like I still need a step to either install my custom requirements.txt file or pip install the custom XBlock that I want to test.

What is the recommended way to do this without having to make local commits to the Dockerfile?

[regisb]

(Ok this is untested but it should work)

I assume you are referring to a custom repo that is publicly available, e.g: https://github.com/appsembler/edx-platform.git The "markdown" branch that you are trying to build should also be available. Finally, the appsembler-specific requirements should be added to a "standard" requirements file, e.g: "github.txt".

You should run:

docker build -t regis/openedx:latest -t regis/openedx:ginkgo --build-arg EDX_PLATFORM_REPOSITORY=https://github.com/appsembler/edx-platform.git --build-arg=EDX_PLATFORM_VERSION=markdown openedx/

Then, all commands should now run with this new version of the container (except the build command, of course).

Unfortunately, there is currently no way to build an image with a local edx-platform version without patching the Dockerfile. But it would be a good idea to implement this feature. We would have to use COPY. I'll leave this issue open to try to think of something.

[natea]

Thanks Regis. I found this to be useful for understanding how build args and runtime args work in Docker: https://blog.manifold.co/arguments-and-variables-in-docker-94746642f64b?source=linkShare-2a51ce69d98e-1528722537

On Sun, Jun 10, 2018 at 11:18 PM Régis B. notifications@github.com wrote:

(Ok this is untested but it should work)

I assume you are referring to a custom repo that is publicly available, e.g: https://github.com/appsembler/edx-platform.git The "markdown" branch that you are trying to build should also be available. Finally, the appsembler-specific requirements should be added to a "standard" requirements file, e.g: "github.txt".

You should run:

docker build -t regis/openedx:latest -t regis/openedx:ginkgo --build-arg EDX_PLATFORM_REPOSITORY=https://github.com/appsembler/edx-platform.git --build-arg=EDX_PLATFORM_VERSION=markdown openedx/

Then, all commands should now run with this new version of the container (except the build command, of course).

Unfortunately, there is currently no way to build an image with a local edx-platform version without patching the Dockerfile. But it would be a good idea to implement this feature. We would have to use COPY https://docs.docker.com/engine/reference/builder/#copy. I'll leave this issue open to try to think of something.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/regisb/openedx-docker/issues/36#issuecomment-396133910, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHQyGkPuGgWxbWWAnlI1v4tMjc_ID0jks5t7gvCgaJpZM4UiBLH .

--

Website: http://www.appsembler.com Blog: http://www.appsembler.com/blog Twitter: http://twitter.com/natea LinkedIn: http://linkedin.com/in/natea Phone: +1 (617) 702-4331

[regisb]

Right. When I started working on this project I was almost completely new to Docker and I had a lot of trouble understanding what were the good practices for settings things up at build time, as opposed to runtime. For instance, I used to believe that volumes could be mounted at build time (they can't).

Maybe we don't need to use a COPY statement: what we want to do is to run an image with custom edx-platform and requirements: it is already possible to use a custom edx-platform, but what we need is requirements that don't need to be updated at every run. To do that, maybe we can mount the python dependencies repository in a volume that would be preserved from one run to the next. I'll try to think about it.

[frob]

I was able to do this by adding the line to the Docker file just under the other pip install lines

RUN pip install --src xblocks/contrib -r requirements/edx/private.txt

This isn't perfect because it requires a couple of other pieces for this to work. One thing is that the build will fail if there isn't a private.txt file. So this will need to be wrapped in an inline bash if -f statement. The second piece is that I am unsure about the location of the src xblocks/contrib. I tried it in the same location as the edx venv source and ran into issues. Also, I have been reading about issues with xblocks not installing properly if they are not in the edx-platform directory. Since themes are in themes I put xblocks in xblocks.

This change would also need to be combined with a make command that compiles the private.in file.

Thoughts?

[Edit]

One more thing I did, I added the xblocks directories to the docker-compose.yml volumes.

    volumes:
      - ./src/openedx-platform/xblocks/custom:/openedx/edx-platform/xblocks/custom
      - ./src/openedx-platform/xblocks/contrib:/openedx/edx-platform/xblocks/contrib

[regisb]

@frob I have a couple comments:

`RUN pip install --src xblocks/contrib -r requirements/edx/private.txt`

Why the strange --src location?

I tried it in the same location as the edx venv source and ran into issues

What were they?

I have been reading about issues with xblocks not installing properly if they are not in the edx-platform directory.

AFAIK this is not true. It's perfectly possible to install xblocks from external sources, such as pypi or github for instance.

This isn't perfect because it requires a couple of other pieces for this to work. One thing is that the build will fail if there isn't a private.txt file. So this will need to be wrapped in an inline bash if -f statement.

Yes, we could write:

RUN (stat requirements/edx/private.txt 2> /dev/null && pip install --src ../venv/src -r requirements/edx/) || true

This would make a lot of sense in the Dockerfile for openedx. The only problem is that private.txt and private.in files are not commited in the edx-platform repo. Maybe I should create an empty, unversioned private.txt file that anyone can modify, and copy it to the container during image building.

This change would also need to be combined with a make command that compiles the private.in file.

I disagree: I think compiling private.in would be the responsibility of the repo maintainer.

One more thing I did, I added the xblocks directories to the docker-compose.yml volumes.

This is surprising: you are aware that volumes are not mounted during image building, right? (see my previous comment)

[frob]

AFAIK this is not true. It's perfectly possible to install xblocks from external sources, such as pypi or github for instance.

I am new to edX, and not supper experienced with python/django. I would expect that you are correct.

I tried it in the same location as the edx venv source and ran into issues

What were they?

Conflicting versions, in paticular with the done xBlock. There is a newer version that I was requesting, and pip was asking for feedback (sometimes) this would cause the build to break. I ended up removing my updated doneXblock requirement all together.

This would make a lot of sense in the Dockerfile for openedx. The only problem is that private.txt and private.in files are not commited in the edx-platform repo. Maybe I should create an empty, unversioned private.txt file that anyone can modify, and copy it to the container during image building.

This change would also need to be combined with a make command that compiles the private.in file.

I disagree: I think compiling private.in would be the responsibility of the repo maintainer.

Which repo, also the way that I was compiling the .in file was to use make lms to log into a Docker shell. My thought was that we add a make command that just does that in the Docker env. Otherwise don't we need to run a local venv to run the pip compile command?

This is surprising: you are aware that volumes are not mounted during image building, right? (see my previous comment)

Yes, this is my own configuration. I like to have the source code to things in the project that I am working in. I added a /src directory to store the edx repo and the theme/xblock repos. I find this help with debugging and contribution.

[regisb]

@frob I'm sorry, I forgot to answer your message from November. I'll close this now, as this issue should be solved by http://docs.tutor.overhang.io/en/latest/customise.html#extra-xblocks-and-requirements As usual, feel free to re-open if the proposed solution does not work for you.