optional parameter for Github path to my-passwords.yml

[lpm0073]

Steps to reproduce

The interview would ideally include an optional parameter for a wget callable path to a pre-existing my-passwords.yml file. If the parameter value is provided then this would cause Docker to skip execution of https://raw.githubusercontent.com/edx/configuration/$OPENEDX_RELEASE/util/install/generate-passwords.sh

Unexpected behavior

none.

Additional info (IMPORTANT)

[regisb]

Hi Lawrence! The way I understand it, importing a my-passwords.yml file is useful for migrating an existing Open edX platform. Is this what youhad in mind? However, in order to migrate an existing Open edX install, there are quite a few steps that are required: importing database content, moving uploaded files, etc. So, for now, migration is not supported out of the box.

Also, you suggest to download a my-passwords.yml file with wget. But such a file is supposed to be secret, right? And, in general, if a file can be downloaded with a simple wget command, then it is available to the general public. There are exceptions to this, of course, but I don't think they are needed by many people.

Currently, tutor does not use the my-passwords.yml file generated by configuration/util/install/generate-passwords.sh. Instad, tutor uses the tutor/config.json file, generated by make config.json. I think it shouldn't be too difficult to write a small python that converts my-passwords.yml to config.json, or at least part of it: many values from my-passwords.yml would not be used in config.json. Does that correspond to what you need?

Maybe I fail to understand your usage scenario. Can you please tell me more about it?

[lpm0073]

passwords management is a big part of production management for a few of my clients who, for various reasons, host multiple installations which are prone to sharing code and updates. when possible i try to use a common “my-passwords.yml” file for each client across all of their installations in order to minimize complexity. the repositories are private, so there’s no risk that the credentials would fall into the wrong hands. however, making repos private does introduce challenges w regard to automating work flows since git commands (and wget) to a private repo require a username and password on the command line. see the attached readme documentation from one of my repos that describes how i setup the ubuntu environment on the target, using oAuth with GitHub as an identity provider to achieve uninterrupted work flows.

i’m somewhat familiar with how the my-passwords.yml and server-vars.yml are added to ansible calls on the command line as i’ve occasionally had to modify the edx scripts that initiate these calls. if you also continue to use ansible wrapped inside your container definitions then re-including these two files would ostensibly be straightforward.

best,

Lawrence McDaniel

---

📡 Full-Stack Developer CV: lawrencemcdaniel.com | linkedin.com/in/lawrencemcdaniel Blog: blog.lawrencemcdaniel.com Code Samples: horrors.lawrencemcdaniel.com | github.com/lpm0073 Contact: lpm0073@gmail.com | USA: +1 (415) 766-9012 | MEX: +52 1 (55) 4388-3070

On Jan 4, 2019, at 03:18, Régis Behmo notifications@github.com wrote:

Hi Lawrence! The way I understand it, importing a my-passwords.yml file is useful for migrating an existing Open edX platform. Is this what youhad in mind? However, in order to migrate an existing Open edX install, there are quite a few steps that are required: importing database content, moving uploaded files, etc. So, for now, migration is not supported out of the box.

Also, you suggest to download a my-passwords.yml file with wget. But such a file is supposed to be secret, right? And, in general, if a file can be downloaded with a simple wget command, then it is available to the general public. There are exceptions to this, of course, but I don't think they are needed by many people.

Currently, tutor does not use the my-passwords.yml file generated by configuration/util/install/generate-passwords.sh. Instad, tutor uses the tutor/config.json file, generated by make config.json. I think it shouldn't be too difficult to write a small python that converts my-passwords.yml to config.json, or at least part of it: many values from my-passwords.yml would not be used in config.json. Does that correspond to what you need?

Maybe I fail to understand your usage scenario. Can you please tell me more about it?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/regisb/tutor/issues/112#issuecomment-451392270, or mute the thread https://github.com/notifications/unsubscribe-auth/ABH8wzjufOQH6VSFspYREFAzULqAAcTGks5u_xxygaJpZM4ZnZFm.

[lpm0073]

here, https://github.com/regisb/tutor/blob/master/build/Makefile https://github.com/regisb/tutor/blob/master/build/Makefile, on rows 7 thru 15 seem to be the equivalent lines of code to integrate my-passwords.yml and server-vars.yml into the build arguments?

On Jan 4, 2019, at 03:18, Régis Behmo notifications@github.com wrote:

Hi Lawrence! The way I understand it, importing a my-passwords.yml file is useful for migrating an existing Open edX platform. Is this what youhad in mind? However, in order to migrate an existing Open edX install, there are quite a few steps that are required: importing database content, moving uploaded files, etc. So, for now, migration is not supported out of the box.

Also, you suggest to download a my-passwords.yml file with wget. But such a file is supposed to be secret, right? And, in general, if a file can be downloaded with a simple wget command, then it is available to the general public. There are exceptions to this, of course, but I don't think they are needed by many people.

Currently, tutor does not use the my-passwords.yml file generated by configuration/util/install/generate-passwords.sh. Instad, tutor uses the tutor/config.json file, generated by make config.json. I think it shouldn't be too difficult to write a small python that converts my-passwords.yml to config.json, or at least part of it: many values from my-passwords.yml would not be used in config.json. Does that correspond to what you need?

Maybe I fail to understand your usage scenario. Can you please tell me more about it?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/regisb/tutor/issues/112#issuecomment-451392270, or mute the thread https://github.com/notifications/unsubscribe-auth/ABH8wzjufOQH6VSFspYREFAzULqAAcTGks5u_xxygaJpZM4ZnZFm.

[regisb]

Hmmm ok I think I better understand your requirement. Do you think you could write a small script that converts my-passwords.yml to config.json? This script would copy some of the values from my-passwords.yml to config.json, and preserve other existing values in config.json. We would call the script like this:

convert-secrets.py /path/to/my-passwords.yml /path/to/config.json

If you are unconfortable with writing this script, I can try and give it a go. I don't expect it's going to be more than ~50 lines of code.

Then, maybe we could add this script to the repo, or add it to the documentation, or create a gist on github.

I think there is some misunderstanding about the use of the Ansible scripts inside Tutor: Tutor does not make any use at all of the Ansible scripts from the configuration repo. Instead, the various Open edX components are installed from scratch, with manual instructions inside the docker containers. So, currently, Tutor does not use the my-passwords.yml file at all: this file is an artefact that is specific to the Ansible scripts.

[lpm0073]

yes, i think i can do that. prior to diving into this i first want to verify that we’re not creating an additional, unnecessary layer of configuration files. i’m reviewing the edx-platform repo right now and i’m unpleasantly surprised to see that there are now in excess of 20 configuration files in ./lms/envs/, which frankly speaking, is absurd.

zooming out for a moment, since rebuilding the containers is a slow process, i believe we want to maintain docker images that represent only the application code and its many subsystems; and NOT any prepackaged configuration data that is prone to being replaced. if a docker image were capable of understanding the concept of “late binding” (maybe they already do???) then in an ideal world your Tutor interview would ask for the following additional parameters:

1. URL to edx-platform repo (the “fork”), with the default value being https://github.com/edx/edx-platform.git https://github.com/edx/edx-platform.git

2. git tag for ex-platform repo, with the default being “master”

3. URL to edx-theme repo, with the default being NULL

4. git tag for edx-theme repo, with the default being “master”

5. URL to edx-config repo, with the default being NULL <—this repo would assume the following defined structure

   • /my-passwords.yml
   • /edx_ansible/server-vars.yml
     • this would not be necessary if your interview could receive a URL to a complete config file for “common.py” + two overrides for “dev.py”, and “prod.py"
   • /edxapp/*.json
     • this would also not be necessary
   • /nginx/sites-available/.
   • /ssh/ <— copies of private keys for oAuth
   • /doc
   • /scripts <— bash scripts for common devops tasks (see attached screen shot for example file structure)

6. git tag for edx-config repo, with the default being “master”

Lawrence McDaniel

---

📡 Full-Stack Developer CV: lawrencemcdaniel.com | linkedin.com/in/lawrencemcdaniel Blog: blog.lawrencemcdaniel.com Code Samples: horrors.lawrencemcdaniel.com | github.com/lpm0073 Contact: lpm0073@gmail.com | USA: +1 (415) 766-9012 | MEX: +52 1 (55) 4388-3070

On Jan 4, 2019, at 11:02, Régis Behmo notifications@github.com wrote:

Hmmm ok I think I better understand your requirement. Do you think you could write a small script that converts my-passwords.yml to config.json? This script would copy some of the values from my-passwords.yml to config.json, and preserve other existing values in config.json. We would call the script like this:

convert-secrets.py /path/to/my-passwords.yml /path/to/config.json If you are unconfortable with writing this script, I can try and give it a go. I don't expect it's going to be more than ~50 lines of code.

Then, maybe we could add this script to the repo, or add it to the documentation, or create a gist on github.

I think there is some misunderstanding about the use of the Ansible scripts inside Tutor: Tutor does not make any use at all of the Ansible scripts from the configuration https://github.com/edx/configuration/ repo. Instead, the various Open edX components are installed from scratch, with manual instructions inside the docker containers. So, currently, Tutor does not use the my-passwords.yml file at all: this file is an artefact that is specific to the Ansible scripts.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/regisb/tutor/issues/112#issuecomment-451503592, or mute the thread https://github.com/notifications/unsubscribe-auth/ABH8wyYVogH56rI7PX6FnqYMnAdm8ralks5u_4kLgaJpZM4ZnZFm.

[lpm0073]

sorry, i hit send by accident!!!! one more:

7. a list of xblock repos

On Jan 4, 2019, at 11:02, Régis Behmo notifications@github.com wrote:

Hmmm ok I think I better understand your requirement. Do you think you could write a small script that converts my-passwords.yml to config.json? This script would copy some of the values from my-passwords.yml to config.json, and preserve other existing values in config.json. We would call the script like this:

convert-secrets.py /path/to/my-passwords.yml /path/to/config.json If you are unconfortable with writing this script, I can try and give it a go. I don't expect it's going to be more than ~50 lines of code.

Then, maybe we could add this script to the repo, or add it to the documentation, or create a gist on github.

I think there is some misunderstanding about the use of the Ansible scripts inside Tutor: Tutor does not make any use at all of the Ansible scripts from the configuration https://github.com/edx/configuration/ repo. Instead, the various Open edX components are installed from scratch, with manual instructions inside the docker containers. So, currently, Tutor does not use the my-passwords.yml file at all: this file is an artefact that is specific to the Ansible scripts.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/regisb/tutor/issues/112#issuecomment-451503592, or mute the thread https://github.com/notifications/unsubscribe-auth/ABH8wyYVogH56rI7PX6FnqYMnAdm8ralks5u_4kLgaJpZM4ZnZFm.

[lpm0073]

this diagram might be helpful. in additional to the parameter values described in the previous two emails, the following illustration shows, for a typical platform, where the other custom code snaps into the default installation.

On Jan 4, 2019, at 11:02, Régis Behmo notifications@github.com wrote:

Hmmm ok I think I better understand your requirement. Do you think you could write a small script that converts my-passwords.yml to config.json? This script would copy some of the values from my-passwords.yml to config.json, and preserve other existing values in config.json. We would call the script like this:

convert-secrets.py /path/to/my-passwords.yml /path/to/config.json If you are unconfortable with writing this script, I can try and give it a go. I don't expect it's going to be more than ~50 lines of code.

Then, maybe we could add this script to the repo, or add it to the documentation, or create a gist on github.

I think there is some misunderstanding about the use of the Ansible scripts inside Tutor: Tutor does not make any use at all of the Ansible scripts from the configuration https://github.com/edx/configuration/ repo. Instead, the various Open edX components are installed from scratch, with manual instructions inside the docker containers. So, currently, Tutor does not use the my-passwords.yml file at all: this file is an artefact that is specific to the Ansible scripts.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/regisb/tutor/issues/112#issuecomment-451503592, or mute the thread https://github.com/notifications/unsubscribe-auth/ABH8wyYVogH56rI7PX6FnqYMnAdm8ralks5u_4kLgaJpZM4ZnZFm.

[regisb]

@lpm0073 note that files you attach by email do not get associated to this issue :-/ I'm afraid you need to use the Github user interface to upload screenshots.

I think there is some confusion with how the config.json file is used. First of all, when you modify the config.json file (with make configure for instance) then there is no need to re-build the docker images. The config.json file is used to generate the various "environment" files, such as: docker-compose.yml, the lms/cms settings file (.py, .env.json, *.auth.json), the database migration scripts, etc.

Basically configuration is cleanly separated from the containers. Configuration files are mounted inside the running containers at runtime.

Also, I think we should be careful not to make the interactive configuration step too complex. Normal users will not want to answer complex questions in order to have a heavily customised platform.

This being said, let's review the additional parameters you mentioned.

1. URL to edx-platform repo (the “fork”), with the default value being https://github.com/edx/edx-platform.git
2. git tag for ex-platform repo, with the default being “master”

The edx-platform repo and tag are not configuration parameters. They are not used at runtime, but at build time. This is how they should be overridden: http://docs.tutor.overhang.io/en/latest/customise.html#forked-version-of-edx-platform

3. URL to edx-theme repo, with the default being NULL
4. git tag for edx-theme repo, with the default being “master”

Same as for edx-platform, this is not a configuration parameter, but some data that is used at build time.

I had a long conversation with another Tutor user about how best to integrate custom themes. My personal conclusion was that there are many different ways to checkout a custom theme, and we should not make a choice for the user. Some users will have a github repo, for others it's going to be a private repo with authentication, or even a local folder stored somewhere on disk.

So we make the smallest possible assumption: that the themes are already checked out in build/openedx/themes. See this part of the documentation. I think it's a safe assumption that works for most people. So basically, as a theme maintainer and Open edX administrator, we consider that it's your job to get the right themes in the right folder at build time.

5. URL to edx-config repo, with the default being NULL <—this repo would assume the following defined structure ...
6. git tag for edx-config repo, with the default being “master”

I think it's too much to expect that every user will have an edx-config repository that we can easily checkout (i.e: with no authentication) and with the proper folder structure. Instead, the make configure (or make env) command generates an env/ folder that will be used at runtime to configure the running apps. This folder is not under version control. You are free to modify the files in this folder as you wish. In particular, this folder contains the nginx configuration files, the edxapp/*.json files, a few convenient scripts, etc.

7. a list of xblock repos

Again, when we discussed how best to implement custom xblocks, my conclusion was that there were too many different ways to checkout custom xblocks. So we made the same lightweight assumption as for themes: it is the administrator's responsibility to checkout the xblock code, as per the documentation. Again, this is not a parameter that we want to bother the user with, because:

a. Most users don't want to bother with custom xblocks every time they run make configure. b. It's not part of the runtime configuration, but of the build configuration.

I feel like there is some misunderstanding about how Tutor works. On the other hand, I don't want to create any frustration by rejecting all your propositions. Maybe we can schedule a call so that I can give you a full description of how tutor works, with some interactive Q&A?

[lpm0073]

thanks régis. no, no misunderstanding. each of your responses below assumes that i’m going to rebuild the container, but that is a slow process. in a team-based development environment the platform, theme and config repos change frequently, and as i’ve mentioned, i routinely have to rebuild app servers and/or create variations of these for special use cases. adding a serialized 1-hour container build process to the work flow won’t work.

what i meant by “late binding” is: would it be possible to parameterize these 7 items in such a way that it is NOT necessary to rebuild your containers?

one way might be for the tutor use case to be strictly limited to the 1-button installation that you currently provide, and then all of the configuration is done afterwards using a combination of ansible and bash scripts, as i’ve always done. this would still be a big step forward given that your containers would replace the 90-minute ansible native installation process, and your tutor installation process would presumably be much more reliable. i’m happy test this scenario, but i’ll need a better understanding of where the edx platform files are ultimately being physically stored on destination machine (one of my original questions to you).

Lawrence McDaniel

---

📡 Full-Stack Developer CV: lawrencemcdaniel.com | linkedin.com/in/lawrencemcdaniel Blog: blog.lawrencemcdaniel.com Code Samples: horrors.lawrencemcdaniel.com | github.com/lpm0073 Contact: lpm0073@gmail.com | USA: +1 (415) 766-9012 | MEX: +52 1 (55) 4388-3070

On Jan 4, 2019, at 12:09, Régis Behmo notifications@github.com wrote:

@lpm0073 https://github.com/lpm0073 note that files you attach by email do not get associated to this issue :-/ I'm afraid you need to use the Github user interface to upload screenshots.

I think there is some confusion with how the config.json file is used. First of all, when you modify the config.json file (with make configure for instance) then there is no need to re-build the docker images. The config.json file is used to generate the various "environment" files, such as: docker-compose.yml, the lms/cms settings file (.py, .env.json, *.auth.json), the database migration scripts, etc.

Basically configuration is cleanly separated from the containers. Configuration files are mounted inside the running containers at runtime.

Also, I think we should be careful not to make the interactive configuration step too complex. Normal users will not want to answer complex questions in order to have a heavily customised platform.

This being said, let's review the additional parameters you mentioned.

URL to edx-platform repo (the “fork”), with the default value being https://github.com/edx/edx-platform.git https://github.com/edx/edx-platform.git git tag for ex-platform repo, with the default being “master” The edx-platform repo and tag are not configuration parameters. They are not used at runtime, but at build time. This is how they should be overridden: http://docs.tutor.overhang.io/en/latest/customise.html#forked-version-of-edx-platform http://docs.tutor.overhang.io/en/latest/customise.html#forked-version-of-edx-platform URL to edx-theme repo, with the default being NULL git tag for edx-theme repo, with the default being “master” Same as for edx-platform, this is not a configuration parameter, but some data that is used at build time.

I had a long conversation with another Tutor user about how best to integrate custom themes. My personal conclusion was that there are many different ways to checkout a custom theme, and we should not make a choice for the user. Some users will have a github repo, for others it's going to be a private repo with authentication, or even a local folder stored somewhere on disk.

So we make the smallest possible assumption: that the themes are already checked out in build/openedx/themes. See this part of the documentation http://docs.tutor.overhang.io/en/latest/customise.html#custom-themes. I think it's a safe assumption that works for most people. So basically, as a theme maintainer and Open edX administrator, we consider that it's your job to get the right themes in the right folder at build time.

URL to edx-config repo, with the default being NULL <—this repo would assume the following defined structure ... git tag for edx-config repo, with the default being “master” I think it's too much to expect that every user will have an edx-config repository that we can easily checkout (i.e: with no authentication) and with the proper folder structure. Instead, the make configure (or make env) command generates an env/ folder that will be used at runtime to configure the running apps. This folder is not under version control. You are free to modify the files in this folder as you wish. In particular, this folder contains the nginx configuration files, the edxapp/*.json files, a few convenient scripts, etc.

a list of xblock repos Again, when we discussed how best to implement custom xblocks, my conclusion was that there were too many different ways to checkout custom xblocks. So we made the same lightweight assumption as for themes: it is the administrator's responsibility to checkout the xblock code, as per the documentation http://docs.tutor.overhang.io/en/latest/customise.html#extra-xblocks-and-requirements. Again, this is not a parameter that we want to bother the user with, because:

a. Most users don't want to bother with custom xblocks every time they run make configure. b. It's not part of the runtime configuration, but of the build configuration.

I feel like there is some misunderstanding about how Tutor works. On the other hand, I don't want to create any frustration by rejecting all your propositions. Maybe we can schedule a call so that I can give you a full description of how tutor works, with some interactive Q&A?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/regisb/tutor/issues/112#issuecomment-451522523, or mute the thread https://github.com/notifications/unsubscribe-auth/ABH8wyAjRHj822_I36UX2fWw0qiUFv4Pks5u_5jbgaJpZM4ZnZFm.

[lpm0073]

régis, your message below suggests that config.json is part of the overall docker architecture? ie is it something that i should spend time to study in the docker documentation this weekend?

On Jan 4, 2019, at 12:09, Régis Behmo notifications@github.com wrote:

@lpm0073 https://github.com/lpm0073 note that files you attach by email do not get associated to this issue :-/ I'm afraid you need to use the Github user interface to upload screenshots.

I think there is some confusion with how the config.json file is used. First of all, when you modify the config.json file (with make configure for instance) then there is no need to re-build the docker images. The config.json file is used to generate the various "environment" files, such as: docker-compose.yml, the lms/cms settings file (.py, .env.json, *.auth.json), the database migration scripts, etc.

Basically configuration is cleanly separated from the containers. Configuration files are mounted inside the running containers at runtime.

Also, I think we should be careful not to make the interactive configuration step too complex. Normal users will not want to answer complex questions in order to have a heavily customised platform.

This being said, let's review the additional parameters you mentioned.

URL to edx-platform repo (the “fork”), with the default value being https://github.com/edx/edx-platform.git https://github.com/edx/edx-platform.git git tag for ex-platform repo, with the default being “master” The edx-platform repo and tag are not configuration parameters. They are not used at runtime, but at build time. This is how they should be overridden: http://docs.tutor.overhang.io/en/latest/customise.html#forked-version-of-edx-platform http://docs.tutor.overhang.io/en/latest/customise.html#forked-version-of-edx-platform URL to edx-theme repo, with the default being NULL git tag for edx-theme repo, with the default being “master” Same as for edx-platform, this is not a configuration parameter, but some data that is used at build time.

I had a long conversation with another Tutor user about how best to integrate custom themes. My personal conclusion was that there are many different ways to checkout a custom theme, and we should not make a choice for the user. Some users will have a github repo, for others it's going to be a private repo with authentication, or even a local folder stored somewhere on disk.

So we make the smallest possible assumption: that the themes are already checked out in build/openedx/themes. See this part of the documentation http://docs.tutor.overhang.io/en/latest/customise.html#custom-themes. I think it's a safe assumption that works for most people. So basically, as a theme maintainer and Open edX administrator, we consider that it's your job to get the right themes in the right folder at build time.

URL to edx-config repo, with the default being NULL <—this repo would assume the following defined structure ... git tag for edx-config repo, with the default being “master” I think it's too much to expect that every user will have an edx-config repository that we can easily checkout (i.e: with no authentication) and with the proper folder structure. Instead, the make configure (or make env) command generates an env/ folder that will be used at runtime to configure the running apps. This folder is not under version control. You are free to modify the files in this folder as you wish. In particular, this folder contains the nginx configuration files, the edxapp/*.json files, a few convenient scripts, etc.

a list of xblock repos Again, when we discussed how best to implement custom xblocks, my conclusion was that there were too many different ways to checkout custom xblocks. So we made the same lightweight assumption as for themes: it is the administrator's responsibility to checkout the xblock code, as per the documentation http://docs.tutor.overhang.io/en/latest/customise.html#extra-xblocks-and-requirements. Again, this is not a parameter that we want to bother the user with, because:

a. Most users don't want to bother with custom xblocks every time they run make configure. b. It's not part of the runtime configuration, but of the build configuration.

I feel like there is some misunderstanding about how Tutor works. On the other hand, I don't want to create any frustration by rejecting all your propositions. Maybe we can schedule a call so that I can give you a full description of how tutor works, with some interactive Q&A?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/regisb/tutor/issues/112#issuecomment-451522523, or mute the thread https://github.com/notifications/unsubscribe-auth/ABH8wyAjRHj822_I36UX2fWw0qiUFv4Pks5u_5jbgaJpZM4ZnZFm.

[regisb]

No, the config.json file is purely related to Tutor. It's the mechanism that tutor employs to store configuration variables, which in turn will be used to render the environment templates. I encourage you to take a look at the make substitute command: https://github.com/regisb/tutor/blob/master/Makefile#L19

[regisb]

I think I am starting to better understand your problem. The main question is not "how to import already existing passwords from my-passwords.yml?" (that's only a part of the problem), but "how to quickly build docker images from forked edx-platform repos?".

There is a short, pessimistic answer to that question, and there is a longer, optimistic answer.

"No, we can't build the openedx image faster"

Building the openedx docker image takes about 20 minutes on a server with good bandwidth. That's a rough estimate, I did not realise a serious benchmark. Travis.ci needs ~45 minutes to build and upload all the docker images.

Practically all of this time is taken by two different steps:

1. Install requirements (apt, pip, npm)
2. Generate and collect assets

These two steps cannot be accelerated. And of course, when we checkout a fork of edx-platform, then the assets and the requirements of this repo may change. Thus, when we checkout an edx-platform fork, we need to run steps 1 and 2. Ergo it's not possible to make the build any faster.

"Yes, we can build a new openedx image pretty quickly"

If, by some chance, we checkout a fork of edx-platform for which assets and requirements are the same as the base release, then there is no need to run steps 1 and 2 again! Instead, we can create a new docker image that inherits the base image and modifies the repo origin and version.

Dockerfile.fork:

FROM regis/openedx:latest

ARG EDX_PLATFORM_REPOSITORY
ARG EDX_PLATFORM_VERSION
RUN git remote add fork $EDX_PLATFORM_REPOSITORY \
    && git fetch fork \
    && git checkout $EDX_PLATFORM_VERSION

You can then build this docker image with your custom repo and version, and assign it a custom tag:

docker build -t lawrence/myimage:latest --build-arg="EDX_PLATFORM_REPOSITORY=https://github.com/lawrence/edx-platform.git" --build-arg=" EDX_PLATFORM_VERSION=mytag" Dockerfile.fork

And then you can deploy this image instead of the original image, as per the documentation.

What do you think?

[lpm0073]

thank you régis, this is enormously helpful information. yes, i think the work flow you describe below works. for many reasons, it would be beneficial if there were a “community supported” docker image for the “current” platform version (for example, hawthorn.master)

if i understand everything correctly then in your point #2 below, “build this docker image with your custom repo and version, and assign it a custom tag”, i should also do the following:

1. specify the URL to the custom theme
2. add Xblocks to the private requirements.txt

then, to confirm: to deploy this image i would only need to run the following: /regisb/tutor/deploy/local/Makefile all as this would take care of the ssl certificate issuance, pulling my most recently committed edx-platform fork, db migrations, pulling my most recently committed theme repo, static asset compilation, and daemonizing the services on the local server.

to confirm, this still leaves pending how to customize edx-plaform parameters currently managed with server-vars.yml and the json files in /edx/app/edxapp/. but it sounds like we could potentially integrate something into config.json, yes?

• please confirm that this much of my understanding is correct and then afterwards i’ll create a first draft of a visual workflow diagram with additional details.

Lawrence McDaniel

---

📡 Full-Stack Developer CV: lawrencemcdaniel.com | linkedin.com/in/lawrencemcdaniel Blog: blog.lawrencemcdaniel.com Code Samples: horrors.lawrencemcdaniel.com | github.com/lpm0073 Contact: lpm0073@gmail.com | USA: +1 (415) 766-9012 | MEX: +52 1 (55) 4388-3070

On Jan 5, 2019, at 06:02, Régis Behmo notifications@github.com wrote:

I think I am starting to better understand your problem. The main question is not "how to import already existing passwords from my-passwords.yml?" (that's only a part of the problem), but "how to quickly build docker images from forked edx-platform repos?".

There is a short, pessimistic answer to that question, and there is a longer, optimistic answer.

"No, we can't build the openedx image faster"

Building the openedx docker image takes about XXX on a server with good bandwidth. That's a rough estimate, I did not realise a serious benchmark. Travis.ci needs ~45 minutes to build and upload all the docker images.

Practically all of this time is taken by two different steps:

Install requirements (apt, pip, npm) Generate and collect assets These two steps cannot be accelerated. And of course, when we checkout a fork of edx-platform, then the assets and the requirements of this repo may change. Thus, when we checkout an edx-platform fork, we need to run steps 1 and 2. Ergo it's not possible to make the build any faster.

"Yes, we can build a new openedx image pretty quickly"

If, by some chance, we checkout a fork of edx-platform for which assets and requirements are the same as the base release, then there is no need to run steps 1 and 2 again! Instead, we can create a new docker image that inherits the base image and modifies the repo origin and version.

Dockerfile.fork:

FROM regis/openedx:latest

ARG EDX_PLATFORM_REPOSITORY ARG EDX_PLATFORM_VERSION RUN git remote add fork $EDX_PLATFORM_REPOSITORY \ && git fetch fork \ && git checkout $EDX_PLATFORM_VERSION You can then build this docker image with your custom repo and version, and assign it a custom tag:

docker build -t lawrence/myimage:latest --build-arg="EDX_PLATFORM_REPOSITORY=https://github.com/lawrence/edx-platform.git" --build-arg=" EDX_PLATFORM_VERSION=mytag" Dockerfile.fork And then you can deploy this image instead of the original image, as per the documentation http://docs.tutor.overhang.io/en/latest/customise.html#running-a-different-openedx-docker-image.

What do you think?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/regisb/tutor/issues/112#issuecomment-451650016, or mute the thread https://github.com/notifications/unsubscribe-auth/ABH8wxaQlXkTWMiG-Ri6gpiLn5EHPfWoks5vAJRWgaJpZM4ZnZFm.

[regisb]

Lawrence, you are almost completely right. Regarding the theme, you should not specify its url, but check it out directly to build/openedx/themes before building the openedx image.

Also, there is already a "community-supported" image for Open edX Hawthorn: it's the image that is downloaded when you run make update. This image is located here: https://hub.docker.com/r/regis/openedx Just to be clear: when running Tutor, you have a choice between building the image yourself or downloading it. By default, when you run make all, the image is downloaded: it's usually faster for servers with good bandwidth. That image was built with the default theme and no custom xblock.

[lpm0073]

that makes even better sense. ok then. it sounds like we have our deployment plan. i’m ready to do a 2nd video work session whenever convenient for you.

Lawrence McDaniel 📡 Full-Stack Developer & Business Analyst CV: lawrencemcdaniel.com | linkedin.com/in/lawrencemcdaniel Code Samples: horrors.lawrencemcdaniel.com | github.com/lpm0073 Contact: lpm0073@gmail.com | USA: +1 (415) 766-9012 | MEX: +52 1 (55) 4388-3070 Sent from my iPhone

On Jan 6, 2019, at 5:18 AM, Régis Behmo notifications@github.com wrote:

Lawrence, you are almost completely right. Regarding the theme, you should not specify its url, but check it out directly to build/openedx/themes before building the openedx image.

Also, there is already a "community-supported" image for Open edX Hawthorn: it's the image that is downloaded when you run make update. This image is located here: https://hub.docker.com/r/regis/openedx Just to be clear: when running Tutor, you have a choice between building the image yourself or downloading it. By default, when you run make all, the image is downloaded: it's usually faster for servers with good bandwidth. That image was built with the default theme and no custom xblock.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[regisb]

Ok, so I'll close this now. Feel free to re-open if you think some changes are needed in Tutor.