Project has become cumbersome

[frob]

With all the recent changes, this project has become cumbersome and difficult to work with. I feel as though it has lost its way or purpose.

I really like this project and I have contributed to it as often as I am able. But I think it may be reaching too far too quickly. I believe it needs to roll-back or redefine some recent changes.

Steps to reproduce

Try to do things step by step according to the documentation.

Unexpected behavior

When following the quickstart guide the site will come up, anything more than that requires more intimate knowledge of openedx, docker, and tutor than should be required. I find myself constantly turn to to grep to search for the correct Makefile to invoke for a command --and even if I find it I don't always know if it is the correct command to run.

It is no longer apparent where the configuration is located or what config files I should be editing.

Additional info (IMPORTANT)

Include the output of the make info command.

07:42 $ make info
make: *** No rule to make target `info'.  Stop.

The make info command doesn't work in the root of the project.

[regisb]

Hey @frob, Thanks for taking the time to write this! You are one of the early adopters of tutor and I value your opinion very much.

I sympathize with your feeling that the project has become much more complex. One the goals of this project is to keep it simple, so that people actually understand what is going on when they deploy Open edX. If that goal is not achieved, then we need to re-think the project architecture.

Let's try to clarify why tutor has evolved in this more complex machinery where we need multiple makefiles to get a platform working. I was working on deploying Open edX on Kubernetes and a lot of the code was being duplicated. Pretty much all of the configuration and environment code, actually. So, in good DNRY spirit, I split the deployment code in two parts: local, for local deployment on a single server with docker-compose, and k8s, for deployment on an existing Kubernetes cluster. This made me realise that the code to build the images was cleanly separated from both local and k8s modules, so I moved the image-building code to a dedicated folder.

This explains why the code is now split in the following file hierarchy:

build/
deploy/
    local/
    k8s/

Now, my feeling is that much of the code complexity is due to the fact that the bulk of the codebase is written inside Makefiles. Would you agree with that? Sometimes I feel that tutor would be much more simple if everything was packaged inside a nice little executable that would take care of everything. But then:

1) There are many projects which have attempted to improve on Make and have failed. 2) Understanding the code would require looking at the code from inside the executable file.

So this is where I'm standing right now: I agree with you that tutor would require a more simple architecture, but I'm trying to balance features and simplicity as best as I can.

The way I see things, a tutor executable would have a command line interface similar to:

tutor build openedx --themes=./themes --requirements=./requirements.txt
tutor build forum

tutor run-local ./myplatformconfig.yml
tutor run-k8s ./myplatformconfig.yml

With this approach, we could have many features (custom image names, docker repositories, etc) all configurable in a single yml file. All the complexity would be hidden inside the tutor executable code, but it would not make the CLI any more complicated.

What do you think? @silviot @lpm0073 @natea @roblhibbard I'm curious to have your opinion as well.

[frob]

K8s is the logical next step, and is where I was going to be working toward in the coming months. I just think that there are some other things that we can/should focus on first.

@regisb I understand the reason for breaking up the deployments and mostly prefer the new directory structure over the old one --with one exception: Can we bring back config?

config/
build/
deploy/
    local/
    k8s/

I think that would make this project so much easier to use. I find it really difficult to hunt down configuration after migrating. I wouldn't expect k8s to use different configuration than the local/dev deployments, but I haven't really tried yet (I was working more on the stability of the existing deployment before moving onto Kubernetes).

The way I see things, a tutor executable would have a command line interface similar to

I agree, I have had some good results with using click to build cli. However, I think it should be a separate project. The tutor-cli should also have a goal of being completely portable with a brew/pip based dependency free install --which is no small task in-itself. For a good example, checkout the AWS SAM-CLI.

In the mean-time. It would be good to contextualize the Makefile command to their purpose based on the documentation. The docs are organized pretty well, but following them often means changing directories multiple times, depending on what one is doing.

As it is, I need to continue my work on backups, migrations, and local deployment, in my old fork to keep my projects moving and I wont be able to contribute code here for a while --but I will continue to be vocal in the issue queue while we work toward the the stated goal:

Keep it simple, so that people actually understand what is going on when they deploy Open edX.

[regisb]

K8s is the logical next step, and is where I was going to be working toward in the coming months. I just think that there are some other things that we can/should focus on first.

Are you talking about additional features, like backups and image name customization? I agree that we need those features, but I think we can move forward in parallel on both fronts. You see, I'm a bit hurried by the Open edX conference, in March, where I'd like to present a stable tutor interface that works well with Kubernetes.

Can we bring back config?

Two things happened to the config/ folder: 1) It was renamed to env/. This helped to make the difference between generating the config.json file (make config.json) and rendering the templates (make env). 2) It was moved to deploy/ (well, to deploy/env). The environment is inside the deploy/ folder because it is used by both local and k8s subfolders, so it makes sense to have it at the common root. However, env is not used from inside the build folder, so it's no use having it at the project root.

I think that would make this project so much easier to use. I find it really difficult to hunt down configuration after migrating. I wouldn't expect k8s to use different configuration than the local/dev deployments, but I haven't really tried yet (I was working more on the stability of the existing deployment before moving onto Kubernetes).

You're right: the k8s deployment uses the exact same environment as the local one.

I agree, I have had some good results with using click to build cli. However, I think it should be a separate project.

Again, I want to try to keep the project as simple as possible, so I don't want people to checkout two different projects. Also, it's really a pain to keep both projects synchronised through version numbers. So what I think I'll do is have a src/ folder at the root of the project, which will contain the source code of the tutor binary.

The tutor-cli should also have a goal of being completely portable with a brew/pip based dependency free install --which is no small task in-itself. For a good example, checkout the AWS SAM-CLI.

I want to have a cross-platform, single-file executable that people can just download and run. It's possible to create a single-file executable in Python, with pyinstaller, but it's not going to be cross-platform. So I think I'll go with either Go or Rust. I understand it makes the code harder to read for Python developers who might not know Go or Rust yet (I'm in this category myself), but the cross-platform requirement is important for me.

EDIT: true cross-platform binaries don't exist in go or rust, in the sense that we are not going to have a single binary that works everywhere, but I'd like to be able to generate binaries for other platforms from my linux computer.

In the mean-time. It would be good to contextualize the Makefile command to their purpose based on the documentation. The docs are organized pretty well, but following them often means changing directories multiple times, depending on what one is doing.

I'm not sure what you mean by that. Can you give an example of contextualization?

I will continue to be vocal in the issue queue while we work toward the the stated goal

please do!

[frob]

In the mean-time. It would be good to contextualize the Makefile command to their purpose based on the documentation. The docs are organized pretty well, but following them often means changing directories multiple times, depending on what one is doing.

I'm not sure what you mean by that. Can you give an example of contextualization?

Group Makefile commands based on the docs, rather than the directory they happen to be in.

As an example, in order to follow the docs for implementing comprehensive theming one must find commands across two directories that are not nested together.

Contextualizing the Makefile commands would mean that in a single directory, where theme development might happen; maybe the build/openedx/themes directory, we could group all theme/asset functionality.

[regisb]

You're right, the theme building required the assets command, but I've been thinking that the assets command is not really useful. So I took this opportunity to get rid of it and thus simplify the theme building process. Instead of running make assets, the assets-copying step is performed directly by docker-compose. See https://github.com/regisb/tutor/commit/772e8257c37bda6ea9a550d771a4b06efcaa954f

[frob]

I was thinking more along the lines of adding a Makefile to the build/openedx/themes directory with all the asset commands.

assets: ## builds assets and copies them over to nginx
...

assets-watch: ## automatically builds the theme when changes happen. Used for development.
...

generate-theme: ## automatically clones a barebones theme into theme directory
...

Something like this to aid in the task of theme development. Much like putting all the build commands in the build directory would make it easier to manage build and images.

[regisb]

assets: ## builds assets and copies them over to nginx

This command is now unnecessary.

assets-watch: ## automatically builds the theme when changes happen. Used for development.

We are not actually re-building the image every time the theme changes. We are only compiling development assets. Thus I think it makes sense to keep this in the deploy/local folder.

generate-theme: ## automatically clones a barebones theme into theme directory

Unfortunately, there is no unique "app store-like" source of themes from which we could simply download multiple themes. Also, there are many different ways to checkout a theme: from a remote repository, with/without credentials, from a subfolder, from an svn repo (god forbids), download from a .tar.gz at a given url, etc. So I think it makes sense to leave it to the user to checkout the theme inside the right folder.

Most of the work should happen inside the deploy/local folder. And then, once the theme has been finalized, the build-openedx command should be executed in the build folder.

But then again, everything is going to get much simpler once we have a single tutor binary for all commands.

[frob]

I know there is no unique app store-like source of themes. I was thinking more of a skeleton or cookie cutter (if there is one) OpenEdx theme being generated (or cloned) in the correct location and all necessary image rebuilding being done.

To be honest, I haven't gotten the new theme stuff to work at all yet, but I haven't had a chance to try super hard. I have my own deadlines coming up really soon.

Most of the work should happen inside the deploy/local folder. And then, once the theme has been finalized, the build-openedx command should be executed in the build folder.

That is kind of counter intuitive, for a front-end dev to be running commands out-side from where the theme is being developed might be bit of a hurdle.

This is what I mean by context, right now you are developing with the persona of someone doing ops. You need to put yourself in the mindset of the person doing front-end work and think about what will make that job easier when designing the commands for front-end development.

Think about the personas of the users of Tutor. These are the things that I could see Tutor being used for:

• front-end dev
• ops/deployment management
• xblock/lti development & testing
• Core Open Edx Development

[regisb]

@frob can you try to work on a theme and tell me where the pain points are? I tried to make it easy for a theme designer to work on a theme without having to rebuild the entire image, or even recompile all assets. You'll see that assets re-compilation (with watch-themes) is much faster than with the native update_assets command.

[frob]

Can you describe the expected workflow for a theme designer? I cannot figure it out based on the documentation. There was an issue where you described how the containers dealt with assets but didn't go into great detail on the workflow for someone creating a theme.

[regisb]

That's a first pain point :) Indeed, I initially thought that the documentation was better on this topic, but I now realize that there is no clear indication on how to proceed for developing a theme.

[regisb]

There you go: http://docs.tutor.overhang.io/en/latest/dev.html#customised-themes

[regisb]

There you go: http://docs.tutor.overhang.io/en/latest/dev.html#customised-themes

[natea]

@frob already mentioned Click for writing CLI tools, but these two other Python tools for writing a CLI might be useful: fire and argparse (part of the Python standard library) https://hackernoon.com/tools-for-writing-python-cli-applications-ba52db1e454f

[roblhibbard]

sorry that I took so long to reply, things have been busy for me lately. I like what you have done with the platform. You have been very helpful to me on all steps of the way, especially since I am fairly new at this. I am currently working on getting the Jupyter xblock added in. found some useful tidbits from another on how to implement. So I have been focused on that. I can get this xblock to work on a brand new install and only using the demo class. Once I import my previous databases if fails.

thank you for all your hard work,

Robert H

[frob]

@roblhibbard you should open a new issue about that topic specifically. Please don't change the topic of topics other people have opened.

[regisb]

Hey everyone! I finally completed the rewrite of tutor. It took much longer than I expected, but I took the opportunity to add a couple new features. For instance, there is now a web UI for remote administration: http://docs.tutor.overhang.io/en/latest/webui.html

Tutor can now be installed in different ways: http://docs.tutor.overhang.io/en/latest/install.html I ended up choosing python. The binaries are bundles with pyinstaller (https://pyinstaller.readthedocs.io/en/stable/). It works well, although the resulting binaries are bigger than expected (~33Mb). I followed your advice and used click (http://click.palletsprojects.com/), which is awesome.

You will have to re-learn most commands, I'm afraid. It's the cost to pay for a cleaner, easier to maintain CLI. Please let me know what you think!

[natea]

Hi Regis, Thanks for the update! Great to hear that you've made so much progress. I just tried downloading the Mac binaries, and it's not clear from the file that is downloaded, what one is supposed to do with it. It doesn't have a .pkg or .dmg extension, so I can't just click on it to initiate a normal MacOSX install procedure. Do I need to run something on the command line? Nate

[regisb]

@natea it's a single-file binary, so you can put it somewhere in your PATH or run it directly. Would it make more sense for Mac users to get a dmg file?

[lpm0073]

hi regis, yes, installing via a dmg image would be the more normal work flow.

Lawrence McDaniel

---

📡 Full-Stack Developer CV: lawrencemcdaniel.com | linkedin.com/in/lawrencemcdaniel Blog: blog.lawrencemcdaniel.com Code Samples: horrors.lawrencemcdaniel.com | github.com/lpm0073 Contact: lpm0073@gmail.com | USA: +1 (415) 766-9012 | MEX: +52 1 (55) 4388-3070

On Feb 10, 2019, at 00:32, Régis Behmo notifications@github.com wrote:

@natea https://github.com/natea it's a single-file binary, so you can put it somewhere in your PATH or run it directly. Would it make more sense for Mac users to get a dmg file?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/regisb/tutor/issues/121#issuecomment-462107912, or mute the thread https://github.com/notifications/unsubscribe-auth/ABH8w0313A_8pNQut9PdHU0OJoJTcTbDks5vL7zygaJpZM4Z7vMW.

[regisb]

Do you guys have any idea how to generate dmg files? The mac world is foreign to me.

[lpm0073]

régis, i’ll volunteer to research this. pls give me until tuesday (at the latest).

Lawrence McDaniel 📡 Full-Stack Developer & Business Analyst CV: lawrencemcdaniel.com | linkedin.com/in/lawrencemcdaniel Code Samples: horrors.lawrencemcdaniel.com | github.com/lpm0073 Contact: lpm0073@gmail.com | USA: +1 (415) 766-9012 | MEX: +52 1 (55) 4388-3070 Sent from my iPhone

On Feb 10, 2019, at 10:14 AM, Régis Behmo notifications@github.com wrote:

Do you guys have any idea how to generate dmg files? The mac world is foreign to me.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[frob]

@regisb Here are some useful links. I would also recommend supporting homebrew for installation of any binary.

For generating mac apps from python, this is the goto example project. https://py2app.readthedocs.io/en/latest/

For windows there is p2exe which does the same thing for windows. https://pypi.org/project/py2exe/

[lpm0073]

good call frank. i’ll also look into how to setup a package for homebrew.

Lawrence McDaniel

---

📡 Full-Stack Developer CV: lawrencemcdaniel.com | linkedin.com/in/lawrencemcdaniel Blog: blog.lawrencemcdaniel.com Code Samples: horrors.lawrencemcdaniel.com | github.com/lpm0073 Contact: lpm0073@gmail.com | USA: +1 (415) 766-9012 | MEX: +52 1 (55) 4388-3070

On Feb 11, 2019, at 09:09, Frank Anderson notifications@github.com wrote:

@regisb https://github.com/regisb Here are some useful links. I would also recommend supporting homebrew for installation of any binary.

For generating mac apps from python, this is the goto example project. https://py2app.readthedocs.io/en/latest/ https://py2app.readthedocs.io/en/latest/ For windows there is p2exe which does the same thing for windows. https://pypi.org/project/py2exe/ https://pypi.org/project/py2exe/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/regisb/tutor/issues/121#issuecomment-462362178, or mute the thread https://github.com/notifications/unsubscribe-auth/ABH8w5c6KaO4fnfveUumzN4iHDRBJNtpks5vMYfGgaJpZM4Z7vMW.

[frob]

We should reopen this issue or start a new one about all this.

[regisb]

168