Initial development of installer

martinburchell commented 2 years ago

@RudolfCardinal I think this is ready for you to take a look.

How it works:

Whenever we tag the repository and push the tag, the release.yml GitHub Action creates a release and copies the file installer/installer.sh into the release.

At the moment the user needs to run:

curl -L https://github.com/RudolfCardinal/crate/releases/download/installer-test-9/installer.sh
chmod u+x crate_docker_installer.sh
./crate_docker_installer.sh

When this PR is merged into master they will do:

curl -L https://github.com/RudolfCardinal/crate/releases/latest/download/installer.sh
chmod u+x crate_docker_installer.sh
./crate_docker_installer.sh

This script downloads all of the CRATE source code from the release and bootstraps a python script, which builds the docker images, creates the config files and databases, anonymises the demo database and starts the web application
If all the mandatory CRATE_DOCKER_* environment variables are set, the installer will run automatically. If any of these variables are unset, the user will be prompted to enter them through basic dialogs. See the installer.yml GitHub Action for a fully automated setup.
The user can then go and modify the configuration for their own setup.

Still to do:

I've changed the sample crate_anon_config.ini and crateweb_local_settings.py to include placeholders that are substituted by the installer script. These are indicated by @@name_of_option@@, which works OK except that for a sqlalchemy URL you end up with something like: url = @@dest_db_engine@@://@@dest_db_user@@:@@dest_db_password@@@@@dest_db_host@@:@@dest_db_port@@/@@dest_db_name@@?charset=utf8 and the python example is no longer valid code so the documentation fails to build. We could take the same approach as for the crate_anon_config.ini file and keep this in a string. The installer.yml GitHub action should fail if we introduce a coding error into the string.
I've added some wrapper scripts to installer.py which do the same as your linux/windows bash scripts. Currently there isn't an uninstall. That has to be done at the Docker level.

RudolfCardinal commented 2 years ago

Thanks! I ran curl -L https://github.com/RudolfCardinal/crate/releases/download/installer-test-7/installer.sh | bash (as a normal user) and it made progress, but got this:

Successfully installed certifi-2021.10.8 charset-normalizer-2.0.12 click-8.0.4 flake8-4.0.1 idna-3.3 mccabe-0.6.1 prompt-toolkit-3.0.22 pycodestyle-2.8.0 pydantic-1.9.0 pyflakes-2.4.0 python-on-whales-0.32.0 requests-2.27.1 tqdm-4.62.3 typer-0.4.0 typing-extensions-4.1.1 urllib3-1.26.8 wcwidth-0.2.5
+ python /home/rudolf/crate/installer/installer.py install
Warning: Input is not a terminal (fd=0).
Traceback (most recent call last):
  File "/home/rudolf/crate/installer/installer.py", line 863, in <module>
    main()
  File "/home/rudolf/crate/installer/installer.py", line 823, in main
    installer.install()
  File "/home/rudolf/crate/installer/installer.py", line 63, in install
    self.configure()
  File "/home/rudolf/crate/installer/installer.py", line 88, in configure
    self.configure_config_files()
  File "/home/rudolf/crate/installer/installer.py", line 103, in configure_config_files
    self.setenv(
  File "/home/rudolf/crate/installer/installer.py", line 335, in setenv
    value = value()
  File "/home/rudolf/crate/installer/installer.py", line 272, in get_docker_config_host_dir
    return self.get_user_dir(
  File "/home/rudolf/crate/installer/installer.py", line 345, in get_user_dir
    dir = input_dialog(title=title, text=text,
  File "/home/rudolf/.virtualenvs/crate-installer/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 890, in run
    return loop.run_until_complete(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/rudolf/.virtualenvs/crate-installer/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 815, in run_async
    return await _run_async2()
  File "/home/rudolf/.virtualenvs/crate-installer/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 791, in _run_async2
    result = await _run_async()
  File "/home/rudolf/.virtualenvs/crate-installer/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 732, in _run_async
    result = await f
EOFError

RudolfCardinal commented 2 years ago

(Is that some sort of failure to recognize a terminal, so dialogue boxes are failing? I ran it from a standard Ubuntu bash terminal.)

martinburchell commented 2 years ago

@RudolfCardinal oh yes. Maybe you can't just pipe it into bash like that when there are dialogs. Let me take a look.

In the meantime try saving the script and executing it.

RudolfCardinal commented 2 years ago

OK. Small things:

The Ctrl-N autocomplete didn't work for any field, and it might be worth clarifying in the dialogue boxes that directories are on the host (I tried /crate_config thinking it was Docker-relative; although that was probably daft I might not be the only one).
Re CRATE username: is that for the CRATE web interface?

Then:

+ python /home/rudolf/crate/installer/installer.py install
unknown flag: --rm
See 'docker --help'.

Usage:  docker [OPTIONS] COMMAND

[...]

Run 'docker COMMAND --help' for more information on a command.

To get more help with docker, check out our guides at https://docs.docker.com/go/guides/

Traceback (most recent call last):
  File "/home/rudolf/crate/installer/installer.py", line 863, in <module>
    main()
  File "/home/rudolf/crate/installer/installer.py", line 823, in main
    installer.install()
  File "/home/rudolf/crate/installer/installer.py", line 65, in install
    self.create_local_settings()
  File "/home/rudolf/crate/installer/installer.py", line 429, in create_local_settings
    self.run_crate_command("crate_print_demo_crateweb_config > "
  File "/home/rudolf/crate/installer/installer.py", line 660, in run_crate_command
    self.run_bash_command(
  File "/home/rudolf/crate/installer/installer.py", line 667, in run_bash_command
    docker.compose.run("crate_workers",
  File "/home/rudolf/.virtualenvs/crate-installer/lib/python3.8/site-packages/python_on_whales/components/compose/cli_wrapper.py", line 366, in run
    result = run(full_cmd, tty=tty)
  File "/home/rudolf/.virtualenvs/crate-installer/lib/python3.8/site-packages/python_on_whales/utils.py", line 150, in run
    raise DockerException(
python_on_whales.exceptions.DockerException: The docker command executed was `/usr/bin/docker compose run --rm crate_workers /bin/bash -c source /crate/venv/bin/activate; crate_print_demo_crateweb_config > $CRATE_WEB_LOCAL_SETTINGS`.
It returned with code 125
The content of stdout can be found above the stacktrace (it wasn't captured).
The content of stderr can be found above the stacktrace (it wasn't captured).

... am I using the wrong version of Docker? (Should that be checked explicitly?) Mine is Docker version 20.10.7, build 20.10.7-0ubuntu5~20.04.2.

martinburchell commented 2 years ago

OK. Small things:

* The Ctrl-N autocomplete didn't work for any field, and it might be worth clarifying in the dialogue boxes that directories are on the host (I tried /crate_config thinking it was Docker-relative; although that was probably daft I might not be the only one).

OK

* Re CRATE username: is that for the CRATE web interface?

It might be the superuser for the Django app?

... am I using the wrong version of Docker? (Should that be checked explicitly?) Mine is Docker version 20.10.7, build 20.10.7-0ubuntu5~20.04.2.

I'm on 20.10.12, installed from https://docs.docker.com/engine/install/ubuntu/. What does docker compose version say? For me it's v2.2.2.

RudolfCardinal commented 2 years ago

This is probably a version thing. docker-compose wasn't installed and Installing via apt gave me version 1.25.0 as the output of docker-compose version. But I see that I need docker compose version and it is docker compose being executed by the script. I think that is new-ish; but a bit odd if Docker 20.10.7 to 20.10.12 brings a new major command (they're not being very true to semantic version principles if so!). Docker 20.10.7 is the latest I get with Ubuntu 20.04 LTS. Do you think it's worth a check for "docker compose" and fall back to "docker-compose" if absent (or moan/stop if neither work)? Or should I install from elsewhere?

martinburchell commented 2 years ago

@RudolfCardinal I've updated the documentation and instructions above so you now run:

. <(curl -L https://github.com/RudolfCardinal/crate/releases/download/installer-test-7/installer.sh)

RudolfCardinal commented 2 years ago

If it crashes, that command exits the shell you start from (because of the set -e command, I presume). How about this?

curl -L https://github.com/RudolfCardinal/crate/releases/download/installer-test-7/installer.sh --output crate_docker_installer.sh && bash --init-file crate_docker_installer.sh

That keeps an interactive shell but when it crashes, it just leaves you where it started. (But it creates a temporary file. Could delete it or could not worry.)

RudolfCardinal commented 2 years ago

That was based on https://serverfault.com/questions/368054 -- I see there's also a tip on removing the file itself from within the file!

RudolfCardinal commented 2 years ago

Merge in now or do the docs require tweaking as above? Sorry, I have lost track between this one and the other with all the https bits. Do merge if you're happy!

martinburchell commented 2 years ago

Merge in now or do the docs require tweaking as above? Sorry, I have lost track between this one and the other with all the https bits. Do merge if you're happy!

I've got more to do on this one.

martinburchell commented 2 years ago

@RudolfCardinal I've pushed some more changes.

Outside of the installer, the sample config files will now be generated with the old examples rather than @@ placeholders
The docs should now be fixed
The installer aborts with a more helpful message if docker compose is too old
Now 3 lines to download and run the installer :disappointed:
I've ditched the Python Prompt Toolkit dialogs in favour of basic prompts with working auto-complete and validation
The CRATE_DOCKER_* environment variables get saved in a temporary file which can be sourced

How does that look now?

RudolfCardinal commented 2 years ago

one-line installer (for dev version), using "--fail" to pick up server-side errors (I've amended the docs accordingly):

curl --location https://github.com/RudolfCardinal/crate/releases/download/installer-test-9/installer.sh --fail --output crate_docker_installer.sh && chmod u+x crate_docker_installer.sh && ./crate_docker_installer.sh

ucam-department-of-psychiatry / crate

Initial development of installer #64