Add docker compose test to gh workflow

[autumnjohnson]

This pull request includes the latest changes from the main branch, as well as the following improvements:

1. Adds a GitHub workflow test to build the docker-compose.yml file
2. Adds gcc dependency to backend Docker container, which is required for obspy
3. Adjusts installation order in backend Dockerfile to speed installation
4. Includes minor improvements to README file, including formatting for Markdown links and addition of port to localhost deployment URL

Note: This pull request does not remove the instructions for installation with Python venv and pip. Instead, instructions for installation with conda will be included in a subsequent pull request.

[jmsaurel]

@autumnjohnson : the compilation was successful.

However, when running the docker with docker-compose up, it fails because I guess it tries to use the default http port 80, which is already in use on my computer.

yasmine-frontend is up-to-date
Starting yasmine-backend ... 
Starting yasmine-backend ... error

ERROR: for yasmine-backend  Cannot start service yasmine-backend: driver failed programming external connectivity on endpoint yasmine-backend (0e46c6ea0487a79ce4fc1653dab37134d590a08d3a1daf5ff2cc4afb86c489f8): Error starting userland proxy: listen tcp4 0.0.0.0:80: bind: address already in use

ERROR: for yasmine-backend  Cannot start service yasmine-backend: driver failed programming external connectivity on endpoint yasmine-backend (0e46c6ea0487a79ce4fc1653dab37134d590a08d3a1daf5ff2cc4afb86c489f8): Error starting userland proxy: listen tcp4 0.0.0.0:80: bind: address already in use
ERROR: Encountered errors while bringing up the project.

When running the conda installation, I can use options --port, --host to specify on which port and IP I want to bind the webapp (and --debug also for more messages).

How can I pass those options to yasmine through docker ?

[autumnjohnson]

Hi @jmsaurel,

Yes, port 80 appears to already be in use on your computer, likely from a previous run of Yasmine. I recommend killing whatever process is using that port. Otherwise, if you look at the docker-compose.yml file, you will see:

       ports:
           - "80:80"

You can change this to something like the following and run it again:

       ports:
           - "90:90"

[fabienengels]

Hi @jmsaurel,

Yes, port 80 appears to already be in use on your computer, likely from a previous run of Yasmine. I recommend killing whatever process is using that port. Otherwise, if you look at the docker-compose.yml file, you will see:

       ports:
           - "80:80"

You can change this to something like the following and run it again:

       ports:
           - "90:90"

I think there is a typo, the second number isn't suppose to change as it's a fixed port inside the container. And for the first one, I would suggest something like 8080 (or anything over 1024) as you need to be root to open a port between 1 and 1024. Ex : "8080:80"

[jmsaurel]

@autumnjohnson , ok so changing the backend port to 8880 worked and I was able to access the yasmine frontend on port 1841 as it is configured. I have in fact an Apache configured on port 80 on my computer.

@fabienengels : there is indeed probably something strange. When installing with conda, I never bothered with this internal port for the frontend to communicate with the backend. It shouldn't have been 80 for the conda or venv install.

Or maybe, it's a mix between frontend and backend port ? The frontend being on port 80 by default and the backend on port 1841 ?

[jmsaurel]

Another comment now that it works : would it be possible to configure the docker so that the sqlite database and the libraries (AROL and NRL) repos are permanent ?

As it is configured now, the sqlite database is cleaned at each restart (down and up). And the NRL is completely downloaded again at each start.

[autumnjohnson]

Hi @jmsaurel

Yes, of course. I return from vacation on Thursday of next week and will look into it them.

[autumnjohnson]

Hi @fabienengels

This sounds reasonable: "8080:80." I added the change to this branch as a commit.

[autumnjohnson]

Hi @jmsaurel

As it is configured now, the sqlite database is cleaned at each restart (down and up).

The folder /opt/yasmineof the backend Docker container is being persisted across restarts. On my machine, this folder maps to /var/lib/docker/volumes/yasmine-stationxml-editor_yasmine-backend/_data.

The issue is that the backend Docker container is using /opt/YASMINE as its working directory. This folder and changes made to it are not persisted across restarts.

For whatever reason, the Docker container regards /opt/YASMINE and /opt/yasmine as different locations on the file path. This behavior is not consistent across operating systems: for example, cd /opt/yasmine and cd /opt/YASMINE go to the same location on my machine.

It's usually safest to avoid upper-case file path names for this reason. I've renamed instances of this folder and pushed the changes. After pulling the commits, I recommend running the following commands to be sure everything is clean:

docker-compose down
docker volume rm yasmine-stationxml-editor_yasmine-backend
docker-compose build --no-cache
docker-compose up

You may want to rename your local data folder toyasmine as well for use with the Python virtual environment installation.

[autumnjohnson]

Hi @jmsaurel Would we rather the docker-compose installation use the same persistent data directory on the host machine as the Python virtual environment installation? Namely, c://yasmine or /opt/yasmine depending on your operating system. As mentioned, it is currently using something like /var/lib/docker/volumes/yasmine-stationxml-editor_yasmine-backend/_data. This way you can use the same data regardless of whether you are running the application with Python or Docker.

Even better would be a simple ~/yasmine for both to avoid writing to the root directory of the host machine.

[jmsaurel]

Hello @autumnjohnson ,

Sorry, I was on the field in la Réunion island last week. I will have a look at this updated docker file this week.

From a standalone user perspective, it makes sense to have this permanent repository configured as ${HOME}/yasmine. Do you know if there is an equivalent of ${HOME} on Windows for Python install ?

The /opt/yasmine and c:\\yasmine also works well. And I agree, they should be lower-case, I don't think there are any reasons to have them upper-case.

In any case (${HOME} or/opt`), the yasmine directory should be in lower-case and should be the same for both Python and docker install.

Jean-Marie.

[autumnjohnson]

Hi @jmsaurel

Do you know if there is an equivalent of ${HOME} on Windows for Python install ?

Windows environments have a similar notion to $HOME. For instance, using a tilde like cd ~ takes you to the current user's home directory on either OS.

My intention was to use a tilde in the docker-compose file, but it appears Docker prefers absolute paths rather than ones that have to be expanded at runtime when binding host directories.

The /opt/yasmine and c:\yasmine also works well.

Yes. However, I am not sure if there is a way to conditionally tell Docker to mount a directory depending on which operating system it is running - either Windows or otherwise - as in the Python install.

In any case (${HOME} or /opt`), the yasmine directory should be in lower-case and should be the same for both Python and docker install.

I agree. A possible solution would be to use the "current working directory" indicated by ./ in the docker-compose file. This would mean the yasmine directory would be generated or should be moved to the root of the repository itself. Paths relative to the working directory are not ambiguous across machines and could be used for the Python install as well.

The directory yasmine-stationxml-editor would then have three folders: yasmine, backend, and frontend. If we add the yasmine directory to the .gitignore file, it will not be committed along with the rest of the project.

[jmsaurel]

Hi @autumnjohnson ,

I just tested your latest version. Everything works and the databases and settings are now persistent.

I don't have much experience with docker. I see that the persistent files are stored inside the docker volume. In my case, this is /var/lib/docker/volumes/yasmine-stationxml-editor_yasmine-backend.

So I guess there is in fact nothing to change in the backend/yasmine/app/settings.py main settings file. This is internal to the docker and not representative of the real directory on the host computer.

The docker version thus seems to me ready now for publication.

Regarding the conda python installation, I would rather use a directory that doesn't require root, sudo or admin rights. Maybe something like $HOME/yasmine-data. We could find the $HOME by using the os.path.expanduser('~') python class in backend/yasmine/app/settings.py main settings file.

[autumnjohnson]

Hi @jmsaurel

I pushed a couple commits to the branch. We can rollback these changes if the previous behavior is preferred. Either way I think we are close to concluding this pull request.

I have configured the application to use the same host directory for Python and Docker installations. This directory is used by Python irregardless of the host machine's OS.

If not present, the directory is generated at the root of the repository as yasmine-stationxml-editor/data.

I have tested this behavior with docker-compose and do not anticipate issues with the Python installation venv. However, I would appreciate your input here.

[jmsaurel]

@autumnjohnson , I just pulled your latest commit and tested it.

Unfortunately, I have a docker issue at the very beginning.

> docker-compose build --no-cache
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.yasmine-backend.build contains unsupported option: 'network'
services.yasmine-frontend.build contains unsupported option: 'network'

Could it be related to the relative old version of docker-compose included in the latest Debian ? docker-compose version 1.25.0, build unknown

Otherwise, I agree with the modification you propose to include a data directory in the code directory and to maintain this directory the same with python or docker install.

[autumnjohnson]

Hi @jmsaurel

In yesterday's commit, I included a change to the docker-compose file to reference the containers by Dockerfile as opposed to image name.

While I thought I could slip this optimization in, it might have caused these problems. I have reverted that portion of the code. This does not affect the changes I made to include a data directory. Do you mind pulling the latest commit and trying it again?

[jmsaurel]

Hi @autumnjohnson , your last push works.

And I was able to copy an older /opt/YASMINE directory in the data one and I recovered my work.

I think we are done with this PR.