Test does not run on Windows

[nicost]

I am setting up a test environment for imagej-omero (or omero-imagej, can not keep those two straight;).

Running the test.sh script in imagej-omero/.github seems to go pretty well, it succeeds in pulling down the correct version of omero-test-infra, and running a docker image. However, it ends with:

inflating: /c/projects/Altos/imagej-omero/.github/.github/imagejomero_omerodata.tar
Backup ready
+ .omero/persist.sh --restore /c/projects/Altos/imagej-omero/.github/.github
Mounting volumes and performing restore
Restore github_dbdata from /c/projects/Altos/imagej-omero/.github/.github
find: 'C:/Program Files/Git/data': No such file or directory

I am running git from git Bash. The binary is located at/mingw64/bin/git. There is a 'C:/Program Files/Git/', but no data directory in there. For the fun of it, I created one there, but that changed nothing on the output.

Since this code is hidden somewhere in the docker image, I am unsure how to trouble shoot this. Any pointers on how to proceed?

[joshmoore]

I am setting up a test environment for imagej-omero (or omero-imagej, can not keep those two straight;).

imagej-omero, but yes, I can understand the confusion 🙂

Since this code is hidden somewhere in the docker image, I am unsure how to trouble shoot this. Any pointers on how to proceed?

I think the issue is in https://github.com/ome/omero-test-infra/blob/master/persist.sh and so you should have that checked out under .omero. The command being run is a Unix-style find. My best guess is that the git bash environment behaves slightly unexpectedly.

You might try running bash +x .omero/persist.sh --restore /c/projects/Altos/imagej-omero/.github/.github to see more of what is happening at each step.

[nicost]

$ bash +x .omero/persist.sh --restore /c/projects/Altos/imagej-omero/.github/.github
Mounting volumes and performing restore
Restore github_dbdata from /c/projects/Altos/imagej-omero/.github/.github
find: 'C:/Program Files/Git/data': No such file or directory

and:

$ docker run --rm -v github_dbdata:/data ubuntu find /data -mindepth 1 -delete
find: 'C:/Program Files/Git/data': No such file or directory

New to docker, but this does not make much sense to me. Who brings in the "C:/Program Files/Git"?

[joshmoore]

Sorry, @nicost. Getting too late. That should have been bash -x.

[nicost]

$ bash -x .omero/persist.sh --restore /c/projects/Altos/imagej-omero/.github/.github
+ set -u
+ set -e
+++ dirname -- .omero/persist.sh
++ cd -P -- .omero
++ pwd -P
+ dir=/c/projects/Altos/imagej-omero/.github/.omero
+ . /c/projects/Altos/imagej-omero/.github/.omero/utils
++ set -e
++ set -u
++ '[' '!' -z ']'
++ get_project_name
+++ get_app_name
+++ tr -cd '[[:alnum:]]'
+++ tr /A-Z/ /a-z/
++++ tr - _
+++++ basename /c/projects/Altos/imagej-omero/.github
++++ echo .github
+++ echo .github
++ echo github
+ PROJECT=github
+ p_or_b=backup
+ '[' 2 -lt 1 ']'
+ '[' --restore == --restore ']'
+ p_or_b=restore
+ shift
+ backup_path=/c/projects/Altos/imagej-omero/.github/.github
+ shift
+ main
+ echo 'Mounting volumes and performing restore'
Mounting volumes and performing restore
+ for v in ${PROJECT}_dbdata ${PROJECT}_omerodata
+ '[' restore == backup ']'
+ '[' restore == restore ']'
+ restore_volume github_dbdata /c/projects/Altos/imagej-omero/.github/.github
+ volume_name=github_dbdata
+ backup_destination=/c/projects/Altos/imagej-omero/.github/.github
+ echo 'Restore github_dbdata from /c/projects/Altos/imagej-omero/.github/.github'
Restore github_dbdata from /c/projects/Altos/imagej-omero/.github/.github
+ docker run --rm -v github_dbdata:/data ubuntu find /data -mindepth 1 -delete
find: 'C:/Program Files/Git/data': No such file or directory

Still no sign where that "C:/Program Files/Git" comes from...

[joshmoore]

Aha. This looks to be a Windows+Docker weirdness. See https://stackoverflow.com/questions/48427366/docker-build-command-add-c-program-files-git-to-the-path-passed-as-build-argu

If you specify command-line options starting with a slash, POSIX-to-Windows path conversion will kick in converting e.g. "/usr/bin/bash.exe" to "C:\Program Files\Git\usr\bin\bash.exe". When that is not desired -- e.g. "--upload-pack=/opt/git/bin/git-upload-pack" or "-L/regex/" -- you need to set the environment variable MSYS_NO_PATHCONV temporarily, like so:

In this case:

MSYS_NO_PATHCONV=1 docker run --rm -v github_dbdata:/data ubuntu find /data -mindepth 1 -delete

or alternatively:

docker run --rm -v github_dbdata:/data ubuntu find //data -mindepth 1 -delete

[nicost]

Thanks! That worked. I like the first solution best, since it kind o self-documents. Would you like me to open a PR?

Discovered a similar issue in the test.sh script. It needs: export COMPOSE_PATH_SEPARATOR=":" because Windows uses ";" as the compose path separator by default.

Most things now run, however, I get (posting only relevant excerpt, can add more if useful):

Container github-web-1  Starting
Container github-db-1  Starting
Container github-omero-1  Starting
Container github-web-1  Started
Container github-db-1  Started
Container github-omero-1  Started
#1 [internal] load build definition from Dockerfile
#1 sha256:a5daa7dcdc4a5663005deb8eb6a772f74070fa8beb1537194dc7d4137995d606
#1 transferring dockerfile: 2B done
#1 DONE 0.0s
failed to solve with frontend dockerfile.v0: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount521134892/Dockerfile: no such file or directory
Container github-web-1  Stopping
Container github-db-1  Stopping
Container github-db-1  Stopping
Container github-web-1  Stopping
Container github-omero-1  Stopping
Container github-omero-1  Stopping

Any clues where I should look?

[joshmoore]

Would you like me to open a PR?

By all means. Thanks!

Any clues where I should look?

The only idea I have off-hand is that the process isn't in the right directory so that the Dockerfile isn't where it is expected.

Do you know if it's possible to reproduce your Windows+Docker setup in GitHub Actions?

[nicost]

You were totally correct again! Turns out that I should run the test script from the topdirectory as.github/test.sh. That goes a whole lot further. However:

1. The Docker image starts a omero webserver running OMERO.web 5.7 rather than 5.6 as requested in the OMERO_WEB_VERSION environment variable in the test.sh script. Is there another way to specify the desired version ofe the OMERO web server. Probably less important, I also do not know what credentials to use.
2. The Docker image disappears with:

   #10 ERROR: executor failed running [/bin/sh -c mvn clean install -DskipTests]: exit code: 1
   ------
   > [6/6] RUN mvn clean install -DskipTests:
   ------
   executor failed running [/bin/sh -c mvn clean install -DskipTests]: exit code: 1
   Container imagejomero-db-1  Stopping

I don't actually know what to expect or even what I try to accomplish. I believe the goal is to have this Docker image running to run the mavev tests against, but there is not much documentation to go on.

[joshmoore]

1. The Docker image starts a omero webserver running OMERO.web 5.7 rather than 5.6 as requested in the OMERO_WEB_VERSION environment variable in the test.sh script. Is there another way to specify the desired version ofe the OMERO web server

Odd. I don't have an immediate answer. We might need to look at what tags you have locally. (A sidenote: the latest at https://hub.docker.com/r/openmicroscopy/omero-web/tags is 5.14)

2. The Docker image disappears with:

That looks like the build of https://github.com/imagej/imagej-omero/blob/master/Dockerfile has failed (at line 9) which is roughly where I was here: https://github.com/imagej/imagej-omero/pull/107#issuecomment-684790330

[nicost]

Odd. I don't have an immediate answer. We might need to look at what tags you have locally. (A sidenote: the latest at https://hub.docker.com/r/openmicroscopy/omero-web/tags is 5.14)

Specifying 5.14 gives me 5.14;) I am a bit confused about versioning of the various components. It does not seem to follow semver. Is there a document you can point me at to help me understand how the versioning works?

That looks like the build of https://github.com/imagej/imagej-omero/blob/master/Dockerfile has failed (at line 9) which is roughly where I was here: https://github.com/imagej/imagej-omero/pull/107#issuecomment-684790330

Looks like the next problems are in the imagej-omero repo. Sorry, hard coming in here to untangle who is doing what. Thanks!

[sbesson]

Specifying 5.14 gives me 5.14;) I am a bit confused about versioning of the various components. It does not seem to follow semver. Is there a document you can point me at to help me understand how the versioning works?

Hi @nicost, for each x.y.z release of OMERO.server & OMERO.web, there should always be a corresponding x.y.z Docker tag so that consumers can deploy a specific version of the app. In addition to this tag and according to a fairly common Docker practice, we are also maintaining and updating on each release a set of tags allowing to subscribe and track the minor, major and latest versions of the images. For instance if OMERO.web 5.14.1 were released, this would result in the generation a new openmicroscopy/omero-web:5.14.1 image and an update of the openmicroscopy/omero-web:5.14, openmicroscopy/omero-web:5 and openmicroscopy/omero-web:latest images.

You are correct this policy is underdocumented at the moment. I assume the README of https://hub.docker.com/r/openmicroscopy/omero-server and https://hub.docker.com/r/openmicroscopy/omero-web would be a good place to expand unless you can think of another entrypoint where you would like to see this information?

[nicost]

My question referred to the versioning of the individual OMERO components (web, server, more?) Do those go in unisone, or individually? Do they follow semver or not, what is the latest version (and probably a few more similar questions)? No doubt this is described somewhere, but I do not know where to look.

[sbesson]

Ah I misunderstood the scope of your question. The confusion probably comes from the fact this policy has evolved towards moving aways from monolithic releases and adopting semver across all components.

Historically, the server and web components were living in the same code base, released together and minor version increments were used for DB/API changes.

As part of the Python 3 migration which was officially released in January 2020 - see annoucement, the Python components have been split into their own repositories (omero-py, omero-web) are are now released independently of the server. We have used the opportunity to use standard Python packaging and also version these components according to semver.

On the server front, there has been no breaking API/DB changes to the server components in last the 3 years and mostly patch releases.

In short summary:

• the new omero-py, omero-web packages are versioning using semver
• OMERO.web 5.x deployments should be compatible OMERO.server 5.6.x deployments