Open porunov opened 1 year ago
porunov
commented
1 year ago Notice, I restarted previously passed dist tests for the commit https://github.com/JanusGraph/janusgraph/commit/666a81cda3c79dfa0df19013dca0c656a9b7d9ee
Now not only they failed for Java 11, but also for Java 8 as well. See failed job: https://github.com/JanusGraph/janusgraph/actions/runs/5539504707/job/15297672033
porunov
commented
1 year ago @JanusGraph/committers if anyone can take a look at this issue it would be really helpful. This issue blocks any other PRs from being merged.
I tried to investigate the issue, but couldn't figure out the problem of the failing testJanusGraphServer test.
The test passes both for Java 8 and Java 11 locally on my machine, but fails on GitHub actions.
The test fails when we are executing the line 155 (runTraversalAgainstServer(createGraphSONMessageSerializer());) of the AbstractJanusGraphAssemblyIT class.
Observations:
Any ideas or thoughts are very welcome. Thank you!
porunov
commented
1 year ago Issue which could be related: #1826
porunov
commented
1 year ago @farodin91 @FlorianHockmann looks like the tests are failing due to some environment changes (probably due to docker updates as noted here: https://github.com/JanusGraph/janusgraph/issues/1826#issuecomment-1435953116). If I revert 2 commits related to moving docker from its separate repository into the main repository the tests are passing.
You can see that dist tests are passing in the next branch where I reverted those 2 commits: https://github.com/porunov/janusgraph/commits/bugfix/dist-tests-docker
As I understand, the work of moving Docker builds into the main repository is still Work In Progress. As such, would you be OK if I revert those two commits to unblock other PRs, and instead create a separate branch with those commits? After the work of moving Docker builds into the main repo is finished and all tests are passing then we can merge those commits back into the “master” branch again. If you prefer, to fix the issue instead, I’m fully for that. I personally wasn’t following the Docker related work much, thus, fixing the issue myself will be more challenging. If you don’t have enough capacity on this issue I would prefer reverting those commits (they don’t fully work anyways) and use dedicated Docker repository instead as we used to have. Let me know what you think about it. Thank you!
farodin91
commented
1 year ago @farodin91 I will later today look into this issue.
farodin91
commented
1 year ago @porunov I will try get out a log output which would help to debug. Currently I'm thinking of out of memory or out of storage.
porunov
commented
1 year ago @porunov I will try get out a log output which would help to debug. Currently I'm thinking of out of memory or out of storage.
Make sense. It's just strange that with no any code changes these tests started to fail in master branch, even for those commits which were passing previously. First the tests started to fail for Java 11 only, but after several days those tests started to fail for Java 8 as well. I initially assumed it has to be related to Java updates which were going during that time for zulu distribution, but when tried to set Java version to previous versions in #3896 - nothing happened.
First I confirmed that for Java 8.0.372+7 it was passing initially (when only Java 11 tests were failing), but after several days the same PR #3896 started to fail for Java 8.0.372+7 as well - which is strange.
Thus, it feels like the issue could be related to something else (like docker updates, OS updates, maven updates, etc.).
It's not fully clear yet what causes such behavior.
porunov
commented
1 year ago Not sure, but could also be related to the file janusgraph-dist/docker/Dockerfile
I see it has base image and OS versions which means that they are using latest tag:
ARG BASE_IMAGE=eclipse-temurin:11-jre
FROM debian:buster-slim as builderPotentially, something could be changed in the latest version which causes tests to fail. For example, default closed ports or some incompatible updates. I think those version should be explicitly defined instead of relying always on latest version.
That said, I'm not sure if those latest versions are related to this issue or not.
farodin91
commented
1 year ago @porunov I could deactivate the docker build process for now. It would be easy possible without reverting and testing the issue in my current open PR: https://github.com/JanusGraph/janusgraph/pull/3842
farodin91
commented
1 year ago It looks like updating runs-on Ubuntu from 20.04 to 22.04 seems to work.
porunov
commented
1 year ago It looks like updating runs-on Ubuntu from 20.04 to 22.04 seems to work.
Interesting why suddenly JanusGraph Docker container stopped to work in Ubuntu 20.04 but works in Ubuntu 22.04. Do you know if we start Docker container eclipse-temurin:11-jre inside another Docker container ubuntu-22.04?
If so, should we fix the version of that Docker container to be a concrete version like eclipse-temurin:11.0.20_8-jre instead of always using a the latest version eclipse-temurin:11-jre?
farodin91
commented
1 year ago eclipse-temurin:11-jre is just used for the docker image. The image is used for the test. This is the last executed test.
porunov
commented
1 year ago This issue was fixed in #3842 by upgrading from Ubuntu 20 to Ubuntu 22. We don't know why it suddenly started to failing in Ubuntu 20, even so it was working before. I assume some dependencies were changed in Ubuntu 20 which resulted in the failures, but we didn't investigate what changed exactly and we don't know if this will be changed in Ubuntu 22 in the future or not. Thus, we are good for now, but JanusGraph builds might break later for Ubuntu 22. From the following comment I assume it's related to Docker version: https://github.com/JanusGraph/janusgraph/issues/1826#issuecomment-1435953116
I also think that this is the main problem for users commented in #1826 . Perhaps it will help investigating this issue in the future.
I don't think there is anyone working on this issue, thus, I'm removing milestone from this ticket because it doesn't block the builds anymore and all tests pass in master branch now.
Please include configurations and logs if available.
For confirmed bugs, please report:
masterStack Trace (if you have one)