Open turkeylurkey opened 2 years ago
Attempt to reproduce and see how broken dev mode is when this happens. Also try to reproduce on another platform. Try a few times to see if intermittent. Try with large feature list to see if slow install features is more likely to cause it.
EDIT: Based on https://github.com/OpenLiberty/ci.maven/issues/1070#issuecomment-1125438002 this looks to be a unique issue that is occurring on slower machines where the application is updating before the features are installed. Following the feature definition not found error noted above, if a user makes subsequent source file changes that does not guarantee that the application works as expected as the subsequent file changes may not trigger install feature. If generate-features detects a new feature and install feature is triggered, that may correct the issue. "R" + Enter to restart the container does ensure that the correct features are installed and the application works as expected.
Looks to be a duplicate of https://github.com/OpenLiberty/ci.maven/issues/1070 but the suggested workaround of copying server.xml & generated-features.xml from the target dir noted here: https://github.com/OpenLiberty/ci.maven/issues/1070#issuecomment-747709455 does not seem to work.
Was not able to reproduce reliably with the current Dockefile on MacOS, Java 11, Maven 3.8.4.
After further investigation there are two main issues:
COPY --chown=1001:0 target/liberty/wlp/usr/servers/defaultServer/configDropins/overrides/generated-features.xml /config/configDropins/overrides/
In this scenario, when the generated-features.xml file in the local target directory is updated, the generated-features.xml file on the container is not updated. This is confirmed by checking the inode numbers (ls -i generated-features.xml
). Prior to a change the inode numbers are the same, after a change they vary with the file on the container still referencing the old inode number.
This occurs for many of the files on Linux that use a single file mount. The workaround is to copy entire directories in the Dockerfile, ie.
COPY --chown=1001:0 target/liberty/wlp/usr/servers/defaultServer/configDropins/overrides /config/configDropins/overrides
or to restart the server and container with "R" + Enter.
In dev mode, the proposed solution is to restart the container on Linux when a server configuration file has changed. We can extend this existing fix for a similar issue: https://github.com/OpenLiberty/ci.common/pull/171.
liberty:install-feature
which in turn calls featureUtility to run on the container.
When we restart the container without rebuilding the image, we are stopping the container and starting a new container. Since we are not rebuilding the image, the RUN features.sh
of the Dockerfile does not run and we are not running liberty:install-feature
, so the features will not be properly installed on the new container. This can be confirmed by comparing the features .mf
files located on the container (/lib/features/) before restarting and after.A proposed solution would be to call install features in the restartServer logic in the event that we do not rebuild the image (as RUN features.sh
would not have ran): https://github.com/OpenLiberty/ci.common/blob/9ad1bb92bafd1fa3a31bbea52742e176ac7639a2/src/main/java/io/openliberty/tools/common/plugins/util/DevUtil.java#L1820-L1824
Dockerfile for reference:
FROM icr.io/appcafe/open-liberty:kernel-slim-java8-openj9-ubi
# Add config
COPY --chown=1001:0 target/liberty/wlp/usr/servers/defaultServer/server.xml /config/
COPY --chown=1001:0 target/liberty/wlp/usr/servers/defaultServer/bootstrap.properties /config/
COPY --chown=1001:0 target/liberty/wlp/usr/servers/defaultServer/configDropins/overrides/liberty-plugin-variable-config.xml /config/configDropins/overrides/
COPY --chown=1001:0 target/liberty/wlp/usr/servers/defaultServer/configDropins/overrides/generated-features.xml /config/configDropins/overrides/
RUN features.sh
# Add application
COPY --chown=1001:0 target/liberty/wlp/usr/servers/defaultServer/apps/demo-devmode-maven.war /config/apps/
After adding a new Java file the endpoint is not visible. Tested on Linux.
First start with a kernel image: kernel-slim-java8-openj9-ubi.
Using the demo-devmode project add a command to Dockerfile to copy generate-fearures.xml to the image: COPY --chown=1001:0 target/liberty/wlp/usr/servers/defaultServer/configDropins/overrides/generated-features.xml /config/configDropins/overrides/
With dev mode running add the file SystemLivenessCheck.java to the project. As seen in the output below source compilation is successful.
A new feature is generated and added to the container.
The web application is started before
generated-features.xml
is copied totarget
. The new endpoint is not found. If you restart the container the events happen in order and the endpoint is found. After container restart: