Cannot check out Jenkins core Git repository on new Linux container agents

basil commented 2 years ago

Service(s)

ci.jenkins.io

Summary

Git checkouts of Jenkins core are currently failing with the error shown in this build:

[2022-09-15T15:50:24.593Z] using credential app-ci.jenkins.io
[2022-09-15T15:50:29.417Z] Cloning the remote Git repository
[2022-09-15T15:50:29.417Z] Cloning with configured refspecs honoured and without tags
[2022-09-15T15:50:31.205Z]  > JGit fetch # timeout=10
[2022-09-15T15:50:55.421Z] remote: Enumerating objects
[2022-09-15T15:50:55.421Z] remote: Counting objects
[2022-09-15T15:50:55.423Z] remote: Compressing objects
[2022-09-15T15:50:55.610Z] Receiving objects
[2022-09-15T15:51:23.409Z] Resolving deltas
[2022-09-15T15:51:39.085Z] Updating references
[2022-09-15T15:51:41.260Z] Avoid second fetch
[2022-09-15T15:51:41.260Z] Checking out Revision c982dec3d0db85d2064efba3e42166ca6c03a458 (master)
Malformed input or input contains unmappable characters: /home/jenkins/workspace/Core_jenkins_master/core/src/test/resources/hudson/model/UserIdMigratorTest/migrateMultipleUsers/users/zzzက/._config.xml2413907229913945153.tmp

This is the case for all builds on the main branch and all PR builds. It seems to be happening on the Linux container agents.

A build from just 20 hours ago worked:

[2022-09-14T20:00:55.021Z] using credential app-ci.jenkins.io
[2022-09-14T20:00:59.116Z] Cloning the remote Git repository
[2022-09-14T20:00:59.116Z] Cloning with configured refspecs honoured and without tags
[2022-09-14T20:01:04.174Z]  > JGit fetch # timeout=10
[2022-09-14T20:01:26.704Z] remote: Enumerating objects
[2022-09-14T20:01:26.704Z] remote: Counting objects
[2022-09-14T20:01:26.708Z] remote: Compressing objects
[2022-09-14T20:01:26.897Z] Receiving objects
[2022-09-14T20:01:33.313Z] Resolving deltas
[2022-09-14T20:01:51.066Z] Avoid second fetch
[2022-09-14T20:01:51.067Z] Checking out Revision a957c67cfa58c86080d43ed2906e35e96fc89f08 (master)
[2022-09-14T20:01:48.988Z] Updating references
[2022-09-14T20:01:58.980Z] Commit message: "Revert "[JENKINS-68712] updateCenter successfully installs nothing where there's nothing to install" (#7108)"
[2022-09-14T20:01:59.844Z] [GitHub Checks] GitHub check (name: Jenkins, status: in_progress) has been published.
[2022-09-14T20:01:59.844Z] [GitCheckoutListener] Skipping recording, since SCM 'git https://github.com/jenkinsci/jenkins.git' already has been processed

Note that this was also using JGit on Linux container agents. I saw Stephane's message "Improvement and change of image for linux jnlp-maven agents" to the developer mailing list, and I suspect that this has introduced a change in behavior.

I am actually familiar with this "java.nio.file.InvalidPathException: Malformed input or input contains unmappable characters" error in Java. It happens when the default character set cannot express a character that is being used. In this case the code base has a directory named zzz\341\200\200 (Unicode characters escaped in this message for readability, but not escaped in the filename checked into Git) which is intentionally being used to simulate a user with a Unicode character in their username.

I suspect that prior to the new image for Linux JNLP Maven agents, the character set being used was UTF-8. I suspect that it may now be ASCII or some other encoding that cannot express the Unicode characters that are in these filenames in Jenkins core:

core/src/test/resources/hudson/model/UserIdMigratorTest/migrateMultipleUsers/users/zzz\341\200\200/config.xml
core/src/test/resources/hudson/model/UserIdMigratorTest/scanExistingUsersOldLegacy/users/zzz\341\200\200/config.xml
test/src/test/resources/hudson/model/UserIdMigratorTest/migrateMultipleUsers/users/zzz\341\200\200/config.xml

Reproduction steps

This is reproducible by building https://ci.jenkins.io/job/Core/job/jenkins/job/master/ or any PR.

basil commented 2 years ago

Since I doubt we are explicitly setting a character encoding for Java with -Dfile.encoding, I think Java is just picking up the system's default encoding via /etc/default/locale. I suspect this used to contain LANG=en_US.UTF-8 but perhaps no longer does?