Compile-time race-condition during generation of ja.sjis.po

Ecordonnier commented 1 year ago

Steps to reproduce

I am compiling vim as part of yocto poky version kirkstone, which is downloading vim's source-code using git. The compilation works most of the time, but on some specific build-machine, the compilation of vim always fails. This is a combination of a racy build-system and the usage of a cross-compiler as CC variable:

poky kirkstone currently uses version 9.0.1527 (commit c28e7a2b2f23dbd246a1ad7ad7aaa6f7ab2e5887 )
the issue is that when the timestamp of the file src/po/ja.sjis.po is newer than the timestamp of the file src/po/ja.po, the Makefile src/po/Makefile ( https://github.com/vim/vim/blob/master/src/po/Makefile#L105 ) attempts to call "sjiscorr" in order to re-generate the file ja.sjis.po. However, the binary file sjiscorr is an ARM64 binary since I am cross-compiling Vim for ARM64. Thus, the compilation of vim fails.

Relying on the timestamp of src/po/ja.sjis.po is racy, because the timestamp of this file changes whenever git updates this file. So for instance when cloning the master branch and then checking out version 9.0.1527, the timestamp of ja.sjis.po is older than the timestamp of ja.po and Make will attempt to regenerate the file:

ecordonnier@lj8k2dq3:~/repos$ git clone https://github.com/vim/vim.git
Cloning into 'vim'...
remote: Enumerating objects: 170115, done.
remote: Counting objects: 100% (4879/4879), done.
remote: Compressing objects: 100% (213/213), done.
remote: Total 170115 (delta 4739), reused 4692 (delta 4666), pack-reused 165236
Receiving objects: 100% (170115/170115), 147.76 MiB | 26.58 MiB/s, done.
Resolving deltas: 100% (144271/144271), done.
ecordonnier@lj8k2dq3:~/repos$ cd vim/
ecordonnier@lj8k2dq3:~/repos/vim$ ls -l src/po/ja.po src/po/ja.sjis.po 
-rw-rw-r-- 1 ecordonnier ecordonnier 311785 Aug 17 17:28 src/po/ja.po
-rw-rw-r-- 1 ecordonnier ecordonnier 273989 Aug 17 17:28 src/po/ja.sjis.po
ecordonnier@lj8k2dq3:~/repos/vim$ git checkout v9.0.1527
...
ecordonnier@lj8k2dq3:~/repos/vim$ ls -l src/po/ja.po src/po/ja.sjis.po 
-rw-rw-r-- 1 ecordonnier ecordonnier 311799 Aug 17 17:29 src/po/ja.po
-rw-rw-r-- 1 ecordonnier ecordonnier 273989 Aug 17 17:28 src/po/ja.sjis.po

Expected behaviour

The build-system should not be racy
Make should maybe not assume that sjiscorr compiled with CC can run on the host?

Version of Vim

9.0.1527

Environment

Ubuntu 22.04. Poky version kirkstone

Logs and stack traces

| Starting make in the src directory.
| If there are problems, cd to the src directory and run make there
| cd src && make first
| make[1]: Entering directory '/workdir/build/tmp-glibc/work/armv8a-linux/vim/9.0.1527-r0/git/src'
| make[2]: Entering directory '/workdir/build/tmp-glibc/work/armv8a-linux/vim/9.0.1527-r0/git/src/po'
| OLD_PO_FILE_INPUT=yes msgfmt -v -o cs.mo cs.po
| OLD_PO_FILE_INPUT=yes msgfmt -v -o fr.mo fr.po
| OLD_PO_FILE_INPUT=yes msgfmt -v -o ja.mo ja.po
| OLD_PO_FILE_INPUT=yes msgfmt -v -o pl.mo pl.po
| OLD_PO_FILE_INPUT=yes msgfmt -v -o sk.mo sk.po
| rm -f cs.cp1250.po
| iconv -f utf-8 -t euc-jp ja.po | \
|   LANG=C sed -e 's/charset=[uU][tT][fF]-8/charset=euc-jp/' -e 's/# Original translations/# Generated from ja.po, DO NOT EDIT/' > ja.euc-jp.po
| OLD_PO_FILE_INPUT=yes msgfmt -v -o ko.mo ko.po
| iconv -f iso-8859-2 -t cp1250 cs.po | \
|   LANG=C sed -e 's/charset=ISO-8859-2/charset=cp1250/' -e 's/# Original translations/# Generated from cs.po, DO NOT EDIT/' > cs.cp1250.po
| rm -f pl.UTF-8.po
| rm -f pl.cp1250.po
| rm -f sk.cp1250.po
| OLD_PO_FILE_INPUT=yes msgfmt -v -o zh_CN.mo zh_CN.po
| iconv -f iso-8859-2 -t utf-8 pl.po | \
|   LANG=C sed -e 's/charset=ISO-8859-2/charset=UTF-8/' -e 's/# Original translations/# Generated from pl.po, DO NOT EDIT/' > pl.UTF-8.po
| iconv -f iso-8859-2 -t cp1250 pl.po | \
|   LANG=C sed -e 's/charset=ISO-8859-2/charset=cp1250/' -e 's/# Original translations/# Generated from pl.po, DO NOT EDIT/' > pl.cp1250.po
| OLD_PO_FILE_INPUT=yes msgfmt -v -o zh_TW.mo zh_TW.po
| iconv -f iso-8859-2 -t cp1250 sk.po | \
|   LANG=C sed -e 's/charset=ISO-8859-2/charset=cp1250/' -e 's/# Original translations/# Generated from sk.po, DO NOT EDIT/' > sk.cp1250.po
| OLD_PO_FILE_INPUT=yes msgfmt -v -o cs.cp1250.mo cs.cp1250.po
| make[3]: Entering directory '/workdir/build/tmp-glibc/work/armv8a-linux/vim/9.0.1527-r0/git/src/po'
| ccache aarch64-linux-gcc  -march=armv8-a+crc -fstack-protector-strong  -D_FORTIFY_SOURCE=1 -Wformat -Wformat-security -Werror=format-security --sysroot=/workdir/build/tmp-glibc/work/armv8a-linux/vim/9.0.1527-r0/recipe-sysroot -o sjiscorr sjiscorr.c
| 0 translated messages.
| OLD_PO_FILE_INPUT=yes msgfmt -v -o pl.cp1250.mo pl.cp1250.po
| 0 translated messages.
| OLD_PO_FILE_INPUT=yes msgfmt -v -o pl.UTF-8.mo pl.UTF-8.po
| 0 translated messages.
| 0 translated messages.
| OLD_PO_FILE_INPUT=yes msgfmt -v -o sk.cp1250.mo sk.cp1250.po
| 0 translated messages.
| 0 translated messages.
| 0 translated messages.
| 0 translated messages.
| 0 translated messages.
| 0 translated messages.
| 0 translated messages.
| 2782 translated messages.
| OLD_PO_FILE_INPUT=yes msgfmt -v -o ja.euc-jp.mo ja.euc-jp.po
| In file included from /workdir/build/tmp-glibc/work/armv8a-linux/vim/9.0.1527-r0/recipe-sysroot/usr/include/bits/libc-header-start.h:33,
|                  from /workdir/build/tmp-glibc/work/armv8a-linux/vim/9.0.1527-r0/recipe-sysroot/usr/include/stdio.h:27,
|                  from sjiscorr.c:6:
| /workdir/build/tmp-glibc/work/armv8a-linux/vim/9.0.1527-r0/recipe-sysroot/usr/include/features.h:412:4: warning: #warning _FORTIFY_SOURCE requires compiling with optimization (-O) [-Wcpp]
|   412 | #  warning _FORTIFY_SOURCE requires compiling with optimization (-O)
|       |    ^~~~~~~
| 2782 translated messages.
| make[3]: Leaving directory '/workdir/build/tmp-glibc/work/armv8a-linux/vim/9.0.1527-r0/git/src/po'
| rm -f ja.sjis.po
| iconv -f utf-8 -t cp932 ja.po | ./sjiscorr > ja.sjis.po
| /bin/sh: ./sjiscorr: cannot execute binary file: Exec format error
| make[2]: *** [Makefile:105: ja.sjis.po] Error 126
| make[2]: Leaving directory '/workdir/build/tmp-glibc/work/armv8a-linux/vim/9.0.1527-r0/git/src/po'
| make[1]: *** [Makefile:2074: languages] Error 2
| make[1]: Leaving directory '/workdir/build/tmp-glibc/work/armv8a-linux/vim/9.0.1527-r0/git/src'
| make: *** [Makefile:29: first] Error 2

k-takata commented 1 year ago

The build-system should not be racy

The commit e978b4534a5e10471108259118c0ef791106fd92 should have updated src/po/ja.sjis.po, however, it wasn't updated.

Make should maybe not assume that sjiscorr compiled with CC can run on the host?

We should use HOSTCC or a similar variable, I think.

chrisbra commented 1 year ago

I think I tried editing the file but my vim seemed to cause encoding issues. So I left it alone.

Ecordonnier commented 1 year ago

Make should maybe not assume that sjiscorr compiled with CC can run on the host?

We should use HOSTCC or a similar variable, I think.

FYI yocto provides "BUILD_CC" (for the host) and "CC" (for the target) in the environment. Yocto does not provide HOST_CC (but it's easy to add a patch for vim in yocto if needed).

k-takata commented 1 year ago

I think I tried editing the file but my vim seemed to cause encoding issues. So I left it alone.

It will be automatically updated if you run make in src/po/. I've done it in #12840.

chrisbra commented 1 year ago

ah thanks!

k-takata commented 1 year ago

Make should maybe not assume that sjiscorr compiled with CC can run on the host?

This has not been fixed yet. Reopening.

@Ecordonnier How did you run configure when you cross-compiling?

Ecordonnier commented 1 year ago

@k-takata this is the code calling configure during a yocto build: https://github.com/yoctoproject/poky/blob/master/meta/recipes-support/vim/vim.inc#L41

do_configure () {
    cd src
    rm -f auto/*
    touch auto/config.mk
    aclocal
    autoconf
    cd ..
    oe_runconf
    touch src/auto/configure
    touch src/auto/config.mk src/auto/config.h
}

I'm attaching a full log in case you want to have a look, but honestly if you are not familiar with yocto the complexity is high, so it may be easier to focus on a simpler minimal cross-compile use-case and replace CC with BUILD_CC where appropriate:

NOTE: Running ./configure --build=x86_64-linux --host=aarch64-poky-linux --target=aarch64-poky-linux --prefix=/usr --exec_prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/libexec --datadir=/usr/share --sysconfdir=/etc --sharedstatedir=/com --localstatedir=/var --libdir=/usr/lib --includedir=/usr/include --oldincludedir=/usr/include --infodir=/usr/share/info --mandir=/usr/share/man --disable-silent-rules --disable-dependency-tracking --with-libtool-sysroot=/home/ecordonnier/repos/poky/build/tmp/work/cortexa57-poky-linux/vim/9.0.1592/recipe-sysroot --disable-gpm --disable-gtktest --disable-xim --disable-netbeans --disable-desktop-database-update --with-tlib=ncurses --with-modified-by='Poky Maintainers poky@lists.yoctoproject.org' ac_cv_small_wchar_t=no ac_cv_path_GLIB_COMPILE_RESOURCES=no vim_cv_getcwd_broken=no vim_cv_memmove_handles_overlap=yes vim_cv_stat_ignores_slash=no vim_cv_terminfo=yes vim_cv_tgetent=non-zero vim_cv_toupper_broken=no vim_cv_tty_group=world STRIP=/bin/true --disable-static --enable-acl --enable-gui=gtk3 --enable-nls --disable-selinux --disable-canberra --with-features=big --with-x vim_cv_timer_create=yes

Full log: log.do_configure.txt

vim / vim