enhancement of link2symlink is needed to support gentoo portage merge

[zxq432]

Problem description

the link2symlink feature produce 2 .l2s. files for the original source file. a base file .l2s..0002 and an intermediate symbolic link file .l2s.0001, these 2 files will be placed in the same folder as the origninal source file if PROOT_L2S_DIR is unset or empty, or placed in the folder of $PROOT_L2S_DIR if it exists or hard ln command fails if it not exists.

gentoo portage emerge compiles source package and install the package in folder $ED first, and then move the files from $ED to $EROOT.

if PROOT_L2S_DIR is unset or empty, a package produced hard link under $ED. link2symlink produce .l2s.0001 and .l2s..0002 under $ED, after emerge move the files under $ED to $EROOT, the intermediate symbolic link file .l2s.*0001 and other symlink will fail.

if PROOT_L2S_DIR is set correctly, position of .l2s.0001 and .l2s..0002 will not change in the process of emerge, other symlink will not fail after move from $ED to $EROOT. but this may sometimes lead to name space collistion in the folder of $PROOT_L2S_DIR, either overwrite or block of same file name will make trouble.

tracking the position of .l2s. files and reset symlink will work but not so easy. using a new env variable PROOT_L2S_ROOT, and place .l2s. files in a folder with ${PROOT_L2S_ROOT} as path prefix for the absolute file path of the original link source file might be not hard to implement. just using the way as if PROOT_L2S_DIR is unset and prepend ${PROOT_L2S_ROOT} to the absolute path of the .l2. files will do the work, some times more work is needed to creat the dirctory needed. with this changes, the flat structure of one folder of .l2s. files is changed to a tree structure of .l2s.* files.

Steps to reproduce

Expected behavior

Additional information

[michalbednarski]

Name collisions are already handled by proot, both with and without PROOT_L2S_DIR set

Using following test script I'm getting 1 and 2 in final output despite name clash

#!/bin/sh
set -xe
rm -f a b c d

echo 1 > a
ln a b
ln b c
rm a b

echo 2 > a
ln a b
ln b d
rm a b

cat c d

[zxq432]

an example to produce failed files without PROOT_L2S_DIR, which can usually lead to a failed package emerge :

localhost ~ # cat l mkdir ED1 ED2 EROOT echo ED1a1 >ED1/a1 echo ED2a2 >ED2/a2 ln ED1/a1 ED2/ED1a1 ln ED2/a2 ED1/ED2a2 ls -al E find ED? ( -type f -o -type l ) -exec mv '{}' EROOT \; ls -al E localhost ~ # . l ED1: total 0 drwxr-xr-x 2 root root 3488 Sep 12 23:11 . drwx------ 6 root root 3488 Sep 12 23:11 .. lrwxrwxrwx 1 root root 102 Sep 12 23:11 .l2s.a10001 -> /root/ED1/.l2s.a10001.0002 -rw-r--r-- 1 root root 6 Sep 12 23:11 .l2s.a10001.0002 -rw-r--r-- 1 root root 6 Sep 12 23:11 ED2a2 -rw-r--r-- 1 root root 6 Sep 12 23:11 a1

ED2: total 0 drwxr-xr-x 2 root root 3488 Sep 12 23:11 . drwx------ 6 root root 3488 Sep 12 23:11 .. lrwxrwxrwx 1 root root 102 Sep 12 23:11 .l2s.a20001 -> /root/ED2/.l2s.a20001.0002 -rw-r--r-- 1 root root 6 Sep 12 23:11 .l2s.a20001.0002 -rw-r--r-- 1 root root 6 Sep 12 23:11 ED1a1 -rw-r--r-- 1 root root 6 Sep 12 23:11 a2

EROOT: total 0 drwxr-xr-x 2 root root 3488 Sep 12 23:11 . drwx------ 6 root root 3488 Sep 12 23:11 .. ED1: total 0 drwxr-xr-x 2 root root 3488 Sep 12 23:11 . drwx------ 6 root root 3488 Sep 12 23:11 ..

ED2: total 0 drwxr-xr-x 2 root root 3488 Sep 12 23:11 . drwx------ 6 root root 3488 Sep 12 23:11 ..

EROOT: total 0 drwxr-xr-x 2 root root 3488 Sep 12 23:11 . drwx------ 6 root root 3488 Sep 12 23:11 .. lrwxrwxrwx 1 root root 102 Sep 12 23:11 .l2s.a10001 -> /root/ED1/.l2s.a10001.0002 -rw-r--r-- 1 root root 6 Sep 12 23:11 .l2s.a10001.0002 lrwxrwxrwx 1 root root 102 Sep 12 23:11 .l2s.a20001 -> /root/ED2/.l2s.a20001.0002 -rw-r--r-- 1 root root 6 Sep 12 23:11 .l2s.a20001.0002 lrwxrwxrwx 1 root root 98 Sep 12 23:11 ED1a1 -> /root/ED1/.l2s.a10001 lrwxrwxrwx 1 root root 98 Sep 12 23:11 ED2a2 -> /root/ED2/.l2s.a20001 lrwxrwxrwx 1 root root 98 Sep 12 23:11 a1 -> /root/ED1/.l2s.a10001 lrwxrwxrwx 1 root root 98 Sep 12 23:11 a2 -> /root/ED2/.l2s.a20001 localhost ~ #

=========

if PROOT_L2S_DIR set correctly, there's no problem. but the file name becomes a bit complicated, and the original file position is lost. using PROOT_L2S_ROOT to create a tree structure for the .l2s.* files. more original position information of the origninal file will be kept, this sometimes might make tracking problems easier. for example, set PROOT_L2S_ROOT and then proot-distro install some distro, where the hard linked files is will be revealed, as the original position is tracked by the tree structure.

[zxq432]

an example to produce failed files without PROOT_L2S_DIR, which can usually lead to a failed package emerge :

localhost ~ # cat l mkdir ED1 ED2 EROOT echo ED1a1 >ED1/a1 echo ED2a2 >ED2/a2 ln ED1/a1 ED2/ED1a1 ln ED2/a2 ED1/ED2a2 ls -al E find ED? ( -type f -o -type l ) -exec mv '{}' EROOT \; ls -al E localhost ~ # . l ED1: total 0 drwxr-xr-x 2 root root 3488 Sep 12 23:11 . drwx------ 6 root root 3488 Sep 12 23:11 .. lrwxrwxrwx 1 root root 102 Sep 12 23:11 .l2s.a10001 -> /root/ED1/.l2s.a10001.0002 -rw-r--r-- 1 root root 6 Sep 12 23:11 .l2s.a10001.0002 -rw-r--r-- 1 root root 6 Sep 12 23:11 ED2a2 -rw-r--r-- 1 root root 6 Sep 12 23:11 a1

ED2: total 0 drwxr-xr-x 2 root root 3488 Sep 12 23:11 . drwx------ 6 root root 3488 Sep 12 23:11 .. lrwxrwxrwx 1 root root 102 Sep 12 23:11 .l2s.a20001 -> /root/ED2/.l2s.a20001.0002 -rw-r--r-- 1 root root 6 Sep 12 23:11 .l2s.a20001.0002 -rw-r--r-- 1 root root 6 Sep 12 23:11 ED1a1 -rw-r--r-- 1 root root 6 Sep 12 23:11 a2

EROOT: total 0 drwxr-xr-x 2 root root 3488 Sep 12 23:11 . drwx------ 6 root root 3488 Sep 12 23:11 .. ED1: total 0 drwxr-xr-x 2 root root 3488 Sep 12 23:11 . drwx------ 6 root root 3488 Sep 12 23:11 ..

ED2: total 0 drwxr-xr-x 2 root root 3488 Sep 12 23:11 . drwx------ 6 root root 3488 Sep 12 23:11 ..

EROOT: total 0 drwxr-xr-x 2 root root 3488 Sep 12 23:11 . drwx------ 6 root root 3488 Sep 12 23:11 .. lrwxrwxrwx 1 root root 102 Sep 12 23:11 .l2s.a10001 -> /root/ED1/.l2s.a10001.0002 -rw-r--r-- 1 root root 6 Sep 12 23:11 .l2s.a10001.0002 lrwxrwxrwx 1 root root 102 Sep 12 23:11 .l2s.a20001 -> /root/ED2/.l2s.a20001.0002 -rw-r--r-- 1 root root 6 Sep 12 23:11 .l2s.a20001.0002 lrwxrwxrwx 1 root root 98 Sep 12 23:11 ED1a1 -> /root/ED1/.l2s.a10001 lrwxrwxrwx 1 root root 98 Sep 12 23:11 ED2a2 -> /root/ED2/.l2s.a20001 lrwxrwxrwx 1 root root 98 Sep 12 23:11 a1 -> /root/ED1/.l2s.a10001 lrwxrwxrwx 1 root root 98 Sep 12 23:11 a2 -> /root/ED2/.l2s.a20001 localhost ~ #

=========

if PROOT_L2S_DIR set correctly, there's no problem. but the file name becomes a bit complicated, and the original file position is lost. using PROOT_L2S_ROOT to create a tree structure for the .l2s.* files. more original position information of the origninal file will be kept, this sometimes might make tracking problems easier. for example, set PROOT_L2S_ROOT and then proot-distro install some distro, where the hard linked files is will be revealed, as the original position is tracked by the tree structure.

[zxq432]

without PROOT_L2S_DIR for proot-distro gentoo, emerge failed with the following message:

>>> /usr/libexec/git-core/git-remote-ftp
!!! Failed to stat in movefile()
!!! /usr/libexec/git-core/.l2s.git-remote-http0001
!!! [Errno 2] No such file or directory: b'/usr/libexec/git-core/.l2s.git-remote-http0001'

>>> Failed to install dev-vcs/git-2.33.0-r1, Log file:

>>>  '/var/tmp/portage/dev-vcs/git-2.33.0-r1/temp/build.log'

[zxq432]

enhance the proot with a plain text log file in $PROOT_L2S_DIR folder for each link made by link2sysmlink, and a tool script to get rid of unnecessary links.

the following is a flat list of the .l2s. file in /.l2s folder, there are total 730 .l2s. files for 365 hard link groups, and 244 of the groups with a regular .l2s.*0001 files associated with 224 files each being the only single hard linked file to each of the .l2s.*0001 files. this kind of wrapper of a single file using link2symlink is really unnecessary.

llocalhost /.l2s # ls -alt .l2s* | wc
    730    7300   65305

localhost /.l2s # ls -alt | grep -v ^l | grep 1$ | wc
    244    2196   18729

after a run of "emaint sync -a", there is a huge file over 1BG: .l2s.tmp_pack_Qe2Jyu0001.0001

localhost /.l2s # ls -alt | head       
total 0
drwxr-xr-x  2 root root     253952 Sep 14 05:36 .
lrwxrwxrwx  1 root root        102 Sep 14 05:36 .l2s.a0001 -> /.l2s/.l2s.a0001.0001
-rw-r--r--  1 root root         29 Sep 14 05:34 .l2s.a0001.0001
lrwxrwxrwx  1 root root        114 Sep 14 03:17 .l2s.tmp_idx_fe4JsQ0001 -> /.l2s/.l2s.tmp_idx_fe4JsQ0001.0001
lrwxrwxrwx  1 root root        114 Sep 14 03:17 .l2s.tmp_pack_Qe2Jyu0001 -> /.l2s/.l2s.tmp_pack_Qe2Jyu0001.0001
-rw-r--r--  1 root root   88636220 Sep 14 03:17 .l2s.tmp_idx_fe4JsQ0001.0001
-rw-r--r--  1 root root 1081614493 Sep 14 03:17 .l2s.tmp_pack_Qe2Jyu0001.0001
drwxr-xr-x 23 root root       3488 Sep 14 03:10 ..
lrwxrwxrwx  1 root root        106 Sep 14 00:43 .l2s.perlbug0002 -> /.l2s/.l2s.perlbug0002.0002

localhost /.l2s # find -L /var/db/repos/gentoo -samefile .l2s.tmp_pack_Qe2Jyu0001.0001 -print -quit
/var/db/repos/gentoo/.git/objects/pack/pack-c70cc34a96b5b66f4f14504b666df6f7003a7417.pack
localhost /.l2s # 

you have no clue of the original position of the original file. find it using "find -L /var /usr /lib -samefile .l2s.tmp_pack_Qe2Jyu0001.0001 -print -quit" will take a long time. with PROOT_L2S_ROOT enhancement by a tree structure, it might be easier if the position of the original file was not change, using "find -L /var/db/repos/gentoo -samefile .l2s.tmp_pack_Qe2Jyu0001.0001 -print -quit" with result immediately.

[zxq432]

updated in https://github.com/termux/proot/issues/186