Open lostdusty opened 1 year ago
Could it be that $PREFIX
is placed in some case-insensitive filesystem such as FAT32? Note that ndk-sysroot
contains both of:
$PREFIX/include/linux/netfilter/xt_dscp.h
$PREFIX/include/linux/netfilter/xt_DSCP.h
According to duf
, where $PREFIX
is, /data
is F2FS:
Do I need to format /data
as EXT4?
The issue was indeed the filesystem on /data
. Unsure if related but this phone came with a Samsung eMMC storage chip, which may be the reason /data
was F2FS. Now with EXT4 ndk-sysroot installs with no issues, see the screenshot below:
Maybe add an warning to users about this behavior? I've seen in #5016 that other users using Motorola phones had the same issue. Although, only certain devices had this issue.
2 years late, but @BruceMustache, @EdoZullo and @marcdw1289, if you people are still having issues ndk-sysroot, change the filesystem of /data
in TWRP from F2FS to EXT4. See this thread on Stack Exchange if you don't know how to do it. You might need to reflash your ROM after changing the filesystem, as it wipes the contents of /data
.
Looking back at my old posts and checking things now, the filesystem was not(?) and is not an issue for me.
I multibooted on both the Moto G5 Plus potter and G5s Plus sanders (still do on that one). The failure to install ndk-sysroot only happened with OmniROM, Nougat and Oreo. That ROM is no longer used. The package installs on existing ROMs so far (Pie and Oreo). Will verify others just in case.
Apparently, case-insensitive/casefold
support in f2fs
was added by google for android.
https://unix.stackexchange.com/questions/611116/why-case-insensitive-option-in-ext4-was-needed
https://source.android.com/docs/core/storage/sdcardfs-deprecate
https://www.phoronix.com/news/F2FS-Case-Folding-Patches
Apparently, only enabled for f2fs
partitions during format.
https://cs.android.com/search?q=content:%22needs_casefold%22&sq=
It should be possible to check if its enabled on ext4
userdata
partition with dumpe2fs -h
. For f2fs
, /system/bin/dump.f2fs
can be run with root, but not sure if lists features, like dumpe2fs
. But if its enabled, running touch testfile; test -f TESTFILE && echo 1
should print 1
.
It is also possible to disable it on an empty directory with chattr -F
without having to format to ext4
. The lsattr
command can be used to check if F
is enabled. The /system/bin/chattr
binary on A11 supports the option. I don't have a f2fs
partition to check, I forced formatted my lineage A11 to ext4
, it defaults to f2fs
and android avd images for A11 and A13 are ext4
. Could create an image though and see, although don't really have time. Maybe termux-app
should check on new installs as well and disable it on files
dir. Others with f2fs
/data
partition should check though. mount | grep " /data "
should show partition type.
https://manpages.debian.org/buster/e2fsprogs/chattr.1.en.html
Apparently, case-insensitive/
casefold
support inf2fs
was added by google for android.
This however is a somewhat recent change, dating to 2019 (according to phoronix.com)
It should be possible to check if its enabled on
ext4
userdata
partition withdumpe2fs -h
.
It lists the features, however i don't know which one is it:
/system/bin/dump.f2fs
can be run with root, but not sure if lists features
Unsure if the issue is on my side, but there is no dump.f2fs
on my /system/bin
.
But if its enabled, running
touch testfile; test -f TESTFILE && echo 1
Uh-oh
It is also possible to disable it on an empty directory with
chattr -F
So a fix for the issue would be using chattr/lsattr before unpacking ndk-sysroot?
It lists the features, however i don't know which one is it:
Your Filesystem Features
does not list casefold
, so your file system is case sensitive (testfile
!= TESTFILE
) and how it should be for termux.
Unsure if the issue is on my side, but there is no dump.f2fs on my /system/bin.
You need to be inside a rooted shell to find and execute it since its owned by root
. Running sudo dump.f2fs
won't work either since search is made without root with my sudo
script. Its a symlink to fsck.f2fs
.
Maybe its not available on your android rom.
Uh-oh
Why? That's how it should be. If filesystem was case insensitive, then you would see 1
. I assume you are running on your new ext4
filesystem and have not reverted back to f2fs
.
So a fix for the issue would be using chattr/lsattr before unpacking ndk-sysroot?
Yes, preferably on empty /data/data/com.termux/files
directory before setting up termux bootstrap, so that other random things don't fail. But I need someone with f2fs
filesystem by default to do testing and confirm this.
Also note that don't format to ext4
on A11+
if your default is f2fs
and you don't know what you are doing. Filesystem and mount flags need to be changed in /vendor/etc/fstab.qcom
on your ROM and in /system/etc/recovery.fstab
and /system/etc/twrp.flags
in twrp recovery image, otherwise /data
partition won't mount in android or receovery and you will get in a bootloop for former.
I have an Android 12 device and I've done some tests.
Currently, /data
on my device is case-sensitive. And I mkdir -p test
on $HOME
, /system/bin/toybox chattr +F test
, and touch test/abc && touch test/ABC && echo 1
has output 1. So it seems that this check is not a valid one. Alternatively, (echo 123 > test/abc) && (echo 456 > test/ABC) && test $(cat test/abc) = '456'
exit with code 0. It may be able to do such kind of check.
You need to be inside a rooted shell to find and execute it since its owned by
root
.
Why? That's how it should be.
Never mind, i understood it wrong.
touch test/abc && touch test/ABC && echo 1
You will get 1
echoed because touch
command is going to succeed. You need to run test -f
or ls
on ABC
.
. So it seems that this check is not a valid one.
If your filesystem supports casefold
but its not enabled on a directory and you do chattr +F
, its going to get enabled on that directory (any any future sub directories) and 1
is supposed to be echoed for touch testfile; test -f TESTFILE && echo 1
Even ext4
partitions should support casefold
on Android 11+
for external storage via tune2fs
since sdcardfs
is deprecated. What is the filesystem type and features?
Since your second command works, we can assume +F
worked, since cat
of abc
and ABC
should return latest change.
https://source.android.com/docs/core/storage/sdcardfs-deprecate
/data
is f2fs
on my device. I forgot to say it. And touch testfile; test -f TESTFILE && echo 1
echoes 1 under the folder $HOME/test
.
Okay. Thanks. I assume it does not echo 1
if chattr -F
was run on an empty directory first.
Needs testing on another device?
Trying to find a way to get filesystem features of f2fs
, I don't think dump.f2fs
shows it as per quick look in source. And your version seems to be older than my fsck.f2fs 1.30.0 (2019-09-24)
.
dump.f2fs
was added in 2013
, so weird that it isn't available on your device.
ext4
casefold
enabling is done at https://cs.android.com/android/platform/superproject/+/android-12.0.0_r34:system/core/fs_mgr/fs_mgr.cpp;l=524
Needs testing on another device?
Sure.
Can you paste the output of mount
?
You asking me? Magisk has mounted on the binaries, so versions may vary from AOSP.
mount | grep f2fs
/dev/IuwAO/.magisk/block/system_root on /system/bin/fsck.f2fs type ext4 (ro,seclabel,relatime,discard)
/dev/IuwAO/.magisk/block/system_root on /system/bin/make_f2fs type ext4 (ro,seclabel,relatime,discard)
/dev/IuwAO/.magisk/block/system_root on /system/bin/sload_f2fs type ext4 (ro,seclabel,relatime,discard)
No lol, I mean those who use f2fs. Need to start somewhere to investigate...
Apparently f2fs_io
command should show flags for file, including casefold
, but f2fs-tools
package isn't available in termux
and command is not available on Android.
https://manpages.debian.org/testing/f2fs-tools/f2fs_io.8.en.html
I compiled it from source for aarch64
in termux from https://android.googlesource.com/platform/external/f2fs-tools/+/refs/heads/master/README
So running ./f2fs_io getflags /path/to/mount_point
should show the flags.
Attaching binary in later comment. Also should be possible to add package to termux but failed to build entire package in termux, due to possibly missing libselinux1-dev
, but ran make
in tools/f2fs_io
and it succeeded.
apt install autoconf automake binutils libtool libuuid
./autogen.sh
./configure
cd tools/f2fs_io
make
./f2fs_io
Unfortunately, on my A11 device, if I format f2fs partition with casefold
, it fails to mount. Will try with avd later. Note that there is minimum limit for f2fs
image, 30mb failed, had to use 100mb.
# dd if=/dev/zero of=./f2fs.img bs=1M count=0 seek=100
0+0 records in
0+0 records out
0 bytes copied, 5.9167e-05 s, 0.0 kB/s
# make_f2fs -f -O casefold ./f2fs.img
F2FS-tools: mkfs.f2fs Ver: 1.13.0 (2019-09-24)
Info: Disable heap-based policy
Info: Debug level = 0
Info: Trim is enabled
Info: No support kernel version!
Info: Segments per section = 1
Info: Sections per zone = 1
Info: sector size = 512
Info: total sectors = 204800 (100 MB)
Info: zone aligned segment0 blkaddr: 512
Info: [./f2fs.img] Discarding device
Info: fallocate(PUNCH_HOLE|KEEP_SIZE) is failed
Info: Overprovision ratio = 25.000%
Info: Overprovision segments = 22 (GC reserved = 16)
Info: format successful
# losetup /dev/block/loop0 ./f2fs.img
# mkdir -p /mnt/f2fs_mount
# mount -v -t f2fs /dev/block/loop0 /mnt/f2fs_mount
try '/dev/block/loop0' type 'f2fs' on '/mnt/f2fs_mount'
mount: '/dev/block/loop0'->'/mnt/f2fs_mount': Invalid argument
# dmesg | grep CONFIG_UNICODE
F2FS-fs (loop0): Filesystem with casefold feature cannot be mounted without CONFIG_UNICODE
# cat /proc/config.gz | gunzip | grep CONFIG_UNICODE
# CONFIG_UNICODE is not set
# cat /sys/fs/f2fs/features/casefold
supported
Can you paste the output of
mount
?
Since the XT1032 is now using EXT4, here's the output of my main device (Redmi 10, Android 12):
eos:/ $ mount | grep f2fs
/dev/block/dm-11 on /data type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,gc_merge,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,extent_cache,mode=adaptive,active_logs=6,reserve_root=23833,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier)
/dev/block/dm-11 on /data/user/0 type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,gc_merge,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,extent_cache,mode=adaptive,active_logs=6,reserve_root=23833,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier)
/dev/block/dm-11 on /data_mirror/data_ce/null type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,gc_merge,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,extent_cache,mode=adaptive,active_logs=6,reserve_root=23833,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier)
/dev/block/dm-11 on /data_mirror/data_ce/null/0 type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,gc_merge,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,extent_cache,mode=adaptive,active_logs=6,reserve_root=23833,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier)
/dev/block/dm-11 on /data_mirror/data_de/null type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,gc_merge,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,extent_cache,mode=adaptive,active_logs=6,reserve_root=23833,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier)
/dev/block/dm-11 on /data_mirror/cur_profiles type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,gc_merge,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,extent_cache,mode=adaptive,active_logs=6,reserve_root=23833,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier)
/dev/block/dm-11 on /data_mirror/ref_profiles type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,gc_merge,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,extent_cache,mode=adaptive,active_logs=6,reserve_root=23833,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier)
Weird that another device I have (GT-S6293T) have almost everything on EXT4:
rootfs / rootfs ro,relatime 0 0
tmpfs /dev tmpfs rw,nosuid,relatime,mode=755 0 0
devpts /dev/pts devpts rw,relatime,mode=600 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
none /acct cgroup rw,relatime,cpuacct 0 0
tmpfs /mnt/asec tmpfs rw,relatime,mode=755,gid=1000 0 0
tmpfs /mnt/obb tmpfs rw,relatime,mode=755,gid=1000 0 0
none /dev/cpuctl cgroup rw,relatime,cpu 0 0
/dev/block/mmcblk0p17 /system ext4 ro,noatime,data=ordered 0 0
/dev/block/mmcblk0p15 /efs ext4 rw,nosuid,nodev,noatime,discard,journal_checksum,journal_async_commit,data=ordered 0 0
/dev/block/mmcblk0p16 /cache ext4 rw,nosuid,nodev,noatime,discard,journal_checksum,journal_async_commit,data=ordered 0 0
/dev/block/mmcblk0p19 /data ext4 rw,nosuid,nodev,noatime,discard,journal_checksum,journal_async_commit,noauto_da_alloc,data=ordered 0 0
/sys/kernel/debug /sys/kernel/debug debugfs rw,relatime 0 0
/data/media /storage/sdcard0 sdcardfs rw,nosuid,nodev,relatime,uid=1023,gid=1023 0 0
Also, the binary you sent, could you also built it for ARM? I can't build f2fs_io for some reason...
~/f2fs-tools $ autoconf
configure.ac:15: error: possibly undefined macro: AM_SILENT_RULES
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
configure.ac:38: error: possibly undefined macro: AM_INIT_AUTOMAKE
configure.ac:67: error: possibly undefined macro: AM_PROG_AR
configure.ac:235: error: possibly undefined macro: AM_CONDITIONA
Try installing automake
Building... I'll update this comment with the output once done.
I have fixed the instructions, don't run autoconf
, the autogen.sh
will automatically do that and run automake
.
configure.sh gave error:
~/f2fs-tools $ ./configure
checking whether make supports nested variables... yes
checking for a BSD-compatible install... /data/data/com.termux/files/usr/bin/install -c
checking whether build environment is sane... yes
checking for a race-free mkdir -p... /data/data/com.termux/files/usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking how to create a pax tar archive... gnutar
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking whether gcc understands -c and -o together... yes
checking whether make supports the include directive... yes (GNU style)
checking dependency style of gcc... gcc3
checking for ar... no
checking for lib... no
checking for link... link -lib
checking the archiver (link -lib) interface... unknown
configure: error: could not determine link -lib interface
Run apt install binutils
for ar
.
While running make_f2fs -f -O casefold ./f2fs.img
I got this error:
~/.../tools/f2fs_io $ sudo make_f2fs -f -O casefold ./f2fs.img
Error: Wrong features casefold
Usage: mkfs.f2fs [options] device [sectors]
[options]:
-a heap-based allocation [default:0]
-c [device path] up to 7 devices excepts meta device
-d debug level [default:0]
-e [cold file ext list] e.g. "mp3,gif,mov"
-E [hot file ext list] e.g. "db"
-f force overwrite the exist filesystem
-g add default options
-i extended node bitmap, node ratio is 20% by default
-l label
-m support zoned block device [default:0]
-o overprovision ratio [default:5]
-O feature1[feature2,feature3,...] e.g. "encrypt"
-q quiet mode
-R root_owner [default: 0:0]
-s # of segments per section [default:1]
-S sparse mode
-t 0: nodiscard, 1: discard [default:1]
-w wanted sector size
-z # of sections per zone [default:1]
-V print the version number and exit
sectors: number of sectors. [default: determined by device size]
May not be supported by your old version of make_f2fs
in /system/bin
. Could try building mkfs.f2fs
from mkfs
sub directory or build image on linux distro since maybe libselinux1-dev
is required to build from source.
https://cs.android.com/android/platform/superproject/+/master:external/f2fs-tools/mkfs/
I've never built C programs before, how do i compile it for arm?
Same commands on arm device should do local compilation. Otherwise, linux distros have f2fs-tools
package that you can install to create image on pc. Maybe installing the package in proot distro in termux works too.
proot distro?
Ideally, f2fs-tools
package should be provided by termux if some package maintainer can spend time on that.
Will need package build dependency libselinux1-dev
as well.
https://cs.android.com/android/platform/superproject/+/master:external/selinux/README.android
I'll install it and update you very soon.
Guess I have a bad karma today:
[*] Checking integrity, please wait...
[*] Extracting rootfs, please wait...
proot info: vpid 1: terminated with signal 11
[*] Writing '/data/data/com.termux/files/usr/var/lib/proot-distro/installed-rootfs/debian/etc/profile'...
/data/data/com.termux/files/usr/bin/proot-distro: line 375: /data/data/com.termux/files/usr/var/lib/proot-distro/installed-rootfs/debian/etc/profile: No such file or directory
I've also attempted with Ubuntu and Void, same error.
You should first check if your arm
device kernel actually supports casefold
. Your post lists 3.4.113
as kernel version, which may be too low for casefold
, unless it was backported by lineage. Something else could be the issue too that got fixed by moving from f2fs
to ext4
, since other people are facing same issue. It was added in 5.11
in linux mainstream, may have been added earlier in android fork.
How can I check it?
cat /proc/config.gz | gunzip | grep CONFIG_UNICODE
cat /sys/fs/f2fs/features/casefold
~/.../tools/f2fs_io $ cat /proc/config.gz | gunzip | grep CONFIG_UNICODE
cat: /proc/config.gz: No such file or directory
gzip: stdin: unexpected end of file
~/.../tools/f2fs_io $ cat /sys/fs/f2fs/features/casefold
cat: /sys/fs/f2fs/features/casefold: No such file or directory
Also, my friend built mkfs
for me, where should I place the file [built]?
What's output of ls /sys/fs/f2fs/features
?
cat: /proc/config.gz: No such file or directory
May not be enabled for your kernel build. Not available in android 5 avd either, or under /boot
.
Also, my friend built mkfs for me, where should I place the file [built]?
Just execute it with ./mkfs.f2fs
from current dir. Could place in $PREFIX/bin
too for $PATH
.
What's output of
ls /sys/fs/f2fs/features
?
Empty.
Just execute it with
./mkfs.f2fs
I'll sleep now, it's a bit late for me, but we can try testing more tomorrow.
Empty.
Should check with root shell, possibly casefold
may not be supported.
I also checked and even though userdata
may support casefold
, AOSP only runs chattr +F
on /data/media
i.e external storage by default and not on /data/data
or /data/user/<user_id>
, which may create lot of other issues. A user could still themselves run chattr +F
on a directory, or some vendor may do it accidentally or for some reason. We should wait for a report for a vendor that is actually doing it before a check makes sense to be added to the app.
Considering that even though casefold
could be an issue on some newer devices causing such problems, it doesn't look like OP or other old arm devices had that as the issue and something else is causing the issue. Maybe the kernel/filesystem driver is borken or something. Since the issues is always the same file or files with different case, could try manually extracting them to same directory and opening them.
I can't check now, but
Since the issues is always the same file or files with different case, could try manually extracting them to same directory and opening them. I've also tried manually unpacking but didnt worked either.
I can't check now, but
Since the issues is always the same file or files with different case, could try manually extracting them to same directory and opening them.
I've also tried manually unpacking but didnt worked either.
I've also tried manually unpacking but didnt worked either.
You mean extracting the deb file tar files or installing with dpkg -i
?
Can post /system/bin/lsattr $PREFIX/include/linux/netfilter | sort
for those that use f2fs?
You mean extracting the deb file tar files or installing with
dpkg -i
?
Both. Extracting the .deb worked fine however.
Should check with root shell, possibly
casefold
may not be supported.
Can post
/system/bin/lsattr $PREFIX/include/linux/netfilter | sort
for those that use f2fs?
Want me to reflash /data as F2FS to post?
Problem description
I've tried to install Python, but one of it dependencies (ndk-sysroot) is giving out this error while installing:
I've already tried to troubleshoot the problem with commands like
apt --fix-broken install
,apt purge
,apt clean
,apt autoclean
,dpkg -i --force-overwrite /data/data/com.termux/files/usr/tmp/apt-dpkg-install-aTTKAF/07-ndk-sysroot_25b-2_arm.deb
, and even reinstalling termux, but no success.Any ideas of what can be causing this issue?
What steps will reproduce the bug?
pkg i python
What is the expected behavior?
Python, and it's dependencies (ndk-sysroot) to be installed successfully.
System information
termux-info:
Using Lineage OS 17 (with Android 10), Magisk, termux:boot is installed (both installed from f-droid).