Improve FileSystem detection

macgills commented 4 years ago

Is your feature request related to a problem? Please describe. Currently filesystem detection can take upwards of 1min and cause problems for users #1662 . This is because we write a file 4GB in size and see if it throws an exception or not.

Describe the solution you'd like Use the mount command and do some string processing to extract the filesystem

Another ticket proposing a cache system #1664

kelson42 commented 4 years ago

Describe the solution you'd like Use the mount command and do some string processing to extract the filesystem

@macgills How would that work exactly?

macgills commented 4 years ago

 try{
        Process mount = Runtime.getRuntime().exec("mount");
        BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(mount.getInputStream()));
        mount.waitFor();

        String extPath = Environment.getExternalStorageDirectory().getAbsolutePath();
        String line;
        while ((line = bufferedReader.readLine()) != null)
        {
            String[] split = line.split("\\s+");
            for(int i = 0; i < split.length - 1; i++)
            {
                if(split[i].contentEquals(extPath) ||
                    split[i].contains("sdcard") ||
                    split[i].contains("_sd") ||
                    split[i].contains("extSd") ||
                    split[i].contains("_SD"))   // Add wildcards to match against here
                {
                    String strMount = split[i];
                    String strFileSystem = split[i+1];

                    // Add to a list/array of mount points and file systems here...
                    Log.i("SDCard", "mount point: "+ strMount + " file system: " + strFileSystem);
                }
            }
        }           
    }catch(IOException e){
        e.printStackTrace();            
    }catch(InterruptedException e){
        e.printStackTrace();            
    }

like this but a bit more complicated. Execute mount command Parse lines of text into models iterate over models finding partial matches on the current preferred storage select best matching model check the filesystem string again a list of known FSs that can support 4GB report that the FS can write 4GB files

kelson42 commented 4 years ago

@macgills So the mount command gives you the list of mounted devices and for each devices the type of the filesystem and you will the be able to know which one are fat32. Questions:

Are you really sure you have the always the right to execute that command?
Are you sure the mount output is always the same (depending the version Android)?
Why this has not been done before (this seems simple)?
On linux, you have a similar result with cat /proc/mounts, if it works might be better to just read this (there is a class here https://gist.github.com/ikonst/3394662)?

macgills commented 4 years ago

I think so
not positive
the rest of these answers
none of the SO answers I have seen have talked about that, they start talking about vold sometimes though

When I implemented an experiment it didn't give me an immediately easy answer say for example we have mount output like so

somedevice realdir exfat
realDir sdCardDir fake

when I iterated through the list the entry I matched on for the directory I was looking for sdcardDir/probably/some/other/path was the 2nd one but it didn't have a valid entry for a file system so I had to redo the search using the realDir to retrieve the result of exfat/CanWrite4GB

kelson42 commented 4 years ago

@macgills I'm not super confident on relying on a system command execution, but if we rely on a Linux Kernel /proc file then I'm more confident. If you can deal with one of these files /proc/mounts or /proc/self/mountinfo I would be quite supportive to try to implement that ticket in 3.2. See also: https://stackoverflow.com/questions/11668284/is-there-any-limitations-in-using-proc-mounts-in-android

macgills commented 4 years ago

It'll take experimentation to figure out which is most reliable, mount probably just reads proc/mounts, if proc/self/mountinfos exists and can be read by us then it sounds like a good option

macgills commented 4 years ago

And to clarify I will be keeping the old implementation, just trying this one first and on failure falling back. Commonsware back in 2014 was warning of these apis when android introduced support for sd cards

kelson42 commented 4 years ago

@macgills OK, seems worth the effort.

macgills commented 4 years ago

Logs extracted on Nexus 8. The storage we are trying to identify, note this is on the device

E/Storage: /storage/emulated/0/Android/data/org.kiwix.kiwixmobile/files

Output of procs/self/mountinfo

E/MOUNTINFO: 316 316 0:1 / / ro master:1 - rootfs rootfs ro,seclabel
E/MOUNTINFO: 317 316 0:12 / /dev rw,nosuid,relatime master:2 - tmpfs tmpfs rw,seclabel,size=923024k,nr_inodes=230756,mode=755
E/MOUNTINFO: 318 317 0:9 / /dev/pts rw,relatime master:3 - devpts devpts rw,seclabel,mode=600
E/MOUNTINFO: 319 317 0:16 / /dev/cpuctl rw,relatime master:4 - cgroup none rw,cpu
E/MOUNTINFO: 320 317 0:17 / /dev/cpuset rw,relatime master:5 - cgroup none rw,cpuset,noprefix,release_agent=/sbin/cpuset_release_agent
E/MOUNTINFO: 321 317 0:20 / /dev/usb-ffs/adb rw,noatime master:20 - functionfs adb rw
E/MOUNTINFO: 322 316 0:3 / /proc rw,relatime master:6 - proc proc rw,gid=3009,hidepid=2
E/MOUNTINFO: 323 316 0:13 / /sys rw,relatime master:7 - sysfs sysfs rw,seclabel
E/MOUNTINFO: 324 323 0:11 / /sys/fs/selinux rw,relatime master:8 - selinuxfs selinuxfs rw
E/MOUNTINFO: 325 323 0:5 / /sys/kernel/debug rw,relatime master:9 - debugfs debugfs rw,seclabel
E/MOUNTINFO: 326 323 0:18 / /sys/fs/pstore rw,relatime master:10 - pstore pstore rw,seclabel
E/MOUNTINFO: 327 316 0:14 / /acct rw,relatime master:11 - cgroup none rw,cpuacct
E/MOUNTINFO: 328 316 0:15 / /mnt rw,relatime master:12 - tmpfs tmpfs rw,seclabel,size=923024k,nr_inodes=230756,mode=755,gid=1000
E/MOUNTINFO: 329 328 0:19 / /mnt/runtime/default/emulated rw,nosuid,nodev,noexec,noatime master:21 - sdcardfs /data/media rw,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid
E/MOUNTINFO: 330 328 0:19 / /mnt/runtime/read/emulated rw,nosuid,nodev,noexec,noatime master:21 - sdcardfs /data/media rw,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=23,derive_gid
E/MOUNTINFO: 331 328 0:19 / /mnt/runtime/write/emulated rw,nosuid,nodev,noexec,noatime master:21 - sdcardfs /data/media rw,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=7,derive_gid,default_normal
E/MOUNTINFO: 332 316 0:10 / /config rw,relatime master:13 - configfs none rw
E/MOUNTINFO: 333 316 253:0 / /system ro,relatime master:14 - ext4 /dev/block/dm-0 ro,seclabel,inode_readahead_blks=8
E/MOUNTINFO: 334 316 253:1 / /vendor ro,relatime master:15 - ext4 /dev/block/dm-1 ro,seclabel,inode_readahead_blks=8
E/MOUNTINFO: 335 316 259:8 / /cache rw,nosuid,nodev,noatime master:17 - ext4 /dev/block/platform/soc.0/f9824900.sdhci/by-name/cache rw,seclabel,noauto_da_alloc,errors=panic,data=ordered
E/MOUNTINFO: 336 316 179:24 / /persist rw,nosuid,nodev,noatime master:18 - ext4 /dev/block/platform/soc.0/f9824900.sdhci/by-name/persist rw,seclabel,nodelalloc,errors=panic,data=ordered
E/MOUNTINFO: 337 316 179:1 / /firmware ro,relatime master:19 - vfat /dev/block/platform/soc.0/f9824900.sdhci/by-name/modem ro,context=u:object_r:firmware_file:s0,uid=1000,gid=1000,fmask=0337,dmask=0227,codepage=437,iocharset=iso8859-1,shortname=lower,errors=remount-ro
E/MOUNTINFO: 338 316 253:2 / /data rw,nosuid,nodev,noatime master:16 - ext4 /dev/block/dm-2 rw,seclabel,noauto_da_alloc,errors=panic,data=ordered,inode_readahead_blks=8
E/MOUNTINFO: 339 316 0:15 /runtime/write /storage rw,relatime master:12 - tmpfs tmpfs rw,seclabel,size=923024k,nr_inodes=230756,mode=755,gid=1000
E/MOUNTINFO: 340 339 0:19 / /storage/emulated rw,nosuid,nodev,noexec,noatime master:21 - sdcardfs /data/media rw,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=7,derive_gid,default_normal
E/MOUNTINFO: 341 339 0:15 /user/0 /storage/self rw,relatime master:12 - tmpfs tmpfs rw,seclabel,size=923024k,nr_inodes=230756,mode=755,gid=1000

Output of proc/mounts

E/MOUNTS: rootfs / rootfs ro,seclabel 0 0
E/MOUNTS: tmpfs /dev tmpfs rw,seclabel,nosuid,relatime,size=923024k,nr_inodes=230756,mode=755 0 0
E/MOUNTS: devpts /dev/pts devpts rw,seclabel,relatime,mode=600 0 0
E/MOUNTS: none /dev/cpuctl cgroup rw,relatime,cpu 0 0
E/MOUNTS: none /dev/cpuset cgroup rw,relatime,cpuset,noprefix,release_agent=/sbin/cpuset_release_agent 0 0
E/MOUNTS: adb /dev/usb-ffs/adb functionfs rw,noatime 0 0
E/MOUNTS: proc /proc proc rw,relatime,gid=3009,hidepid=2 0 0
E/MOUNTS: sysfs /sys sysfs rw,seclabel,relatime 0 0
E/MOUNTS: selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0
E/MOUNTS: debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0
E/MOUNTS: pstore /sys/fs/pstore pstore rw,seclabel,relatime 0 0
E/MOUNTS: none /acct cgroup rw,relatime,cpuacct 0 0
E/MOUNTS: tmpfs /mnt tmpfs rw,seclabel,relatime,size=923024k,nr_inodes=230756,mode=755,gid=1000 0 0
E/MOUNTS: /data/media /mnt/runtime/default/emulated sdcardfs rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid 0 0
E/MOUNTS: /data/media /mnt/runtime/read/emulated sdcardfs rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=23,derive_gid 0 0
E/MOUNTS: /data/media /mnt/runtime/write/emulated sdcardfs rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=7,derive_gid,default_normal 0 0
E/MOUNTS: none /config configfs rw,relatime 0 0
E/MOUNTS: /dev/block/dm-0 /system ext4 ro,seclabel,relatime,inode_readahead_blks=8 0 0
E/MOUNTS: /dev/block/dm-1 /vendor ext4 ro,seclabel,relatime,inode_readahead_blks=8 0 0
E/MOUNTS: /dev/block/platform/soc.0/f9824900.sdhci/by-name/cache /cache ext4 rw,seclabel,nosuid,nodev,noatime,noauto_da_alloc,errors=panic,data=ordered 0 0
E/MOUNTS: /dev/block/platform/soc.0/f9824900.sdhci/by-name/persist /persist ext4 rw,seclabel,nosuid,nodev,noatime,nodelalloc,errors=panic,data=ordered 0 0
E/MOUNTS: /dev/block/platform/soc.0/f9824900.sdhci/by-name/modem /firmware vfat ro,context=u:object_r:firmware_file:s0,relatime,uid=1000,gid=1000,fmask=0337,dmask=0227,codepage=437,iocharset=iso8859-1,shortname=lower,errors=remount-ro 0 0
E/MOUNTS: /dev/block/dm-2 /data ext4 rw,seclabel,nosuid,nodev,noatime,noauto_da_alloc,errors=panic,data=ordered,inode_readahead_blks=8 0 0
E/MOUNTS: tmpfs /storage tmpfs rw,seclabel,relatime,size=923024k,nr_inodes=230756,mode=755,gid=1000 0 0
E/MOUNTS: /data/media /storage/emulated sdcardfs rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=7,derive_gid,default_normal 0 0
E/MOUNTS: tmpfs /storage/self tmpfs rw,seclabel,relatime,size=923024k,nr_inodes=230756,mode=755,gid=1000 0 0

output of mount command

E/COMMAND: rootfs on / type rootfs (ro,seclabel)
E/COMMAND: tmpfs on /dev type tmpfs (rw,seclabel,nosuid,relatime,size=923024k,nr_inodes=230756,mode=755)
E/COMMAND: devpts on /dev/pts type devpts (rw,seclabel,relatime,mode=600)
E/COMMAND: none on /dev/cpuctl type cgroup (rw,relatime,cpu)
E/COMMAND: none on /dev/cpuset type cgroup (rw,relatime,cpuset,noprefix,release_agent=/sbin/cpuset_release_agent)
E/COMMAND: adb on /dev/usb-ffs/adb type functionfs (rw,noatime)
E/COMMAND: proc on /proc type proc (rw,relatime,gid=3009,hidepid=2)
E/COMMAND: sysfs on /sys type sysfs (rw,seclabel,relatime)
E/COMMAND: selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
E/COMMAND: debugfs on /sys/kernel/debug type debugfs (rw,seclabel,relatime)
E/COMMAND: pstore on /sys/fs/pstore type pstore (rw,seclabel,relatime)
E/COMMAND: none on /acct type cgroup (rw,relatime,cpuacct)
E/COMMAND: tmpfs on /mnt type tmpfs (rw,seclabel,relatime,size=923024k,nr_inodes=230756,mode=755,gid=1000)
E/COMMAND: /data/media on /mnt/runtime/default/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid)
E/COMMAND: /data/media on /mnt/runtime/read/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=23,derive_gid)
E/COMMAND: /data/media on /mnt/runtime/write/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=7,derive_gid,default_normal)
E/COMMAND: none on /config type configfs (rw,relatime)
E/COMMAND: /dev/block/dm-0 on /system type ext4 (ro,seclabel,relatime,inode_readahead_blks=8)
E/COMMAND: /dev/block/dm-1 on /vendor type ext4 (ro,seclabel,relatime,inode_readahead_blks=8)
E/COMMAND: /dev/block/platform/soc.0/f9824900.sdhci/by-name/cache on /cache type ext4 (rw,seclabel,nosuid,nodev,noatime,noauto_da_alloc,errors=panic,data=ordered)
E/COMMAND: /dev/block/platform/soc.0/f9824900.sdhci/by-name/persist on /persist type ext4 (rw,seclabel,nosuid,nodev,noatime,nodelalloc,errors=panic,data=ordered)
E/COMMAND: /dev/block/platform/soc.0/f9824900.sdhci/by-name/modem on /firmware type vfat (ro,context=u:object_r:firmware_file:s0,relatime,uid=1000,gid=1000,fmask=0337,dmask=0227,codepage=437,iocharset=iso8859-1,shortname=lower,errors=remount-ro)
E/COMMAND: /dev/block/dm-2 on /data type ext4 (rw,seclabel,nosuid,nodev,noatime,noauto_da_alloc,errors=panic,data=ordered,inode_readahead_blks=8)
E/COMMAND: tmpfs on /storage type tmpfs (rw,seclabel,relatime,size=923024k,nr_inodes=230756,mode=755,gid=1000)
E/COMMAND: /data/media on /storage/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=7,derive_gid,default_normal)
E/COMMAND: tmpfs on /storage/self type tmpfs (rw,seclabel,relatime,size=923024k,nr_inodes=230756,mode=755,gid=1000)

All 3 outputs are much the same for mountinfo we would find this entry

E/MOUNTINFO: 340 339 0:19 / /storage/emulated rw,nosuid,nodev,noexec,noatime master:21 - sdcardfs /data/media

It gives us a useless value for the filesystem sdcardfs but gives us a directory /data/media which when we scan for again gives us

E/MOUNTINFO: 338 316 253:2 / /data rw,nosuid,nodev,noatime master:16 - ext4 /dev/block/dm-2 rw,seclabel,noauto_da_alloc,errors=panic,data=ordered,inode_readahead_blks=8

which gives us the useful value of ext4 which is a real filesystem and does support 4gb files.

I'll probably read from proc/mounts as it has the least fluff in the way of the data I want eg device, mount point and filesystem

kiwix / kiwix-android

Improve FileSystem detection #1663