pbatard
commented
5 years ago Thanks for logging an issue.
As I explained in the e-mail I sent earlier, the "tool" we use to create the ext file system is the exact same code as Linux uses which comes from e2fsprogs and which is maintainer by Theodore Ts'o, who is the main developer for the ext file systems on Linux. So, in terms of "tool", we could hardly be closer to what Linux uses for ext file system creation than we already are. Especially it should be pointed out that, because e2fsprogs has some Windows support, the core of the code was barely modified for our usage. As a matter of fact, if you clone the repository I linked to above and compare it to the code used by Rufus, you'll see that the only differences are for the backend. So really, for all intent and purposes, the code used by Rufus and the code used by the native Linux tools to create an ext file system is virtually identical and comes straight from same people.
Also, as opposed to what many other utilities do, and since Rufus doesn't have an installer in the first place anyway, we don't use external DLLs apart from the system ones (but of course, Windows does not have a system DLL that can be used to format a drive to ext). As a result, anything we reuse is pretty much the same as the official code, which we compile within the application itself. So I hope this will finish to convince you that, no, the issue is not the result of some "lazy" reuse of a dodgy DLL or us trying to (poorly) reinvent the wheel. Instead we spent a lot of effort making sure that we used the code that is closest to what Linux uses. Plus, you have to be very conscious that because Rufus is GPLv3, we can't really go around and reuse elements from closed source utilities, such as AOMEI Parition Assistant, which, last time I checked wasn't Open Source at all (please be mindful of confusing "freeware" with "Open Source", as it's not because something is gratis that it is actually free in the sense of software freedom).
Therefore, our expectation has been that, considering that the only code that we changed was for the I/O backend, and that this I/O backend doesn't appear to have much of an issue when we're writing root inodes and other stuff, the various ext2fs initialization calls we make, which again are supposed to perform the exact same operations as they do on Linux, might result in a file system that is properly initialized. If anything, this would lead us to think that the issue might be with the official code, whose Windows side probably hasn't received as much love as the Linux side, which might explain why the created file system trips e2fsck.
Which means that we now have to analyse the internals of ext2fs file system creation, to figure out why code that, for all intent and purposes, we do expect to behave in the same manner regardless of the platform, might not, and this is going to take a while since we have to familiarize ourselves with exactly the kind of stuff we were trying to avoid (i.e. getting super-knowledgeable about ext2/ext3 intrinsics) when we chose to go with "the reference" for formatting an ext file system.
Realistically then, unless someone helps us with that (for instance by providing some reports of "Rufus writes data X into inode Y at position Z whereas it should write data X' into inode Y' at position Z'", or even better, explaining precisely what we might be missing in the code to make our sequence of calls to what should be the same functions as mkfs.ext3 calls on Linux work), I expect that troubleshooting the issue is likely to take a couple of months at best, because figuring out what exactly might be wrong with ext bitmaps or inodes is not something you can achieve by taking a shortcut.
Also, as I mentioned elsewhere, whereas I did indeed see e2fsck errors (mostly Block bitmap differences on pass 5), I didn't really consider them a major issue considering that we have yet to receive a single report of effective data issues for ext partitions created with Rufus and I've seen some reports from Theodore Ts'o advising people with similar errors that "block allocation bitmap differences and free block/inode accounting information being wrong is normal" under some circumstances and that it is "unlikely (to be) anything serious".
All this to say that, whereas we will be investigating the issue to see if we can fix the e2fsck results, it's unlikely to happen as soon as you wish it will and that if some Linux folks, with knowledge of the ext2/ext3 intrinsics, want to shed some technical light as to what exactly e2fsck seems to complain about, and what we might be missing from our ext initialization (which, it should be pointed out, is not directly derived from mke2fs since, whereas the parts of the e2fsprogs code we reuse are licensed under LGPL, which allows us to include them into a GPLv3 application, unless told otherwise, mke2fs itself is licensed under GPLv2 only, which is not compatible with GPLv3, so we did have to take some guesswork about what needs to be called during ext init) we would appreciate it.
sudodus
MatthewForrester
marcosfrm
lock[bot]
Checklist
<FULL LOG>below.Rufus version: x.y.z- I have NOT removed any part of it.Additionally (if applicable):
[x] I ran a bad blocks check, by clicking Show advanced format options then Check device for bad blocks, and confirmed that my USB is not defective.
[x] I also tried one or more of the following:
[x] If using an image, I clicked on the
(✓)button to compute the MD5, SHA1 and SHA256 checksums, which are therefore present in the log I copied. I confirmed, by performing an internet search, that these values match the ones from the official image. I tested the checksum of the iso file separately at/after downloadingActually a got it via zsync, which has a built-in checksum test.
Issue description
When creating persistent a live drive Rufus makes an ext3 file system in the partition for persistence. This file system has errors. It is still possible to use the file system for example with Lubuntu 19.10, but
Part of the drive space in the file system is lost, wasted, compared to a healthy file system, when Lubuntu or other flavours of Ubuntu is booted. This might not be seen when connected to an operating system that is already running. The size of the lost drive space is small in a 4 GB pendrive, significantly bigger in a 16 GB pendrive, and huge ina 60 GB SSD. I used an SSD in several test runs because the loss is so big, that it very easy to see, and also because it is a high speed device, so that I could do several tests within a rather short time.
The errors in the file system may not create any corruption of files in the beginning, but I am afraid, that it may cause severe problems later on. When booting live-only I noticed that a log directory was mounted (and some of the errors are that some inodes have 2 links).
I tested various things in three different computers and another user tested it in a fourth computer. I ran standard Windows 10 fully up to date in two computers and a 'Windows Insider' version 10 fully up to date in the third computer. After some repetitions things started to look good, but when I specified the starting conditions in a strict way I could reproduce similar (not exactly the same) size of lost drive space.
In these cases between 20 GiB and 40 GiB were lost, already marked as used by df -h when booting into the persistent live system for the first time. A single test starting from exFAT showed 12 GiB lost drive space.
Thinking aloud
It seems to me that the tool to create the file system is buggy, or is using some buggy or not compatible system software (dll-file?).Or maybe you could fix it by compiling with some flags for more strict, less optimized for speed operation. Or you could test if there is a version for ext2 or ext4, that is more robust. I think that ext4 gets much more attention than ext3 nowadays.
Repair
I tested that it was possible to repair the file system with e2fsck -f and then there was no lost drive space, only the normal amount reserved to management (and similar size as that of the linux tool mkusb).
Re-format
I tested also that it was possible to re-format the ext3 file system with AOMEI Parition Assistent (the freeware version), and that file system was working almost correctly, there was a "Resize inode not valid" message. But no other complaint, not the huge amount of errors caused by Rufus, and there was no obvious loss of drive space for the user.
Final words
I wish that you will fix this bug or find a way around it soon, and I am willing to help with testing, when you have new versions of Rufus or simply new ways to set the conditions to make it work.
Good luck Nio
Log