Closed andrewdbate closed 2 years ago
What is Rufus doing when it formats the NTFS partition that is different to the Format dialog in Windows Explorer?
Nothing. Rufus is calling on the same formatting APIs that Windows uses, so formatting using the default dialog or Rufus should not produce any difference in the file system itself.
However, one thing that you may not be doing is ensuring that the NTFS file system is properly unmounted before you eject the drive (because a drive that was not cleanly unmounted leads to all kind of issues, though I'd expect UEFI:NTFS to complain then rather than the Windows installer), which is part of the reason why Rufus calls on checkdisk after it's done. That's the NTFS Fixup (Checkdisk)...
part you see in the log.
IIRC, I had some issues with some NTFS formatted drives unless I forces a checkdisk, but that was long before I added the UEFI:NTFS functionality. Or it may be that you're missing some files when extracting the content or something.
Also, if you're trying to replicate the creation of Windows bootable media from Linux, you may want to have a look at what WoeUSB does, because they've definitely managed to sort out your issue, and you might be able to find precisely what step you are missing in their source.
Some things I have double-checked:
chkdsk
on the drive after copying files, and it reported no errors.diff -r
and the comparison is identical.I have been trying to compare the NTFS file systems as created by Rufus through the Windows API vs that created by the Format dialog screenshotted above because in my opinion there has to be some difference (otherwise I would not get the BSOD each time).
Here is what I have done:
I created a USB flash drive with Rufus as described above. Then I plugged the USB into a Linux computer and deleted all files on the NTFS partition with rm -rf *
. Then I cloned the partition with ntfsclone
for later inspection by running the following command (where sdb1
is the NTFS partition) :
$ sudo ntfsclone --save-image --output rufus-ntfs-no-files.img /dev/sdb1
I then plugged the drive into the Windows machine, and formatted the partition with the Format dialog shown above. (making sure to eject the drive properly etc as mentioned above). I then plugged the USB into the Linux computer, ran rm -rf *
on the NTFS partition as before (which will delete the System Volume Information
directory as before), and cloned the NTFS partition as created by the Format dialog to compare against the one created by Rufus:
$ sudo ntfsclone --save-image --output windowsformat-ntfs-no-files.img /dev/sdb1
Now I copy the contents of the Windows ISO mentioned above to this partition. Attempting to boot on the ThinkPad T440+dock results a BSOD.
Now I restore the (empty) partition as created by Rufus on the Linux machine with:
$ sudo ntfsclone --restore-image --overwrite /dev/sdb1 rufus-ntfs-no-files.img
Now I copy the contents of the Windows ISO exactly as before. Attempting to boot on the ThinkPad T440+dock works as expected.
(I also repeated the above several times. Each time I would restore windowsformat-ntfs-no-files.img
and then copy the Windows ISO files the system would BSOD, and each time I would restore rufus-ntfs-no-files.img
the system would boot.)
Therefore whether the BSOD occurs or not is unrelated to the copying of files (because it is done in the same way both times) and I have double checked that all files are copied. There has to be something different about the NTFS partitions, but I do not understand enough about NTFS to debug this.
I have attempted to use ntfscmp
to compare the two NTFS partitions but I am unable to interpret the output.
Here are the two image files that I refered to above: rufus-ntfs-no-files.zip windowsformat-ntfs-no-files.zip
Do you have any suggestions for next steps? Can you recommend any tools to compare the two images?
So the problem is not how I am copying the files to the partition.
How can we be sure? You have not stated exactly how you are extracting the files from the ISO then copying the files to the drive.
Rufus runs with administrative rights. Are you? Rufus does not add any fancy permissions to the files / folders. Does your process?
With WoeUSB we use Linux to copy the files because we know that it will be a clean copy to NTFS; no inherited permissions, hidden streams or security attributes which may be created by Windows Explorer, Command Prompt or any other process launched from your working Windows environment.
The BSOD suggests that a file on the drive is either missing, inaccessible or corrupted. Take another look at your method of mounting the ISO, extracting its contents and transferring the files to the drive. You may find that one of these is the culprit rather than a false assumption that the format is somehow wrong.
I should have made it clear that I extracted the ISO on Linux and copied the files there.
Please note also that I followed the same steps to copy the files to the USB after restoring the partition with:
$ sudo ntfsclone --restore-image --overwrite /dev/sdb1 rufus-ntfs-no-files.img
And then it booted successfully.
Hence if the steps that I took to copy the files was the problem, then I should not be able to boot after restoring the partition created by Rufus (at least that is my reasoning).
I have also done a diff -r
after copying to make sure that everything was indeed copied correctly.
Again, WoeUSB manage to do exactly what you're doing without producing a BSOD on Linux, so there has to be something in what you're doing that deviates from the normal process.
I can guarantee that Rufus is not doing anything "special" when formatting to NTFS, as we are just asking the regular Windows APIs to format to NTFS for us. There's no fine tuning of NTFS involved, because that's not something the Windows APIs offer.
Considering that you have evidence that two completely separate utilities (WoeUSB and Rufus), running on 2 separate OSes are able to perform as expected, but yours doeesn't. I don't think you should be asking if these other utilities are doing anything special, but you will really need to dig in at what you're doing to find out what it is that you are doing wrong, because all evidence points to your method doing something wrong somewhere, that you have simply not identified yet.
And I hope you can appreciate that there's only so much we can do to try to help you here, when there's nothing special we're doing when it comes to creating the NTFS file system, and it very much looks like you're still looking in the wrong direction to identify your issue if you are trying to pin the problem on the file system.
Windows için Postahttps://go.microsoft.com/fwlink/?LinkId=550986 ile gönderildi
Kimden: Andrew D @.> Gönderilme: 14 Ocak 2022 Cuma 05:48 Kime: @.> Bilgi: @.***> Konu: Re: [pbatard/rufus] Question: How does Rufus format the NTFS partition? Is this different to the Windows Format dialog? (Issue #1851)
I should have made it clear that I extracted the ISO on Linux and copied the files there.
Please note also that I followed the same steps to copy the files to the USB after restoring the partition with:
$ sudo ntfsclone --restore-image --overwrite /dev/sdb1 rufus-ntfs-no-files.img
And then it booted successfully.
— Reply to this email directly, view it on GitHubhttps://github.com/pbatard/rufus/issues/1851#issuecomment-1012700739, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKUSKTHMCGYMW3QDCFMKZSLUV6FG3ANCNFSM5L2GAHDA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you are subscribed to this thread.Message ID: @.***>
Lll,i,i4 5,4 5
Windows için Postahttps://go.microsoft.com/fwlink/?LinkId=550986 ile gönderildi
Kimden: Andrew D @.> Gönderilme: 14 Ocak 2022 Cuma 05:48 Kime: @.> Bilgi: @.***> Konu: Re: [pbatard/rufus] Question: How does Rufus format the NTFS partition? Is this different to the Windows Format dialog? (Issue #1851)
I should have made it clear that I extracted the ISO on Linux and copied the files there.
Please note also that I followed the same steps to copy the files to the USB after restoring the partition with:
$ sudo ntfsclone --restore-image --overwrite /dev/sdb1 rufus-ntfs-no-files.img
And then it booted successfully.
— Reply to this email directly, view it on GitHubhttps://github.com/pbatard/rufus/issues/1851#issuecomment-1012700739, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKUSKTHMCGYMW3QDCFMKZSLUV6FG3ANCNFSM5L2GAHDA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you are subscribed to this thread.Message ID: @.***>
Thank you for your advice. I will look further into the source of WoeUSB (although it is very strange that I only get the BSOD on one machine only out of those I have tried).
So it turns out that WoeUSB does not work with gpt tables. It uses parted
in the script, and parted
is unable to create the NTFS:UEFI partition at the requested offsets. Attempting to do so at the same offsets used by the WoeUSB script results in the error with parted
:
Error: You requested a partition from 16.2GB to 16.2GB (sectors 31700992..31703039).
The closest location we can manage is 16.2GB to 16.2GB (sectors 31700992..31703006).
I am commenting on this here because for me to be able to compare the Rufus vs WoeUSB for the purposes of figuring out what could be causing the above behaviour I need to compare like for like. I can't compare a USB with a GPT table vs a USB with an MBR table.
However, I will keep digging and report back.
So it turns out that WoeUSB does not work with gpt tables. It uses parted in the script, and parted is unable to create the NTFS:UEFI partition at the requested offsets.
This is completely untrue. I use WoeUSB
to create GPT drives that are practically identical to those produced by Rufus. FYI, parted
is perfectly capable of creating both a MBR table ((parted) mklabel msdos
) and GPT types ((parted) mklabel gpt
),
If you look at the WoeUSB
source, you will see that it initialises the drive as GPT, then creates the main NTFS partition starting at 4Mb, with 1Mb at the end where it then creates the UEFI:NTFS partition and dd
's the image there.
Please run WoeUSB
in it's entirety before making false claims. And continue this discussion at WoeUSB's issue tracker as this is no longer related to Rufus
' (and this issue will probably be closed and locked by @pbatard as a result!).
@JonnyTech Here is the relevant extract from the script:
case "${partition_table_type}" in
legacy|msdos|mbr|pc)
parted_partiton_table_argument=msdos
;;
gpt|guid)
parted_partiton_table_argument=gpt
print_error \
'Currently GUID partition table is not supported.\n'
return 2
;;
*)
print_error \
'Partition table not supported.\n'
return 2
;;
esac
So no it does not support GPT!
There is clearly a problem here. You seem to think I'm an incompetent when I am actually trying to debug this so that other people can benefit.
@JonnyTech Yes, parted
can create GPT partition tables (yes I do know that!) but the WoeUSB script cannot.
Please at least do a basic check of what you are doing with WoeUSB before stating that I am making false claims. Perhaps you could run the script yourself and check the output? What makes you think that I have not done that?
I am using the WoeUSB 5.2.4 released version.
But I was hoping to be able to find the issue here because I am somewhat convinced there is some issue here, but I am indeed struggling to pin it down. I was hoping for some assistance rather than rudeness.
I am closing this because obviously there is no appetite here to discuss this. If I ever find out the cause, then I will report back in case this benefits someone else.
@andrewdbate my apologies, you are correct, I was thinking of my fork which does support GPT but it was rejected because WoeUSB wants to suppprt booting from all systems hence the MBR/GRUB layout. Note to self: I must tidy my fork and resubmit a PR to allow creating GPT only drives via a flag. Second note to self: don't reply to GitHub issues from the pub on a Friday night, after a beer. Sorry again if I was rude. Please try modifying the source snippet you quoted to allow the GPT flag (and IIRC prevent GRUB installing). The resulting drive will work, I use it myself regularly.
@andrewdbate, I am interested in knowing why your method is failing once you have that answer. But I'm afraid I am not interested in helping you figure that out when we're really not doing anything special in terms of NTFS formatting.
The log you copy/pasted shows exactly what we're doing there, which is to invoke the system NTFS formatting API, like the default format dialog would do, and using quick format:
Formatting to NTFS (using IFS)
Using cluster size: 4096 bytes
Quick format was selected
Creating file system...
Format completed.
And you can indeed see every operation Rufus is carrying out in the log, so I would concentrate on finding out how you deviate from that when you are carrying them out manually. Note that you'll see that we are disabling file indexing on the file system but I wouldn't spend much time trying to pin your issue on that. For one thing, I would drop the use of ntfsclone
altogether, because if you are suspecting Rufus to do something "special" then you have to be as suspicious of any third utility you use when performing your tests.
For one thing, I would drop the use of ntfsclone altogether, because if you are suspecting Rufus to do something "special" then you have to be as suspicious of any third utility you use when performing your tests.
Very true.
I have also created binary images of the partition with dd
instead of ntfsclone
and the same thing happens.
The only reason I uploaded the image of the partition created with ntfsclone
here is because it is smaller and I was hoping someone sufficiently knowledgeable could have looked at them and identified what the difference is. I do not have sufficient knowledge of NTFS to do this.
The ntfscmp
tool does indeed report differences, but I am unable to interpret the output of ntfscmp
myself.
I have also attempted to format the partition on Linux with mkntfs
with and without the --no-indexing
option, but it makes no difference (i.e., I get the BSOD in both cases).
Also, I would not be so certain that the Format dialog in Windows is going to be doing exactly the same thing as Rufus does with the Windows API. That is not at all obvious to me.
Here are the two image files that I refered to above: rufus-ntfs-no-files.zip windowsformat-ntfs-no-files.zip
Just had a few minutes to look at these. The two filesystem do indeed appear to be identical. The only difference I see is that Windows performed a full format so only the System Volume Information
folder is present, whereas Rufus must have done a quick format because entries for all Windows installer files (about 2000 of them) are shown. Compare the info outputs: compare.zip
Are you able to do a similar image comparison of whole drive and not just the NTFS partition? I suspect that the issue may be in the partition table or elsewhere, although if you state that you only changed the NTFS partition while leaving everything else intact then I am at a loss.
Unfortunately I do not have access to a Windows system, but I tried my fork of WoeUSB (that sets the partition table to GPT and does not install GRUB) and I ended up with a bootable drive that starts the Windows installer successfully. I also hacked together a simple bash script to partition, format and copying the files and that booted too.
At which point exactly did you get the BSOD? Could this be an issue that is isolated to just your computer?
@JonnyTech Thank you very much for taking the time to look at this.
I do believe that this issue is somehow related to this individual machine. It only occurs on my ThinkPad T440 when connected to the dock. It does not occur on the other computers I have. (I really am just trying to figure out what is going wrong on this particular computer so that any tool that I create/contribute to will work for everyone else because USBs created with Rufus do work on that system.)
The BSOD occurs immediately after the loading screen, i.e. the screen that looks some like this:
I do not get to see the "Windows Setup" screen, i.e. the BSOD occurs before the screen that would look something like this:
(Note: screenshots were taken from the web just as examples.)
Could you please point me to your fork of WoeUSB that creates a USB with a GPT table?
When I used WoeUSB, which created an MBR partition table, it worked on this troublesome ThinkPad T440 + dock computer. When I tweaked the script myself to use a GPT table, I got a BSOD. However, although I only changed type of partition table that the script created (and adjusted the partition offsets for the UEFI:NTFS partition because of the GPT secondary data at the end of the drive), perhaps I did something wrong, and I would like to try your fork.
I am more than happy to create an image of the entire USB. So that I know what to provide, shall I create it with dd
? Is there anything else specifically that you'd like me to do?
Many thanks again.
It only occurs on my ThinkPad T440 when connected to the dock. It does not occur on the other computers I have.
Ah, that is very specific indeed. Is it fine without the dock? Have you looked at BIOS updates / settings?
Could you please point me to your fork of WoeUSB that creates a USB with a GPT table?
It is a local fork. I shall upload it when I get to my main development computer, in the next few days. It may be easier for me to post my script instead..
I am more than happy to create an image of the entire USB. So that I know what to provide, shall I create it with
dd
? Is there anything else specifically that you'd like me to do?
That may help you to compare structures, but I am not in a position to download 32Gb, sorry. Once you create images (or using the drives themselves), I suggest booting them with a virtual machine and seeing what happens there. Then tweak VM settings until you can duplicate the fault.
I never get the BSOD when the computer is not docked (e.g., using the USB stick prepared with Rufus but with the NTFS partition reformatted with the Windows Format dialog, it works fine). But when it is in the dock I get the BSOD with the same USB stick each time. When I use the USB stick prepared with Rufus (but the NTFS partition not reformatted), I never get the BSOD.
The ThinkPad T440 is running the latest BIOS firmware from Apr 2020 (version 2.54).
Unfortunately, I have not been able to reproduce it in a VM (I have tried).
I shall be interested to try your script once you make it available.
From memory, I have created this simple gist - it should create a GPT only USB drive from Linux
(I have deleted my previous comment because I realised that I made a mistake in what I wrote. Here is the correction.)
Thank you for posting the script. I have commented on the gist about a warning that I got from dd
because of the incorrect offset calculation, but this is easily fixed.
When I used the script as-is (i.e. with the incorrect offset), I got the same behaviour as I have been commenting above. The ThinkPad T440 would boot from the USB when it was not in the dock, but when it was connected to the dock it would refuse to boot. This is the behaviour I have been reproducing consistently all along.
I then noticed that when I created the USB with the script I had got the following warning from dd
when writing the UEFI:NTFS partition:
dd: writing to '/dev/sdb2': No space left on device
2016+0 records in
2015+0 records out
1031680 bytes (1.0 MB, 1008 KiB) copied, 0.181442 s, 5.7 MB/s
This had been caused because the size of the UEFI:NTFS partition was 2015 sectors instead of the required 2048. This was due to the script using offset -2048s
. However, this offset does not account for the last 33 sectors being reserved for the GPT secondary data.
I then edited the script to use the correct offset for the UEFI:NTFS partition of -2081s
(because 2048+33=2081).
Unfortunately, after rechecking my steps it seems that the ThinkPad T440 still has issues booting from the USB when connected to the dock even with the correct offsets used.
(I do not know why the computer being connected to the dock makes a difference, but it does.)
Thank you again @JonnyTech for posting the script.
If I take JonnyTech's gist with the original offsets (that don't account for the GPT secondary data) and change the partition table type to MBR, and keep everything the same, then the ThinkPad T440 will boot even when in the dock.
@andrewdbate thank you for the comment about the oversight with the end sectors, I shall update my script accordingly. Glad that you found your issue, although it is still odd that Rufus prepared drives work correctly.
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue if you think you have a related problem or query.
Checklist
<FULL LOG>
below.Rufus version: x.y.z
- I have NOT removed any part of it.Additionally (if applicable):
(✓)
button to compute the MD5, SHA1 and SHA256 checksums, which are therefore present in the log I copied. I confirmed, by performing an internet search, that these values match the ones from the official image.Issue description
I have a question (rather than an issue) about Rufus. I hope that is okay.
I am trying to reverse engineer Rufus' workings so that I can port a subset of the functionality to Linux (mostly the ability to create bootable USBs with the UEFI:NTFS bootloader).
I have read the FAQ, readme, issue tracker, the relevant source code, and I now reasonably understand what is involved. I now have determined (what I think is) the sequence of steps needed to create such as USB on Linux, however, I have encountered one issue that totally has me stumped.
I cannot understand why I am observing the following behaviour:
If I create a bootable USB from
Win10_21H2_EnglishInternational_x64.iso
downloaded from the Microsoft website here, using Rufus with the default options, everything works as expected when attempting to boot from the USB with my ThinkPad T440 connected to its docking station. So far so good (so Rufus works as expected).However, if I take this USB drive, and then format the NTFS partition from Windows Explorer (by right-clicking the drive, choosing
Format...
, and formatting with NTFS accepting the defaults), and then copy the contents of the ISO to the USB, when I attempt to boot from the USB on the ThinkPad T440 with the dock I get a BSOD (blue screen of death) with error code0xc000021a
. See attached screenshot.I have tried this multiple times, with different USB memory sticks, and the same behaviour always reproduces.
So my question is this: What is Rufus doing when it formats the NTFS partition that is different to the
Format
dialog in Windows Explorer?Other things I have tried to figure out what is going on
To try to narrow down what is the cause, I created a bootable Windows USB using Rufus as above with the same ISO, then deleted all files on the NTFS partition, and then extracted
Win10_21H2_EnglishInternational_x64.iso
to the partition (in order to test the file copying, or whether Rufus was adding/removing some files). The ThinkPad T440+dock is able to boot from this no problem.So the problem is not how I am copying the files to the partition. It must be in the formatting of the partition. It is only when I first reformat the NTFS partition in Windows with the Format dialog and then extract the ISO that I get the above BSOD.
I have also tried formatting the partition with
mkfs.ntfs
on Linux before copying the Windows ISO files, but I still get the BSOD on that one machine. I have also tried formatting to NTFS with GParted before copying the Windows ISO files, but I still get the BSOD. Clearly Rufus must be doing something different when it formats to NTFS.Therefore Rufus must be doing something special/magic/different when it creates the NTFS partition vs what Windows does from the
Format...
dialog.I can see from the source code and the log that Rufus uses the IFS API, but I would have naively assumed that the Format dialog would have done something similar.
Summary
I appreciate that this is not a bug in Rufus (because Rufus works as expected), however, I could not find any documentation for what Rufus is going that is special/different, nor could I see anything relevant in the source code when I read what I thought was the relevant parts.
I would appreciate any advice because I need to recreate the same behaviour if I am to port to Linux.
Log