search

pbatard / rufus

The Reliable USB Formatting Utility
https://rufus.ie
GNU General Public License v3.0
29.26k stars 2.59k forks source link

UEFI:NTFS - Unable to start driver: [1] or Could not start NTFS partition service: [14] #1213

Closed zanechua closed 4 years ago

zanechua commented 6 years ago

Checklist

Additionally (if applicable):

Issue description

Motherboard: S2600CP4 Bios Revision: R02.06.E006 Rufus version: 3.3

Here's the log from Rufus: https://gist.github.com/zanechua/36c2416c9d8035623a0efb51a176d7af

Here are the errors I am getting on boot: error_1

error_2

EDIT:

So it seems like the last version of Rufus v2 works.

Rufus version: 2.18

Here's the log from Rufus: https://gist.github.com/zanechua/acf8c67c7026814796f7da6414dd6e3e

success

You are mentioning that you went through a BIOS update, so I wonder if that update is preventing EFI executables with IMAGE_SUBSYSTEM_EFI_BOOT_SERVICE_DRIVER from running.

I don't think it's the bios update though since 2.18 works. I haven't formatted this computer in awhile so I might have been using 2.18 and this is the first time I am using 3.x on this computer.

devedse commented 4 years ago

@pbatard , For some reason if I format my USB Stick using UEFI:NTFS option I get the following error (Rufus 3.10):

image

This works fine in Rufus 3.9:

image

pbatard commented 4 years ago

Can you please try some more with 3.10. The UEFI:NTFS creation process has actually not changed between 3.9 and 3.10, in that we are creating 2 partitions, and just dd'ing 512 KB of data onto the smaller one. In other words, none of the data that was changed for UEFI:NTFS between 3.9 and 3.10 can have an impact on the drive creation, because it belongs in the 512 KB, which Rufus doesn't care one bit about (it just writes it as is) and is treated in the same fashion.

You may want to try Atl-V with toggles VDS for formatting, to see if that helps, but there's really nothing that has changed with the manner in which Rufus creates UEFI:NTFS drives between 3.9 and 3.10, so it's probably a transient error, as I have never seen an issue like the one you report on my side. You may also try Atl-Z to zero the drive for a little while before you attempt to create the UEFI:NTFS drive...

pbatard commented 4 years ago

@devedse, any chance for an update again?

I'd like to release a bugfix version of Rufus, so I really would like to have some results from your tests with 3.8, 3.9 and 3.10 when creating UEFI:NTFS drives in the manner I highlighted above:

  1. Choose UEFI:NTFS option in the Boot selection dropdown
  2. Extract the files from efi.zip onto the main NTFS partition (which should be empty apart from the Rufus icon).

Note that I updated the efi.zip archive (in both posts), as it didn't contain quite exactly the files I wanted you to test with, though this wasn't exactly a big deal.

What I am interested is the output you get from your platform when trying to boot these drives. Preferably, I would like you to attempt to boot these drives without going through the Add Boot Option utility you seem to be using, as I get a feeling that it might interfere with the boot, and if you do set the proper boot order in the UEFI firmware (or just temporarily disable AHCI/SATA/NVMe there so that only USB should boot), you shouldn't have to do that at all.

Thanks!

devedse commented 4 years ago

@pbatard , I think somehow my USB Stick got corrupted as I can't format it using windows or DISKPART anymore either.

I now tried with a new USB Stick:

I formatted it using Rufus 3.10, placed the complete EFI directory on the main partition and booted it. I did this using this procedure: https://youtu.be/iRyV4h4DTRQ

This is the result: https://youtu.be/Ra7u8Y5O_LU

What I don't completely understand now is that we have an EFI\boot folder on both the UEFI disk and the main partition. Is this the thing you want?

What I also tried was simply overwriting the files on the UEFI partition with the ones in the EFI.zip file you sent and that resulted in the following: https://youtu.be/DRkyNNmGllo

As you can see, in both these scenario's I do need to manually add the option to boot using this specific UEFI configuration, because if I don't it simply won't boot.

I'll do the first thing using Rufus 3.9 and Rufus 3.8 as well.

devedse commented 4 years ago

I tried Rufus 3.9 too, but the Bios didn't seem to want to boot from UEFI at all from this stick: https://youtu.be/BDeR_SRl9-o

I thought this might've been a fluke so I retried the whole process and got the same result: https://youtu.be/bLqQ1WPUx5s

I then tried Rufus 3.8 which works fine: https://youtu.be/vzKiqIVPDHE

In the last video I also found out that one boot options gets added if UEFI seems to be correct, which is what I tried booting from in the video of 3.8 at the end. This also worked fine. (But doesn't seem to get automatically selected when just sticking in a USB stick)

Hopefully this all helps you debug the problems :)

pbatard commented 4 years ago

Thanks for the testing. Let me first address your first post.

What I don't completely understand now is that we have an EFI\boot folder on both the UEFI disk and the main partition. Is this the thing you want?

Yes, absolutely.

The whole point of UEFI:NTFS is to have a bootloader on a small FAT partition (the one you point to) that loads the NTFS driver and then chain load the actual bootloader on the NTFS partition, so, yes, you definitely want to see a /efi/boot/bootx64.efi on both partitions.

What you ultimately want to do is execute the EFI bootloader that resides on the NTFS partition, so that's why if you don't have a /efi/boot/bootx64.efi, then there's no point (though of course, if you're just checking for failures of the UEFI:NTFS bootloader then you could theoretically test without the NTFS boot loader, but in that case UEFI:NTFS will always fail even when it should work).

In other words the boot process goes like this:

  1. The system (generally) can't boot the NTFS bootloader directly because it can't read NTFS, so it looks for /efi/boot/bootx64 on the small FAT partition (which it can always read) and execute it. This is the UEFI:NTFS bootloader that is provided by Rufus.
  2. The UEFI:NTFS bootloader above attempts to load the missing NTFS driver and, if successful, it hands over the boot process to the /efi/boot/bootx64.efi that resides on the NTFS partition. This usually is the boot loader provided by the Windows ISO (i.e. the Windows installer) or, in our case, since we don't want to have to extract a Windows ISO every time, the /efi/boot/bootx64.efi that I provided in the efi.zip archive above.

What I also tried was simply overwriting the files on the UEFI partition

Please do NOT do that. You should NOT touch the small UEFI_NTFS partition that already contains the /Efi and /Rufus folders. If you do that, then you will effectively erase the UEFI:NTFS bootloader with something that is not UEFI:NTFS (the bootloaders from efi.zip simply display a message, they are not "test" versions of UEFI:NTFS that you should replace the existing UEFI:NTFS with).

As you can see, in both these scenario's I do need to manually add the option to boot using this specific UEFI configuration, because if I don't it simply won't boot.

And now I realize that your boot selection window is not a Microsoft Windows application, but a UEFI menu. So yes, that's what you should use to select the device you want to boot from (though I'm a bit puzzled that Dell forces you to point to a specific file instead of just automatically looking for a boot partition, which is how UEFI is set to proceed... count on Dell to make it a lot less user intuitive than needs to be).

pbatard commented 4 years ago

I have a question though: What happens if you simply select Stick2 without explicitly clicking Add boot option and picking a file (or leave the File name field empty)? Coz, once again, UEFI is designed to automatically locate a and boot a /efi/boot/bootx64.efi boot loader on any partition that it can read, so the one from the FAT partition should be picked automatically by a UEFI compliant system simply by telling it that you want to boot the USB device (the whole device, not an individual partition on that device), in UEFI mode.

pbatard commented 4 years ago

Okay, now to comment on your videos:

If possible, I'd be very curious about what happens if you run the UEFI Shell and try to load the v1.5 NTFS driver from there, and whether you'd see the same errors. Let me prepare a procedure to guide you through that.

pbatard commented 4 years ago

Okay, let's try this test:

  1. Download shell.vhd.gz (no need to extract it once downloaded). This is a disk image with everything you need.
  2. Use it in Rufus to create a drive like you would use any other .iso. Don't worry, Rufus can handle a .gz files. Once done, you will end up with a small drive that contains 2 small partitions, one FAT and one NTFS.
  3. Boot the drive (by pointing to the /efi/boot/bootx64.efi file on the FAT partition if needed
  4. This should open the UEFI Shell. In the Shell environment type:
    • map → This lists all the drives (FS#: that you can access)
    • ver → This provides some information about your UEFI environment.
    • FS#: then dir, with # being replaced by a number from 0 to 9 until you see ntfs-g.efi and ntfs-e.efi being listed from the dir command.
    • load ntfs-e.efi → This loads the EDK2-compiled version of the NTFS file system driver
    • map -r → This lists all the drives and re-maps new one. Don't forget to add the -r option. You should see new FS#: drive(s) being listed from the previous map command, for the NTFS partition. Make sure to remember their numbers.
    • For each new drive that appeared above, type FS#: where # is one of the numbers you picked above then dir until you see a rufus.txt file being listed.
    • Finally type type rufus.txt (yes, type is also a commandline instruction you need to input here) → This should produce the output This is the NTFS volume.
  5. If there is any error with this procedure, then reboot the computer and repeat from step 3. but this time use load ntfs-g.efi to load the gnu-efi version of the driver instead of load ntfs-e.efi and see what happens.

Screenshots from any errors (no need to upload videos) as well as the ouptut of the ver and map commands would be appreciated.

I hope this doesn't sound too much like I'm just ordering you around (feel free to take your time if needed or ask for clarification), because I really appreciate your willingness to help and I am very grateful for the tests you've already performed today!

devedse commented 4 years ago

Sorry for the delayed response but I haven't really been able to find time to look into the issue further. I'll let you know once I've executed the steps you requested.

albertopasqualetto commented 4 years ago

The problem happens also to me, but with 3.9 works.

devedse commented 4 years ago

@pbatard , I've found some time to run through the scenario. I've tried to execute step 3: Boot the drive (by pointing to the /efi/boot/bootx64.efi file on the FAT partition if needed

However if I do this I only manage to see a black screen.

At the end of your post is stated that I should also try to booth with NTFS-G.efi. I've tried that too, but the same black screen appeared again. For NTFS-E.efi the same happened.

What am I doing wrong :smile:?

devedse commented 4 years ago

So if you want we can drop on discord and do some tests together if that's easier for you.

pbatard commented 4 years ago

@devedse, thanks for testing.

However if I do this I only manage to see a black screen.

Interesting. It means that you can't even execute a recent UEFI Shell v2.2 (because that's what the bootx64.efi file on the FAT partition really is), which would tend to indicate that this whole driver incompatibility could indeed have something to do with the suggested theory that the UEFI firmware from these Dell's (and other affected computers) is too old to provide some of the APIs that modern EDK2 relies on...

I need to think about this some more, as well as investigate what breaking changes might have occurred in the EDK2.

if you want we can drop on discord and do some tests together if that's easier for you.

That shouldn't be necessary. Fingers crossed, I'm going to get my hands on a machine that is affected by this issue sometime this week, which should greatly improve my ability to troubleshoot that error. I can always use test reports though, as you can never be too sure that something you're seeing on one machine will be exactly the same on another...

pbatard commented 4 years ago

DAMMIT!

Finally got that Dell machine (an Optiplex 7010), but it looks like it isn't affected by the issue. I tried to downgrade the BIOS just in case, but UEFI:NTFS still works fine with the EDK2 driver there.

However, testing with that machine is giving me some hints as to where the issue might stem from (but not how to fix it yet).

Especially from @devedse's initial report and from #1536, I can see that the Device Paths reported when the issue occurs are a bit unexpected.

For instance, @devedse reports seeing PciRoot(0x0)/Pci(0x19,0x0)/INT13(,0x81)/HD... for the USB drive @Chatplosion reports seeing PciRoot(0)/Pci(0x1D,0x0)/VenHw(AA7BA38A-DAFB-40C3-8D18-B55B39609EF7)/HD..., whereas on the 7010 I am seeing PciRoot(0)/Pci(0x1D,0x0)/USB(0x1,0x0)/USB(0x2,0x0)/HD..., which is the kind of path I'd expect for a USB device.

Moreover, according to this, VenHw(<GUID>) is used when the device path node is not defined by the UEFI specification such as for generic memory mapped controllers. And INT13 is a BIOS interface that is commonly used by devices to provide custom disk access. Both can be used, for instance, by RAID controller cards, since regular SATA wouldn't work.

It is however very puzzling to see either of the above being used for USB devices when UEFI should have a relatively solid implementation to handle USB devices...

Now, the NTFS driver itself should be entirely agnostic to the type of controller being used (the proof of that being that the gnu-efi compiled version of the same code works), but I'm starting to wonder if the EDK2 may not be doing some things behind the scenes where the type of controller being used to access USB devices does matter...

Anyway, while I am trying to source yet another Dell machine that might be able to replicate this issue, I have the following to ask:

  1. Can any of the people affected by this issue tell me if they added extra controller cards to their computer? Especially, did you add a USB 3.0 add-in card?
  2. Could you please run a test with the following image (just open it in Rufus and press START) and report the ouput: test1.vhd.gz

Thanks

Aftabnack commented 4 years ago

This thread helped me resolve my issue with doing this on ubuntu. I have documented it in detail here: https://askubuntu.com/a/1243565/1087068

pbatard commented 4 years ago

A-HA!

I finally got my hand on a problematic machine (Dell Optiplex 390), and now everything is crystal clear, as per pbatard/efifs#21.

The main cause of the problem is that some old UEFI firmwares don't provide a protocol that allows to convert a Device Path (a Device Path is a binary blob that UEFI uses to uniquely indentify a device such as a USB drive or a USB partition) to a Device Path string, which is what I decided to use as a unique identifier for partitions in EfiFs because that's what grub_device_open() takes as a parameter (and I am reusing the GRUB file system drivers).

In other words, it made a lot of sense to use a DevicePath string at the time, especially as, regardless of whether the UEFI firmware did provide a protocol to convert a Device Path to a string, I could always fall back to a function, DevicePathToStr(), that the gnu-efi library provides, that also does the same thing.

But then I added support for EDK2 compilation, skirted around the absence of DevicePathToStr() there, and, since all my testing seemed to work fine, because they were conducted on machines that had the Device Path to string conversion protocol, completely forgot that, unlike gnu-efi, EDK2 did not have a fallback in case the protocol was absent.

So, what happens on the problematic machines is that, when we try to bind the NTFS driver to a partition, the string conversion for the Device Path of that partition fails, and the driver says it can't open that partition (with error [3] Unsupported because the protocol we need is not supported on that platform). And that's the reason why UEFI:NTFS fails with the EDK2 version of the NTFS driver, but works with the gnu-efi version of the same driver, because in case of gnu-efi, if the protocol isn't there, we can still get the string we need by calling the gnu-efi specific function DevicePathToStr()...

I have to say, I am a bit ashamed that I didn't spot this earlier, because, of course, it makes total sense now that it's clear what happened, and I feel like I should have been able to figure this out, without having to go through the trouble of purchasing a couple of these old Dell systems.

Once I have figured out a good way to work around the issue of not being able to rely on Device Path string conversions in EfiFs, I will produce a v1.6 release of the EfiFs drivers and hopefully, this time, that should be the end of this obnoxious issue.

devedse commented 4 years ago

Amazing that you figured it out :), keep us posted!

pbatard commented 4 years ago

Okay, I have now updated the UEFI:NTFS image in Rufus and I will close this issue, as I expect it to be fixed for good.

For those who want to test, I have also uploaded an early TEST version of Rufus 3.11 with the updated UEFI:NTFS here.

startergo commented 4 years ago

Okay, I have now updated the UEFI:NTFS image in Rufus and I will close this issue, as I expect it to be fixed for good.

For those who want to test, I have also uploaded an early TEST version of Rufus 3.11 with the updated UEFI:NTFS here.

do you have the UEFI:NTFS as an img? I want to dd write the img

startergo commented 4 years ago

I just tried the NTFS.efi driver in OpenCore, which is an EDKII build. At boot in the EDK environment upon loading the driver I get:

OC: Driver NTFS.efi at 2 is being loaded...
OC: Driver NTFS.efi at 2 cannot be started - Incompatible Version!
pbatard commented 4 years ago

do you have the UEFI:NTFS as an img? I want to dd write the img

Since the software is 100% Open Source, the latest image can always be downloaded from the GitHub repo here

upon loading the driver I get:

You are most likely not using the proper driver from your arch.

I'm afraid this is NOT the right thread to ask for help about using the driver. This thread is about getting errors [1] or [14] or [3] when using the UEFI:NTFS version installed by Rufus. If you have unrelated questions, please create a thread either in the UEFI:NTFS project or the EfiFs project. Otherwise, I will have no choice but to lock this issue to prevent further unrelated issues.

DoSpamu commented 4 years ago

Thank you for this fix in the 3.11 ver. I haved the same problem on Dell and now all works well.

github-actions[bot] commented 3 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue if you think you have a related problem or query.