Closed zanechua closed 4 years ago
Thanks for opening a new issue.
As I mentioned in the previous one, I suspect that the problem comes from the EfiFs NTFS driver having switched from using the IMAGE_SUBSYSTEM_EFI_RUNTIME_DRIVER
PE type to IMAGE_SUBSYSTEM_EFI_BOOT_SERVICE_DRIVER
, which I suspect some UEFI Firmwares might not like. The problem however is that I had to change that attribute because it was creating issues with other UEFI firmwares, and IMAGE_SUBSYSTEM_EFI_BOOT_SERVICE_DRIVER
is the proper type to use for what UEFI:NTFS is doing.
You are mentioning that you went through a BIOS update, so I wonder if that update is preventing EFI executables with IMAGE_SUBSYSTEM_EFI_BOOT_SERVICE_DRIVER
from running.
Here's what you can do to validate that this is indeed the issue, using the USB you created with Rufus 3.3:
Download CFF Explorer from here. If you don't know which version you should pick, I'd recommend the CFF Explorer (x86 Version, stand-alone, Zip Archive) as this is smaller and it doesn't need to be installed.
Plug in the Flash Drive created by Rufus. Since you're running on Windows 10, you should be able to access both the partitions Rufus created on that drive, including the UEFI:NTFS. However, Windows may not mount it automatically, and Disk Management seems to have trouble displaying small partition to do the following with the GUI, in which case you will have to assign a letter manually using diskpart
. In case you cannot see the UEFI_NTFS
partition, you should:
diskpart
list vol
. This will list all the volumes available, and you should see something like Volume 7 UEFI_NTFS FAT Removable 512 KB Healthy
in your list. Take a note of the volume number for the UEFI_NTFS partition. sel vol #
where #
is the number above (e.g. sel vol 7
)assign letter=X
where X
is any drive letter that is not already in use. After that, you should be able to access the UEFI:NTFS partition from Windows Explorer. A screenshot of the procedure is also provided below:With both CFF Explorer and the UEFI:NTFS partiton available, launch CFF Explorer
Open EFI\Rufus\ntfs_x64.efi
on the UEFI:NTFS partition
On the left pane navigate to Nt Headers → Optional Header. This should allow you to view the Subsystem property and see that it is set to EFI Boot Driver:
Click on EFI Boot Driver and change it to EFI Runtime Driver and then click the Save button (select Yes when it asks you to overwrite the file)
Try to boot the USB again and see if that works better.
Of course, this is just one possibility, and I may be off mark (especially it's possible that I had already changed the subsystem type before 2.18, in which case the above probably won't help).
The other thing that I'd like you to try, to confirm that this has to do with the EFI driver, is to pick the EFI\Rufus\ntfs_x64.efi
from the 2.18 drive and copy it to the 3.3 drive (replacing the existing one) and see how that works. Alternatively, you can try each of the ntfs_x64.efi
from https://efi.akeo.ie/downloads/, starting with EfiFs 1.0 (and then selecting the x64
repository) to see which one works and which one doesn't, as this may help me pinpoint what the issue might be.
Thanks for the very very detailed guide on how to debug this specifically.
I changed the Subsystem header of the v3.3 ntfs_x64.efi file to EFI Runtime Driver like you said but got a different error this time.
However copying the file from 2.18 to the drive created by 3.3 allows the drive to boot perfectly fine.
The ntfs_x64.efi file from the 2.18 drive already has the Subsystem header set to EFI Boot Driver
Thanks for the test. I probably should have checked the driver from 2.18 before I asked you to run that test.
As I mentioned earlier, can I ask you to please try all of the drivers ntfs_x64.efi
from https://efi.akeo.ie/downloads/, starting from efifs-1.0/
and let me know the last version that works with your 3.3 flash drive?
Oh. I misssed that part sorry. Will get to it.
Tested the following: v1 (same hash as v2.18): boots fine v1.1: boots fine v1.2: errors out v1.3: errors out
Boot Images:
So it seems like somewhere between v1.1 -> v1.2 is where a breaking change happened.
Thanks for the update, but next time please don't edit a comment to add new data, as I didn't see your tests with the various version until now! It's much better to create a new entry, as I do get a notification.
There were a lot of changes between 1.1 and 1.2, and since you are the only person with a machine that can replicate the problem, this is probably going to take a lot of back and forth to troubleshoot, so it may take a while.
I guess the first thing I'm going to test, since 1.2 changed the compiler being used, it to try with a different compiler. I can not revert to the old one (Clang/C2) since support for it is being dropped by Microsoft in the latest version of Visual Studio, which is part of the reason why I switched compiler for the binary in the first place. Since version 1.2, the NTFS driver binary is compiled with the EDK2, so a good test will be to try non EDK2 versions.
If you look in https://rufus.ie/testing/ you'll see that I uploaded 2 drivers: ntfs_x64_gcc.efi
and ntfs_x64_msvc.efi
. These are the latest versions of the NTFS driver, but compiled with GCC and MSVC respectively. Can I please ask you to test each one (by removing the _gcc
or _msvc
part and replacing your existing ntfs_x64.efi
) and see how it goes?
Sure.
Can I verify that the hash of the ntfs_x64.efi file in the v3.3 of rufus is 872FC3150C9DC39470CF0E03732642789CF8F82F? and it's 47KB?
Both testing drivers work totally fine and boot up as they should. Only the one included with rufus v3.3 doesn't.
Many thanks for the new tests!
Yes, the sha1sum of ntfs_x64.efi
from Rufus 3.3 is 872fc3150c9dc39470cf0e03732642789cf8f82f
. And it's 46.3 KB in size. Basically, you can find the file included in Rufus if you extract it from the current uefi-ntfs.img
we have in our source tree at https://github.com/pbatard/rufus/tree/master/res/uefi (using something like 7-zip for instance).
It's quite interesting that the test MSVC version works, whereas the original one doesn't, as the EDK2 version used by Rufus uses MSVC for compilation behind the scenes.
So, just to make absolutely sure that this is not an EDK2 issue that has since been fixed, can I please ask you to try one more test? I have just recompiled a new EDK2 driver, which you can find as ntfs_x64_edk2.efi
under https://rufus.ie/testing/. Can you run the same test with that one as you did with the others, and confirm that it fails?
If it does, I guess I'll have to figure out the differences between the way the EDK2 invokes MSVC and the way the regular Visual Studio solution does (besides the use of gnu-efi, which I would assert should be the main difference), so that I can try to produce a driver that works.
Else, I guess I'm going to have to switch to compiling all the drivers with Visual Studio + gnu-efi instead of EDK2, which is fine, but which will increase the size of the binaries and therefore the size of Rufus (the reason I'm using EDK2 + MSVC is because it produces the smallest binaries). And it still won't tell me exactly what it is between your platform and these drivers that is causing this weird incompatibility...
I can confirm that the EDK2 driver fails.
If you need more information let me know and I'll do my best to provide it.
Thanks.
I think right now I need to process all this good data you provided, and come up with some new things to try, to zero the issue further (which probably won't happen before next week at this stage).
At any rate, your testing and your patience with this issue are much appreciated!
I can't reproduce this myself (of course...), but EDK2 does make quite a few drastic changes to the standard MSVC build environment in the interest of saving space. I'm going to go ahead and assume EDK2 itself is not at fault (meaning the headers as opposed to the gnu-efi ones) as I've used both for years with no issues. But comparing ntfs_x64_edk2.efi
vsntfs_x64_msvc.efi
/ntfs_x64_gcc.efi
, there are a number of differences the DXE core loader could be stumbling over in the EDK2 PE files compared to the MSVC/GCC ones:
/ALIGN:0x1000 /FILEALIGN:0x200
first and see if that helps. This is IMO the most likely culprit./BASE:0x180000000
just in case./SUBSYSTEM:EFI_BOOT_SERVICE_DRIVER,1.0
to correct this./RELEASE
to set a checksum, though this will most likely be wrong because EDK2 runs some post-build steps that alter the image. Use something like CFF Explorer to fix this as the very last step.SizeOfStackReserve
, SizeOfStackCommit
, SizeOfHeapReserve
and SizeOfHeapCommit
have all been zeroed. I have no idea if this matters for OP's BIOS (odds are it doesn't) but I would just set these to 0x100000
, 0x1000
, 0x100000
and 0x1000
respectively with a hex editor to be on the safe side.0x168
to get rid of this entry.To get most of the above in order you can put the following in the .inf
file (obviously applies only to that package - for 'affects everything' scope edit tools_def.txt
instead):
[BuildOptions]
MSFT:*_*_*_DLINK_FLAGS = /ALIGN:0x1000 /FILEALIGN:0x200 /BASE:0x180000000 /SECTION:.pdata,!D /MERGE:.rdata=.text /SUBSYSTEM:EFI_BOOT_SERVICE_DRIVER,1.0 /RELEASE
MSFT:*_*_X64_GENFW_FLAGS = --keepexceptiontable --keepzeropending --keepoptionalheader
@Mattiwatti, many thanks for the extended analysis. This should indeed be exceedingly helpful, and I'll try to produce and upload drivers that apply your suggestions, to see if we can validate your hypotheses.
Finally got around to compile a NTFS driver that includes @Mattiwatti's build options suggestion. So let's find out if it's really one of these compilation settings that is really causing the issue or something else.
The driver is called ntfs_x64_edk2_TEST1.efi
and can be found at the usual place.
@zanechua, if you can conduct a test with this driver, as you did with the other ones, I would realy appreciate it.
Also, for the record, the linker issued the following warning, which means that the /MERGE:.rdata=.text
was most likely ignored. But since the other parameters should have been applied that doesn't worry me too much:
LINK : warning LNK4253: section '.rdata' not merged into '.text'; already merged into '.data'
That warning is not an issue, it just means some other EDK2 build setting already applied /MERGE:.rdata=.data
instead. Both options will merge the .rdata section into another one, saving space, but neither of them is either harmful or useful as far as OP's issue goes.
The only difference is that /MERGE:.rdata=.text
will make .rdata contents executable, which is usually not a problem. /MERGE:.rdata=.data
will make .rdata contents writable, which can cause issues when debugging because .rdata is supposed to be for read-only data (hence the name) and any bug where you accidentally write to const
data won't be detected. If your debug build configuration does not merge .rdata into .data then this is not an issue.
Tested ntfs_x64_edk2_TEST1.efi and I still get the same error.
Thanks for the test, it was definitely worth a try. My guess would still have to be that EDK2 has deprecated APIs that gnu-efi hasn't, therefore I will replace all the x86 drivers with their gnu-efi version in the next release of Rufus. For the sake of not increasing the size of the Rufus executable too dramatically, I think I'll keep the ARM/EDK2 versions though, as I would surmise that any ARM UEFI firmware should be modern enough to work with drivers compiled with a recent EDK2.
Hi all,
Just an update, I'm also running into this error using my Dell Lattitude E6520. I'm currently trying to see if 2.18 also resolved this issue.
@devedse, can you please indicate if you're getting errorcode 1 or 14, or some other. Even if you are getting the same error, it always helps to copy the exact message you are getting, just to confirm that you are indeed seeing the same error.
Didn't manage to make a screenshot as I went back to 2.18 and that worked. I'll see if I can create a new usb stick with the 3.3 version.
Also one thing I don't quite understand is that I can select fat32 in Rufus 2.18 but not in 3.3.
I tried 2 diffent UEFI files on the disk, 1:
And 2:
(Just a black screen)
Well, for 2 that's a given. ntfs_x64.exi
is a driver, not a bootloader. You can't boot that.
As to the first test, you are getting a completely different error from the ones mentioned above. Interesting...
Also one thing I don't quite understand is that I can select fat32 in Rufus 2.18 but not in 3.3.
Well, UEFI:NTFS is of course only valid for NTFS, so obviously it makes no sense to have it with FAT. I sure hope you didn't use FAT32 for your 2.18 test, because then it makes your success there irrelevant.
Also, why does the boot option name in your first screen say WinUsb2
?
I'm sorry but your tests look super weird.
Care to post a full log of how you created the USB with Rufus 3.3?
I did use NTFS for both tests. The WinUsb2 is just a name that you can manually enter.
Full process would be:
Same for 2.18 but there it did work.
Still, I would really like to see the full log of you created the USB with Rufus 3.3. Remember, the more data I have, the more I am able to help you.
Will do tomorrow. :) thnx for the help already!
Appreciate that - thanks!
Let me know if you need anything else
Okay, I have now put a TEST version of Rufus 3.4, that contains the updated gnu-efi based NTFS drivers, that I expect to fix @zanechua's original issue.
Can you please download rufus-3.4_TEST1.exe
from here and report?
@pbatard sorry. give me a little bit. been busy with life.
Also one thing I don't quite understand is that I can select fat32 in Rufus 2.18 but not in 3.3.
Well, UEFI:NTFS is of course only valid for NTFS, so obviously it makes no sense to have it with FAT. I sure hope you didn't use FAT32 for your 2.18 test, because then it makes your success there irrelevant.
Btw, does this mean that FAT32 functionality was removed from Rufus 3.3? As I can't find the option anywhere to select it.
does this mean that FAT32 functionality was removed from Rufus 3.3?
Not at all. But obviously, Rufus removes the ability to create a FAT32 drive IF it detects that there is a >4GB file on your image, since it would only produce an error if you tried it (because you won't be able to copy a >4GB file onto a FAT32 file system without truncating it, which means your drive won't work).
For 2.18 and this file I could still select FAT32. But I expect that it would error out later in the process then?
You're most likely not using the same partition scheme and target in 2.18 as you do in 3.x. Please bear in mind that the defaults have changed between 2.18 and 3.0.
Anyway, if you think there is a problem with this, PLEASE open a new issue, as this is completely unrelated to the matter at hand, and this thread is long enough as it is without hijacking it for a matter that is not directly relevant.
@pbatard, I thought opening a completely new thread would also cause some overhead for a simple question.
Back on topic though, I just tested your test version and atleast in my case it works and boots to the Windows Setup.
I just tested your test version and atleast in my case it works and boots to the Windows Setup.
Great. Thanks for the test.
@pbatard I can confirm that 3.4-TESTING works on my machine. :)
Awesome! I will close this issue then, as the next release of Rufus should resolve the issue. Many thanks for your testing and your patience.
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue if you think you have a related problem or query.
Unlocking this thread to give a heads up to the people who were affected that I have switched the UEFI:NTFS file system driver to the EDK2 version in Rufus 3.10 BETA.
The reason is that I am hoping that the recent improvements and bugfixes that were applied to the file system drivers may fix the issue reported above, and using a BETA is a good way to confirm whether that is the case or not, so that, if the problems still persist, I can switch back to the gnu-efi version of the drivers in time for the release.
If you were affected by this, and if you still have an opportunity to test, may I ask you to please try to create an UEFI:NTFS bootable media with Rufus 3.10 BETA (download here), and report if you see a regression? Thank you.
I would love to test this for you but unfortunately the board I was using had a cap blow up and I haven't tried booting it since. If I can replace the cap I might be able to try but it probably won't be anytime soon. Unless @devedse can do it.
No worries. I still appreciate that you took time to reply.
Reopening this issue as I got a report from someone using Rufus 3.10 getting [FAIL] Unable to load driver '\efi\rufus\ntfs_ia32.efi': [3] Unsupported
on an Acer Iconia model W510p:
When using Rufus 3.9 (and therefore an older version of UEFI:NTFS with gnu-efi rather than EDK2 compiled drivers), the error goes away.
I sure wish I had a machine that displays the issue, so that I can figure out exactly why the EDK2 version of the drivers doesn't work, coz having a load failure with the EDK2 drivers when the gnu-efi version works and both drivers are ultimately compiled with the same compiler (VS2019) makes very little sense...
I still have the laptop. I can retest this later. Please keep tagging me and spamming me so I don't forget. :)
Hi guys I see that you are working on a bug which is apparently a incompatibility with different bootloader headers
Actually this new version of Rufus caused me this error and it says "Unable to start Driver" so I used Rufus 3.9 and it worked just fine.
Now that you are working on this I'll be glad to provide you with result of changes being made in order to solve this problem quickly.
So if you need any tests or stuff to develop the proper bootloader I'm all yours
@devedse, @farid220, thanks for offering to help - this is much appreciated.
One thing I am noticing is that the ia32 and aarch64 version of the EDK2 compiled drivers are missing cpu
, file-type
and subsystem
:
But the gnu-efi version (as well as the x64 and arm version of the EDK2 compiled drivers) have those filled as expected:
So, yeah, I can certainly imagine why a UEFI 32-bit platform, like the Acer Iconia model W510p might not want to load the drivers at all, especially if subsystem
is empty.
But if that is the issue, then the only people I expect to experience the [FAIL] Unable to load driver '\efi\rufus\ntfs_ia32.efi': [3] Unsupported
error are those who run a 32-bit UEFI.
@farid220 (and potentially @devedse if you so see an error with Rufus 3.10), can you confirm that your UEFI is 32-bit?
If UEFI:NTFS fails, it's a actually very easy to find that as the line you see on the top, *** UEFI:NTFS (####) ***
tells you precisely which architecture your UEFI firmware uses between the parentheses.
If I can get confirmation that only ia32 (and aarch64) fail, that would provide some confirmation that I have probably identified the issue.
Oh and I have to point out that, if this is the issue, then some UEFI systems seem to be a lot more tolerant than others, as a Raspberry Pi 4 running a UEFI firmware appears to have no trouble loading the aarch64 driver from UEFI:NTFS, even as it is missing the same data as the ia32 one...
Actually, I am able to replicate the issue with QEMU and IA32 OVMF (should have tested that first instead of assuming that I couldn't reproduce this, as was the case with the previous issues):
On the other hand, the ARM64 QEMU UEFI firmware seems to be absolutely fine about a driver that's missing half the relevant fields, including entry-point
:
The other kicker is that, when recompiling the EDK2 drivers using the current version of VS2019, the IA32 PE header fields are properly populated (but not the AARCH64 ones), and I'm also seeing the AARCH64 PE header fields being dropped for the gnu-efi version of the driver...
All in all, it looks like a straight Visual Studio linker bug, that has been existing for a while (all the AARCH64 drivers I compiled with Visual Studio appear to have had this issue regardless of EDK2 or gnu-efi was being used), and that Microsoft appears to have recently introduced and then fixed for the IA32 version. I have validated that we are definitely setting all of the flags that the linker requires to populate the PE header (/MACHINE
, /DLL
, /ENTRY:
, /SUBSYSTEM
), so if those values aren't filled, it's 100% a linker bug.
Whereas the AARCH64 linker issue doesn't seem that problematic if most UEFI firmwares appear to be able to work around it, I'm really annoyed that the IA32 linker bug was introduced right at the time where I released a new version of the EfiFs drivers, because this then got carried into Rufus. Thanks for breaking my stuff, Microsoft!
@pbatard ,
I've just setup a clean USB Stick using Rufus 3.10 (from the releases). I then put it in my laptop and made it use \\...\\efi\rufus\ntfs_x64.efi
but this resulted in just a black screen.
I then downloaded version 1.0 from the website you gave but this resulted in the same issue.
I also tried using the 32 bit version and the exFat version but all of them resulted in just a black screen.
It's been ages since I did anything with this so I'm probably doing something wrong. Any ideas :)?
Edit: Shit I'm reading through the comments and it seems I can't boot from the .efi file, I should use the boot file. Be right back...
Edit2:
Ok I tried with the latest version from Rufus 3.10 again without any changes. This resulted in the following error:
Changing the version of the ntfs_x64.efi file to v1.0 (downloaded from the website you provided) works correctly
@devedse, thanks for wanting to test this.
The way you should create the USB is make sure that you have the Advanced drive properties showing, because then you will see a UEFI:NTFS option in the Boot selection dropdown, which is what you should use.
Then you just need to place a bootx64.efi
, bootia32.efi
in /efi/boot/
on the NTFS partition, and you're good to go. I'm attaching an archive that contains the files you need to extract on the NTFS partition below (make sure you do extract with the directories: It should contain a /efi/boot/
folder):
Repeat that with different versions of Rufus (especially 3.8 or 3.9 vs 3.10) and it will tell me it there exists other problems besides the IA32 load driver
error.
Okay, I'm only seeing your update now.
Even if it results in the same error code, the Could not open partition: [3] Unsupported
is a different issue, since this time the driver was loaded properly.
What I'd still like to confirm is what you get with the method I highlighted above, using versions 3.10, 3.9 and 3.8 of Rufus (which you can find here if needed).
And now I'm starting to see clearer as to what happens for that [FAIL] Unable to load driver '\efi\rufus\ntfs_ia32.efi': [3] Unsupported
error, which boils down to stupid typos and unlucky false flags.
[3] Unsupported
when you load the IA32 drivers is because you are effectively trying to load an AARCH64 driver. Ironically, I had crafted the problematic script to ensure that I wouldn't screw up my uploads, which I renamed and copied over by hand before that...cpu
field was not populated... When looking at the AARCH64 drivers with other analysis tools, everything appears in order, including the subsystem
(which also explains why ARM64 QEMU is fine with).So at least, the Unable to load driver
can be explained fully.
Which leaves us with the Could not open partition: [3] Unsupported
issue, that seems to boil down to using specific hardware as well as whether gnu-efi or EDK2 was used to build the drivers...
Checklist
<FULL LOG>
below.Rufus version: x.y.z
- I have NOT removed any part of it.Additionally (if applicable):
(✓)
button to compute the MD5, SHA1 and SHA256 checksums, which are therefore present in the log I copied. I confirmed, by performing an internet search, that these values match the ones from the official image.Issue description
Motherboard: S2600CP4 Bios Revision: R02.06.E006 Rufus version: 3.3
Here's the log from Rufus: https://gist.github.com/zanechua/36c2416c9d8035623a0efb51a176d7af
Here are the errors I am getting on boot:
EDIT:
So it seems like the last version of Rufus v2 works.
Rufus version: 2.18
Here's the log from Rufus: https://gist.github.com/zanechua/acf8c67c7026814796f7da6414dd6e3e
I don't think it's the bios update though since 2.18 works. I haven't formatted this computer in awhile so I might have been using 2.18 and this is the first time I am using 3.x on this computer.