[issue]: ventoy menu/browser not listing some images

hgkamath commented 1 year ago

Official FAQ

[X] I have checked the official FAQ.

Ventoy Version

1.0.88

What about latest release

Yes. I have tried the latest release, but the bug still exist.

Try alternative boot mode

No. I didn't try these alternative boot modes.

BIOS Mode

UEFI Mode

Partition Style

GPT

Disk Capacity

2Tb

Disk Manufacturer

Seagate

Image file checksum (if applicable)

None

Image file download link (if applicable)

No response

What happened?

A bug happened!

A raw.img.vtoy file (m02_lnx.raw.img.vtoy) is not listed in ventoy boot-menu/browser. ventoy may be skipping other non-iso image files in the directory. I can't see a pattern/reason.

For example: vtoy menu lists seaboot_20191118.raw.img.vtoy but not m02_lnx.raw.img.vtoy . seaboot_20191118.raw.img.vtoy is also bootable

This used to work before and I have in the past booted into this image via ventoy using vtoyboot and all.

misc info

1st partition is ntfs, 1.2 Tb, the filesystem has been fs-checked chkdsk /F /V /X E:
image is 40GiB raw image with fedora-37 linux installed to it, raw img with gpt, 4 partitions (efisys, biosboot, boot-ext4, root-ext4)
It is possible to boot into it from qemu from both windows/Linux

log

PS C:\vol\scoop_01\scoopg\apps\ventoy\current> dir E:\transcend\*vtoy

    Directory: E:\transcend

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----         1/25/2023   1:53 PM    49861885952 m02_lnx.raw.img.vtoy
-a----         12/3/2022   7:50 PM      352321536 seaboot_20191118.raw.img.vtoy

ventoy commented 1 year ago

Can it list if you copy it to the Ventoy USB?

hgkamath commented 1 year ago

I first used qemu in windows to confirm that image is bootable, did some dnf package updates within, updated linux kernel inside to 6.1.11-200, re-applied vtoyboot-1.0.25 and then confirmed that the updates are persistent by cold rebooting.

The image is 40 Gib, My USB stick is only 8 Gib, but I copied the file from E:\ (which is 1st partition on internal drive HDD2) to two partitions of the first internal drive (SSD) (1) hd0, 16, ExFAT (2) hd0, 17, NTFS
Ventoy F2 browser was able to find and list them in both the partitions
Ventoy was also able to boot the images in both these new locations as one would expect.
but Ventoy is still not seeing the file in the original location HDD2: hd2, 1, NTFS
So if Ventoy can list it, then Ventoy will boot it.
However, I still want to place the image in E:\transcend and have it boot from there. The intention as of now, is that m02_lnx.raw.img.vtoy in the HDD is supposed to be a bootable-OS near identical to m01_lnx.raw.img.vtoy in the SSD

Below is what the directory contains after some filtering

PS C:\WINDOWS\system32> cmd /c "dir E:\transcend\" | findstr /v /i checksum | findstr /v /i sha256 | findstr /v /i '\.txt' | findstr /v /i '\.sh' | findstr /v /i changelog
 Volume in drive E is 15KJ_Images
 Volume Serial Number is 1EA6-2771

 Directory of E:\transcend

02/10/2023  01:09 PM    <DIR>          .
02/10/2023  01:09 PM    <DIR>          ..
12/15/2022  09:02 PM       159,383,552 alpine-standard-3.17.0-x86_64.iso
01/04/2022  09:02 PM       906,309,632 archlinux-2022.01.01-x86_64.iso
01/15/2022  01:38 AM     1,481,637,888 Bliss-v15.0-Alpha-android_x86_64-OFFICIAL-foss-202112151838-nb-sd-ksu-gearlock-ex_vt-console_all-mesa-drivers-crocus.iso
01/28/2023  08:57 AM     2,619,355,136 cachyos-gnome-linux-230121.iso
12/15/2022  09:01 PM       380,633,088 clonezilla-live-3.0.2-21-amd64.iso
01/07/2023  05:42 PM    <DIR>          EFI
12/12/2022  05:18 PM     2,037,372,928 Fedora-Workstation-Live-x86_64-37-1.7.iso
11/03/2022  06:04 PM       507,510,784 gparted-live-1.4.0-6-amd64.iso
09/17/2021  07:09 PM    <DIR>          groot_07
09/17/2021  07:29 PM     3,099,203,584 HBCD_PE_x64_v1.0.2_20210701.iso
02/09/2023  09:15 PM     6,446,645,248 HBCD_PE_x64_v1.0.2_20210701.vhdx
01/04/2022  09:19 PM         1,339,392 hdt-0.5.2.iso
11/06/2019  07:33 PM       134,596,608 HP_sp100048_UEFI_Diagnostics-2017.iso
08/11/2022  07:01 PM            32,768 m01_lnx.raw.vlnk.img
02/10/2023  10:07 AM    49,861,885,952 m02_lnx.raw.img.vtoy
02/03/2023  04:38 AM         6,193,152 mt86plus_6.10_64.iso
03/30/2022  09:47 PM    <DIR>          nirsoft
08/11/2022  05:58 PM               512 qq1
08/11/2022  06:01 PM               512 qq2
08/11/2022  06:13 PM               512 qq3
08/11/2022  06:01 PM             2,407 qw1
08/11/2022  06:01 PM             2,407 qw2
08/11/2022  06:13 PM             2,407 qw3
08/11/2022  06:08 PM             2,407 qw3a
12/15/2022  09:12 PM       267,386,880 ReactOS-0.4.14-release-36-g90505bc-Live.iso
12/15/2022  09:20 PM     1,149,517,824 rescuezilla-2.4.1-64bit.jammy.iso
08/11/2022  05:57 PM         1,048,576 sda_2048.bin
08/11/2022  05:58 PM         1,048,576 sdb_2048.bin
08/11/2022  05:58 PM         1,048,576 sdc_2048.bin
12/03/2022  07:50 PM       352,321,536 seaboot_20191118.raw.img.vtoy
09/17/2021  07:14 PM    <DIR>          System Volume Information
08/11/2022  11:07 AM            32,768 test_lnx.raw.vlnk.img
09/17/2021  06:58 PM    <DIR>          ventoy
01/09/2022  10:43 AM       556,431,360 virtio-win-0.1.208.iso
03/30/2022  09:45 PM    <DIR>          win11
08/29/2021  10:31 PM     4,320,526,336 Win8.1_English_x64.iso
08/29/2021  03:24 PM     4,556,128,256 Windows_10_21H1.iso
11/22/2021  11:34 PM     4,600,823,808 Windows_10_21H2.iso
12/04/2021  05:39 PM     4,537,712,640 Windows_11_21H2.iso
09/20/2021  01:56 PM     1,145,044,992 WinXP_WA.vhdx
              49 File(s) 89,131,240,260 bytes
               8 Dir(s)  246,196,875,264 bytes free

I memorized mnemonic of 1st chars to try remember what of this list ventoy had listed in its F2-browser. a a b c c f g h h h m r r s t v 4w So if my memory serves me right
The F2 browser is missing the following images

HBCD_PE_x64_v1.0.2_20210701.vhdx
m01_lnx.raw.vlnk.img (vtoy-vlnk is set to point to another 40Gib file m01_lnx.raw.img.vtoy in hd0,16 )
m02_lnx.raw.img.vtoy
WinXP_WA.vhdx

The absence of the vhdx files is understandable, as ventoy cannot yet, nativeboot Linux on VHDX (webref Linux vDisk Boot Plugin) [EDIT] I read elsewhere that ventoy can nativeboot Windows-7+ on vhdx if ventoy_vhdboot.img is used (ref: ventoy_main_page, doc_vlnk_page , doc_plugin_vhdboot, bugs like ventoy-https://github.com/ventoy/Ventoy/issues/2157 ) . So maybe vhdx files will show if \ventoy\ventoy_vhdboot.img is present.

File properties

PS C:\WINDOWS\system32> get-item E:\transcend\m02_lnx.raw.img.vtoy |Format-List
    Directory: E:\transcend
Name           : m02_lnx.raw.img.vtoy
Length         : 49861885952
CreationTime   : 3/18/2022 6:02:52 PM
LastWriteTime  : 2/10/2023 10:07:36 AM
LastAccessTime : 2/10/2023 10:07:36 AM
Mode           : -a----
LinkType       :
Target         : {}
VersionInfo    : File:             E:\transcend\m02_lnx.raw.img.vtoy
                 InternalName:
                 OriginalFilename:
                 FileVersion:
                 FileDescription:
                 Product:
                 ProductVersion:
                 Debug:            False
                 Patched:          False
                 PreRelease:       False
                 PrivateBuild:     False
                 SpecialBuild:     False
                 Language:

Permissions

PS C:\WINDOWS\system32> cacls E:\transcend\m02_lnx.raw.img.vtoy
E:\transcend\m02_lnx.raw.img.vtoy Everyone:(ID)F

ACLs

PS C:\WINDOWS\system32> get-acl E:\transcend\m02_lnx.raw.img.vtoy
    Directory: E:\transcend
Path                 Owner         Access
----                 -----         ------
m02_lnx.raw.img.vtoy BUILTIN\Users Everyone Allow  FullControl

The file is contiguous

PS C:\WINDOWS\system32> contig -a E:\transcend\m02_lnx.raw.img.vtoy

Contig v1.82 - Contig
Copyright (C) 2001-2022 Mark Russinovich
Sysinternals

E:\transcend\m02_lnx.raw.img.vtoy is defragmented

Summary:
     Number of files processed:      1
     Number unsuccessfully procesed: 0
     Average fragmentation       : 1 frags/file

Partition info inside m02_lnx.raw.vtoy

[root@fedora gana]# sgdisk /dev/vda -p
Disk /dev/vda: 97386496 sectors, 46.4 GiB
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 8833D6B6-ABF3-44CD-8375-595D238C2600
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 97386462
Partitions will be aligned on 2048-sector boundaries
Total free space is 4029 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
1            2048          522239   254.0 MiB   EF00  EFI System Partition
2          522240          526335   2.0 MiB     EF02  BIOS boot partition
3          526336         2099199   768.0 MiB   8300  Linux filesystem
4         2099200        97384447   45.4 GiB    8300

[root@fedora gana]# sgdisk /dev/vda -O

Disk size is 97386496 sectors (46.4 GiB)
MBR disk identifier: 0x00000000
MBR partitions:

Number  Boot  Start Sector   End Sector   Status      Code
1                     1     97386495   primary     0xEE

observations

It seems to be locating all the .iso files without issue.
Doesn't seem file size is the issue. Only 3 of the respective images for the above 4 files are much larger than 4.5 GiB (dvd size) or 3*2^32=3221225472
A couple of months ago, I had resized my first partition form 1Tb to 1.2Tb , also moving my ventoy partition . But I don't think its because of that, because I see Cachy-Linux iso which I downloaded a week ago, which does ventoy-boot.
In the time between issue filing yesterday and this comment, I have ensured that transcend directory is owned by \\.\Users and all files within are owned by \\.\Users with Full permissions for Everyone
Not all .iso or .vhd[x] are expected to be bootable. Nevertheless, the non-bootable-ness should not affect/prevent ventoy from listing it.
- virtio-win-0.1.208.iso is not supposed to be bootable
- WinXP_WA.vhdx was a VM-only bootable experiment.
- HBCD_PE_x64_v1.0.2_20210701.vhdx is nativebootable via windows bcdedit
Notice that the vtoy-vlnk file test_lnx.raw.vlnk.img is listed but m01_lnx.raw.vlnk.img is not.

ventoy commented 1 year ago

Can it be listed if you copy a new file from m02_lnx.raw.img.vtoy and rename it to m02_lnx.raw.img_new.vtoy.

hgkamath commented 1 year ago

I think I found out what the bug might be.

Notice that the two files

08/11/2022  07:01 PM            32,768 m01_lnx.raw.vlnk.img
02/10/2023  10:07 AM    49,861,885,952 m02_lnx.raw.img.vtoy

appear one after the other in dirent inode structure linked list order.

I decided to check if the first file, the vtoy-vlnk file, m01_lnx.raw.vlnk.img, is valid. (webref vlnk files)
The GUI tool PS C:\vol\scoop_01\scoopg\apps\ventoy\current> .\VentoyVlnk.exe , parse button told me that it is invalid. Apparently, I had renamed the file.
So I deleted the vtoy-vlnk file and recreated it as follows
PS C:\vol\scoop_01\scoopg\apps\ventoy\current> .\VentoyVlnk.exe -i H:\m01_lnx.raw.img.vtoy -o E:\transcend\m01_lnx.raw.vlnk.img This fixed the vtoy-vlnk file, and it started to appear in the ventoy menu/browser.

Surprisingly, it also fixed the the non-appearance of m02_lnx.raw.img.vtoy in the ventoy menu/browser, without having to specifically do anything about it.
Both vtoy-vlnk m01_lnx.raw.vlnk.img and the raw-vtoy-image m02_lnx.raw.img.vtoy are now again ventoy-bootable.

So the bug might be that (A) During the population of menu-listing/F2-browser, an invalid vtoy-vlnk is swallowing (omitting) a successor file from the file-linked-list. (B) Somewhat less importantly, an invalid vlnk should perhaps be reported/shown-highlighted-in-red as broken? More generally, any non-bootable image-format could be shown in red, perhaps also configurable by plugson-option.

ventoy commented 1 year ago

Can you reproduce the problem if you make the vlnk file invalid again? (rename the original img file will make the vlnk file invalid)

hgkamath commented 1 year ago

Confirming: yes

First I rename the file to break vtoy-vlnk

PS H:\> Rename-Item .\m01_lnx.raw.img.vtoy m01_Zlnx.raw.img.vtoy
PS H:\> dir H:\m01_Zlnx.raw.img.vtoy
Directory: H:\
Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----         2/10/2023  12:51 PM    49861885952 m01_Zlnx.raw.img.vtoy

reboot to ventoy
I confirm that both files are missing in menu and F2-browser
reboot to win10

I rename the file to restore the vtoy-vlnk

PS H:\> Rename-Item .\m01_Zlnx.raw.img.vtoy m01_lnx.raw.img.vtoy
PS H:\> dir H:\m01_lnx.raw.img.vtoy
Directory: H:\
Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----         2/10/2023  12:51 PM    49861885952 m01_lnx.raw.img.vtoy

reboot to ventoy
I confirm that both files reappear in menu and F2-browser

ventoy commented 1 year ago

Can you reproduce the problem if the img file and vlnk file are in Ventoy USB ? You don't need to really copy the img file to Ventoy USB, you can just copy any file (bigger than 32KB) to Ventoy USB and rename it to xxxx.img and then create a vlnk file for it.

hgkamath commented 1 year ago

Confirming: yes on USB flash-drive as well

I created the same vtoy-vlnk in the USB folder L:\transcend to the image in H:\ , named it m01_lnx.raw.vlnk.img
I also copied a small 1MiB dummy file to the USB folder, file was named m02_lnx.raw.img.vtoy
Did the exact same tests as before, but this time in the UEFI-bios, choosing to boot from a detected USB efi loader (the ventoy efi loader)
In the ventoy menu/F2-browser, when vlnk is broken, both files do not list. When vlnk is repaired both files show.

misc

The USB flash drive is a Kingston 8GB, it had a windows installer/recovery iso flashed onto it.
Ventoy was installed to second partition using non-destructive install some time ago. I just updated it to v1.0.88.
Partitions are NTFS, FAT, FAT32
I copied .ventoyignore in every other directory except L:\transcend
In the ventoy menu, there was total of 4 files, gparted, the 2 created files, and bootmgr.efi from the root folder

log

PS C:\WINDOWS\system32> C:\vol\prog_01\prog\gdisk\gdisk-windows-1.0.9\gdisk64.exe \\.\PhysicalDrive10
GPT fdisk (gdisk) version 1.0.9

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): p
Disk \\.\PhysicalDrive10: 15240576 sectors, 7.3 GiB
Sector size (logical): 512 bytes
Disk identifier (GUID): 5BD95AB7-69B1-4100-A5C9-E1BC6C5C55BD
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 15240542
Partitions will be aligned on 128-sector boundaries
Total free space is 5565 sectors (2.7 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048        14651263   7.0 GiB     0700  Main Data Partition
   2        14651264        14716799   32.0 MiB    0700  VTOYEFI
   3        14716928        15237119   254.0 MiB   0700  Basic data partition

Command (? for help): q

PS L:\transcend> C:\vol\scoop_01\scoopg\apps\ventoy\current\VentoyVlnk.exe -i H:\m01_lnx.raw.img.vtoy -o L:\transcend\m01_lnx.raw.vlnk.img
PS L:\transcend> dir Y:\test_lnx.raw.img
    Directory: Y:\
Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----         8/11/2022  11:07 AM        1048576 test_lnx.raw.img
PS L:\transcend> copy-item Y:\test_lnx.raw.img L:\transcend\m02_lnx.raw.img.vtoy
PS L:\transcend> dir L:\transcend\
    Directory: L:\transcend
Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----         11/3/2022   6:04 PM      507510784 gparted-live-1.4.0-6-amd64.iso
-a----         2/10/2023   6:12 PM          32768 m01_lnx.raw.vlnk.img
-a----         8/11/2022  11:07 AM        1048576 m02_lnx.raw.img.vtoy
-a----         2/10/2023   6:12 PM            710 VentoyVlnk.log

PS C:\vol\scoop_01\scoopg\apps\ventoy\current> .\Ventoy2Disk.exe
version shown was 1.0.84

PS C:\vol\scoop_01\scoopg\apps\ventoy\current> .\Ventoy2Disk.exe VTOYCLI /U /PhyDrive:10 /GPT
upgraded to 1.0.88

ventoy / Ventoy