search

cloudbase / wnbd

Windows Ceph RBD NBD driver
GNU Lesser General Public License v2.1
60 stars 26 forks source link

Big partition not work ntfs signature? #132

Closed vkornilyev1988 closed 1 year ago

vkornilyev1988 commented 1 year ago

My name is Valery, I am writing a diskless system that will be freely available since the markets now only provide such systems for rent. I want to contribute to the free world. I had a problem with the wnbd driver code, I distributed only the section. This is necessary to divide the descriptors into read and write. And the end computer client receives a partition that cannot be mounted above 1.2 TB since it only receives a partition with the NTFS signature in the form of a qemu-nbd -p 2000 -x data /dev/sda1p2 or qemu-nbd -p 2000 -x data \.\G: > windows client not work 4T. Why? What can be done to make wnbd connect a logical disk with a size greater than 1.2 TB, for example 4 or 6 TB, by identifying the ntfs signature. After all, it works up to 1.2 TB. I would like to make a diskless system based on nbd and I have a driver for booting Windows only from NBD devices by add WNBD and my driver boot wnbd, am ready to share, but I really need to solve this problem.

vkornilyev1988 commented 1 year ago

Please help. Please, let's make the nbd protocol more popular. I do it for gaming clubs. 4 or 6 TB logical drive from signatre ntfs

petrutlucian94 commented 1 year ago

Hi,

It works fine for me, here's a sample:

qemu-img create -f qcow2 test_6tb.qcow2 6T

qemu-storage-daemon \
      --blockdev driver=file,node-name=file,filename=/mnt/data/test_6tb.qcow2,cache.direct=on \
      --blockdev driver=qcow2,node-name=qcow2,file=file \
      --nbd-server addr.type=inet,addr.host=0.0.0.0,addr.port=2000 \
      --export type=nbd,id=export,node-name=qcow2,name=test_6tb,writable=on

PS C:\> wnbd-client map test_6tb 13.13.13.13 --port 2000
PS C:\> get-disk -SerialNumber test_6tb | Format-Table -AutoSize

Number Friendly Name  Serial Number HealthStatus OperationalStatus Total Size Partition Style
------ -------------  ------------- ------------ ----------------- ---------- ---------------
3      WNBD WNBD_DISK test_6tb      Healthy      Online                  6 TB RAW

PS C:\> Set-Disk -Number 3 -IsOffline $false
PS C:\> Get-Disk -Number 3 | `
>>     Initialize-Disk -PassThru | `
>>     New-Partition -AssignDriveLetter -UseMaximumSize | `
>>     Format-Volume -Force -Confirm:$false

DriveLetter FriendlyName FileSystemType DriveType HealthStatus OperationalStatus SizeRemaining Size
----------- ------------ -------------- --------- ------------ ----------------- ------------- ----
F                        NTFS           Fixed     Healthy      OK                         6 TB 6 TB

A few questions:

vkornilyev1988 commented 1 year ago

Hi. Sorry it works as you described when the entire disk with the partition table is exported. This has worked before and very well. It doesn’t work when the disk is not exported in the form of qemu-nbd -p 2000 -x data /root/test.qcow2, namely when we try to export a disk partition, here are some examples: Working: linux export: qemu-img create /root/test.qcow2 2T qemu-nbd -p 2000 -x data /root/test.qcow2 wnbd-client: wnbd-client map data x.x.x.x --port 2000 _create a new NTFS partition: get-disk -SerialNumber test6tb | Format-Table -AutoSize Next -> Not working over 1.2 T: We take the previously created image and use the ntfs partition qemu-nbd -c /dev/nbd0 /root/test.qcow2 qemu-nbd -x data -p 2000 /dev/nbd0p2 -> (not nbd but nbd0p2) wnbd-client: wnbd-client map x.x.x.x --port 2000 it's not working over 1.2 TB ntfs

vkornilyev1988 commented 1 year ago

in this example I do not use my patches but use the wnbd driver and wnbd-client as it is on your github

petrutlucian94 commented 1 year ago

I see, that's not a WNBD issue though. You'll need to expose the GPT table when having larger partitions. You can either expose the entire disk through nbd or use LVM like so:

sudo pvcreate /dev/nbd0
sudo vgcreate nbd /dev/nbd0

sudo lvcreate -L 4T -n nbd01 nbd
sudo lvcreate -L 1T -n nbd02 nbd

# create gpt partition
sudo sgdisk -n 1:1MiB: /dev/nbd/nbd01
# load partition
sudo kpartx -a /dev/nbd/nbd01
# initialize ntfs filesystem
sudo mkfs.ntfs -f /dev/mapper/nbd-nbd01p1

# expose the entire logical volume
sudo qemu-nbd -x data -p 2000 /dev/nbd/nbd01
petrutlucian94 commented 1 year ago

As a side note, you may want to consider booting from iSCSI instead of NBD. It's a more robust protocol, better at recovering sessions and can deliver better performance (especially if you intend to use multipath).

https://github.com/ipxe/ipxe/discussions/324 https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/ee619722(v=ws.10)?redirectedfrom=MSDN

vkornilyev1988 commented 1 year ago

Hi. Thanks for the answers and desire to help. But the thing is that I will be targeting Windows, and iscsi is only available for a Windows server, which affects the price for computer clubs. But the point is that I would like to contribute to your development and add another very powerful area of wnbd. The fact is that wnbd showed very good results in terms of performance and speed, as you might not think. Apart from this there is no problem. If you programmatically throw a link to example G: on a physical disk, it works. But I would like to understand why it works up to 1.2T, I use your vanilla client, I don’t use phisysaldrive -> G:?

To shorten the thought, why does it work automatically up to 1.2 TB for you by default?

petrutlucian94 commented 1 year ago

To shorten the thought, why does it work automatically up to 1.2 TB for you by default?

The problem is that you're exposing the partition without the actual partition table (e.g. /dev/nbd0p2 instead of /dev/nbd0 or a logical volume such as /dev/nbd/nbd01), which can lead to very inconsistent results. Windows tries to figure out the partition layout, but without a partition table it won't always manage to do so.

Why does it handle 1.2TB disks but not larger? Hard to say, it might have to do with the partition layout or the way in which Windows handles the situation in which there's no partition table. It's not a WNBD issue and since the solution is to simply expose a partition table, it's not really worth digging further.

But the thing is that I will be targeting Windows, and iscsi is only available for a Windows server

Well, if you intend to boot from remote disks, you don't really need Windows iSCSI support. iPXE would be the one handling the iSCSI connections. On the iSCSI portal (server) side you can just use Linux + tgt. Someone did exactly that with Windows 11: https://github.com/ipxe/ipxe/discussions/324#discussioncomment-1467072. I don't mean to push you away from NBD, I'm just providing a convenient alternative that might suite your use case.

vkornilyev1988 commented 1 year ago

Thanks for the answer. Yes, I understand about tgt and Linux, but the fact of the matter is that I also need to configure the target from Windows Pro, and not server Windows. And that's either write your own iscsi target like ccboot or use what I've already done with qemu-nbd or nbdkit. In this case, Do I understand correctly that your driver transfers control of the storport and it is he who is trying to solve my question above? I just thought you knew how to tell the storport driver that this is GPT and not MBR, and apparently this is what your driver should do if the signature ntfs

petrutlucian94 commented 1 year ago

That's a different layer, Storport drivers do not care about partitions, only about block level storage.

The WNBD driver receives SCSI request blocks and forwards them to userspace daemons, such as the wnbd client acting as a NBD client or rbd-wnbd (Ceph RBD client).

vkornilyev1988 commented 1 year ago

Also, understand, I also added a table of the ibft type, which is called wbft (wnbd table) in ipxe, and everything works except that I can’t move the descriptor to a record because the target \.\G: doesn’t work due to the problem above ntfs one partition and no table , but working to 1.2 TB)

petrutlucian94 commented 1 year ago

So if for whatever reason you can't use expose the entire disk, you could do the following:

vkornilyev1988 commented 1 year ago

Yes, I use a raw image to upload and it works. But I need to configure the game disc for super mode. And I can't use LVM on Windows. I tried using the utility to throw a direct link to dosdev G: \.\physicaldrive2 and it only works when you do the format /Q G: , but the post reconnects and again says that it is not connected to the system. I do not know what to do. I don’t understand why it’s impossible to send a tag to the driver that this is GPT, as was done previously in NBDKit. In principle, you gave me some clarification on the operation of the driver, thank you very much for that. Just tell me, is it possible to somehow tell the system that this is a GPT and not an MBR?

petrutlucian94 commented 1 year ago

I see that nbdkit has some partition plugin [1], allowing you to expose a single partition from an image file: nbdkit --filter=partition --filter=xz file disk.img.xz partition=1. As a hack/convenience, it might prepend a partition table for you, which is not something that qemu-nbd does. It's also outside the scope of the wnbd driver, which exposes the disk as-is, without caring what's inside.

Just curious, is nbdkit no longer available?

[1] https://www.libguestfs.org/nbdkit.1.html

vkornilyev1988 commented 1 year ago

Available on Linux, but under Windows it no longer compiles normally and is lost in the processes when launched :)))! I’m just again interested in what activates 4TB when I do the following dosdev G: \.\physicaldrive2 sequence. After this, Windows opens this section and I can work with it. After that, if I reconnect it again without formatting, I get the error “The device is not connected to the system” .

petrutlucian94 commented 1 year ago

I see, how about nbdkit + WSL or even LVM + WSL2?

vkornilyev1988 commented 1 year ago

Yes, of course, but for this I will need to redirect that same section to WSL, which is also not convenient. Damn, isn't this an insoluble problem? It's a shame that so much has been done and works so well. This is the last step to make something cool. Honestly your driver works very well, great that you did this. What I’m writing, I’ll give an example https://www.ccboot.com/ and I really want it to be on open protocols, something like nbd since microsoft has imposed restrictions on iscsi by windows 10/11 pro.

vkornilyev1988 commented 1 year ago

It is curious that through vnbd it defines it as mbr and through isks rfr gpt both target exactly \\.\G: disk ================================ISCSI==================================== UniqueId : 4343426F6F74202031643564623935313532333639626438 Number : 1 Path : \?\scsi#disk&ven_ccboot&prod_ccboot_iscsi#1&1c121344&0&000001#{53f56307-b6bf-11d0-94f2-00a0c91efb 8b} Manufacturer : CCBoot Model : CCBoot iSCSI SerialNumber : af63068c2921e6da1a342ffa71893103 Size : 3.68 TB AllocatedSize : 4048519430656 LogicalSectorSize : 512 PhysicalSectorSize : 512 NumberOfPartitions : 1 PartitionStyle : GPT IsReadOnly : False IsSystem : False IsBoot : False ================================WNBD=================================== UniqueId : SCSI\DISK&VEN_WNBD&PROD_WNBD_DISK\1&2AFD7D61&0&000000:PC102 Number : 2 Path : \?\scsi#disk&ven_wnbd&prod_wnbd_disk#1&2afd7d61&0&000000#{53f56307-b6bf-11d0-94f2-00a0c91efb8b} Manufacturer : WNBD Model : WNBD_DISK SerialNumber : data Size : 4 TB AllocatedSize : 3207370637312 LogicalSectorSize : 512 PhysicalSectorSize : 512 NumberOfPartitions : 2 PartitionStyle : MBR IsReadOnly : False IsSystem : False IsBoot : False

petrutlucian94 commented 1 year ago

Can you paste the wnbd-client show output as well? Could it be that the iSCSI lun has the entire disk and not just the partition?

vkornilyev1988 commented 1 year ago

I hope I understood you correctly and this is necessary, if anything else is needed, I will draw any conclusion if necessary, and I can even provide direct access to my computers for you if this is necessary)))))) =======================================================================> C:\Users\support>wnbd-client map data 192.168.0.133 --port 2000 14:08:57.479 libwnbd.dll!NbdDaemon::ConnectNbdServer INFO Initializing NBD connection. 14:08:57.485 libwnbd.dll!NbdDaemon::ConnectNbdServer INFO Successfully connected to NBD server: 192.168.0.133. 14:08:57.529 libwnbd.dll!NbdNegotiate WARNING Ignoring unsupported NBD reply info type: 3 14:08:57.529 libwnbd.dll!NbdNegotiate INFO NBD negotiation successful. 14:08:57.529 libwnbd.dll!NbdDaemon::TryStart INFO Retrieved NBD flags: 3181. Read-only: 0, TRIM enabled: 1, FLUSH enabled: 1, FUA enabled: 1. 14:08:57.530 libwnbd.dll!WnbdCreate INFO Mapped device. Connection id: 2. Bus: 0, target: 0, lun: 0. 14:08:57.530 libwnbd.dll!NbdDaemon::TryStart INFO NBD mapping initialized successfully. 14:08:57.530 libwnbd.dll!WnbdRunNbdDaemon INFO NBD mapping created, running as a daemon. Press CTRL-C to stop or use 'wnbd-client unmap'. 14:08:57.531 libwnbd.dll!NbdDaemon::Wait INFO Waiting for the WNBD dispatchers. Received control signal, cleaning up. 14:32:32.477 libwnbd.dll!PnpRemoveDevice INFO Soft removing disk device. 14:32:32.481 libwnbd.dll!WnbdHandleRequest INFO Received disconnect request. 14:32:32.481 libwnbd.dll!WnbdRemove INFO Unmapping device data. 14:32:32.481 libwnbd.dll!NbdDaemon::Wait INFO WNBD dispatchers stopped. 14:32:32.482 libwnbd.dll!WnbdRemove INFO Unmapping device data. 14:32:32.482 libwnbd.dll!WnbdIoctlShow INFO Could not find the specified disk. 14:32:32.483 libwnbd.dll!NbdDaemon::DisconnectNbd INFO Removing NBD connection. 14:32:32.484 libwnbd.dll!NbdDaemon::DisconnectNbd INFO NBD connection closed. 14:32:32.484 libwnbd.dll!NbdDaemon::Wait INFO Waiting for the WNBD dispatchers. 14:32:32.484 libwnbd.dll!RecvExact INFO Request canceled. 14:32:32.486 libwnbd.dll!NbdDaemon::NbdReplyWorker INFO Connection closed. 14:32:32.485 libwnbd.dll!WnbdWaitDispatcher INFO Dispatcher stopped, waiting for any remaining dispatcher threads. 14:32:32.488 libwnbd.dll!NbdDaemon::Wait INFO WNBD dispatchers stopped. 14:32:32.489 libwnbd.dll!NbdDaemon::~NbdDaemon INFO Waiting for NBD reply dispatcher thread. 14:32:32.489 libwnbd.dll!NbdDaemon::~NbdDaemon INFO NBD reply dispatcher stopped.

C:\Users\support>wnbd-client map data 192.168.0.133 --port 2000 14:34:49.173 libwnbd.dll!NbdDaemon::ConnectNbdServer INFO Initializing NBD connection. 14:34:49.175 libwnbd.dll!NbdDaemon::ConnectNbdServer INFO Successfully connected to NBD server: 192.168.0.133. 14:34:49.221 libwnbd.dll!NbdNegotiate WARNING Ignoring unsupported NBD reply info type: 3 14:34:49.221 libwnbd.dll!NbdNegotiate INFO NBD negotiation successful. 14:34:49.221 libwnbd.dll!NbdDaemon::TryStart INFO Retrieved NBD flags: 3181. Read-only: 0, TRIM enabled: 1, FLUSH enabled: 1, FUA enabled: 1. 14:34:49.223 libwnbd.dll!WnbdCreate INFO Mapped device. Connection id: 3. Bus: 0, target: 0, lun: 0. 14:34:49.224 libwnbd.dll!NbdDaemon::TryStart INFO NBD mapping initialized successfully. 14:34:49.224 libwnbd.dll!WnbdRunNbdDaemon INFO NBD mapping created, running as a daemon. Press CTRL-C to stop or use 'wnbd-client unmap'. 14:34:49.224 libwnbd.dll!NbdDaemon::Wait INFO Waiting for the WNBD dispatchers. Received control signal, cleaning up. 14:36:07.142 libwnbd.dll!PnpRemoveDevice INFO Soft removing disk device. 14:36:07.211 libwnbd.dll!WnbdHandleRequest INFO Received disconnect request. 14:36:07.211 libwnbd.dll!WnbdRemove INFO Unmapping device data. 14:36:07.212 libwnbd.dll!NbdDaemon::Wait INFO WNBD dispatchers stopped. 14:36:07.212 libwnbd.dll!WnbdRemove INFO Unmapping device data. 14:36:07.213 libwnbd.dll!WnbdIoctlShow INFO Could not find the specified disk. 14:36:07.213 libwnbd.dll!NbdDaemon::DisconnectNbd INFO Removing NBD connection. 14:36:07.214 libwnbd.dll!NbdDaemon::DisconnectNbd INFO NBD connection closed. 14:36:07.214 libwnbd.dll!RecvExact INFO Request canceled. 14:36:07.218 libwnbd.dll!NbdDaemon::NbdReplyWorker INFO Connection closed. 14:36:07.214 libwnbd.dll!NbdDaemon::Wait INFO Waiting for the WNBD dispatchers. 14:36:07.218 libwnbd.dll!WnbdWaitDispatcher INFO Dispatcher stopped, waiting for any remaining dispatcher threads. 14:36:07.218 libwnbd.dll!NbdDaemon::Wait INFO WNBD dispatchers stopped. 14:36:07.218 libwnbd.dll!NbdDaemon::~NbdDaemon INFO Waiting for NBD reply dispatcher thread. 14:36:07.220 libwnbd.dll!NbdDaemon::~NbdDaemon INFO NBD reply dispatcher stopped.

C:\Users\support>wnbd-client map data 192.168.0.133 --port 2000 14:36:14.196 libwnbd.dll!NbdDaemon::ConnectNbdServer INFO Initializing NBD connection. 14:36:14.197 libwnbd.dll!NbdDaemon::ConnectNbdServer INFO Successfully connected to NBD server: 192.168.0.133. 14:36:14.248 libwnbd.dll!NbdNegotiate WARNING Ignoring unsupported NBD reply info type: 3 14:36:14.249 libwnbd.dll!NbdNegotiate INFO NBD negotiation successful. 14:36:14.249 libwnbd.dll!NbdDaemon::TryStart INFO Retrieved NBD flags: 3181. Read-only: 0, TRIM enabled: 1, FLUSH enabled: 1, FUA enabled: 1. 14:36:14.250 libwnbd.dll!WnbdCreate INFO Mapped device. Connection id: 4. Bus: 0, target: 0, lun: 0. 14:36:14.250 libwnbd.dll!NbdDaemon::TryStart INFO NBD mapping initialized successfully. 14:36:14.251 libwnbd.dll!WnbdRunNbdDaemon INFO NBD mapping created, running as a daemon. Press CTRL-C to stop or use 'wnbd-client unmap'. 14:36:14.251 libwnbd.dll!NbdDaemon::Wait INFO Waiting for the WNBD dispatchers. Received control signal, cleaning up. 14:59:53.967 libwnbd.dll!PnpRemoveDevice INFO Soft removing disk device. 14:59:54.185 libwnbd.dll!WnbdHandleRequest INFO Received disconnect request. 14:59:54.185 libwnbd.dll!WnbdRemove INFO Unmapping device data. 14:59:54.185 libwnbd.dll!NbdDaemon::Wait INFO WNBD dispatchers stopped. 14:59:54.185 libwnbd.dll!WnbdRemove INFO Unmapping device data. 14:59:54.185 libwnbd.dll!WnbdIoctlShow INFO Could not find the specified disk. 14:59:54.185 libwnbd.dll!NbdDaemon::DisconnectNbd INFO Removing NBD connection. 14:59:54.185 libwnbd.dll!NbdDaemon::DisconnectNbd INFO NBD connection closed. 14:59:54.185 libwnbd.dll!NbdDaemon::Wait INFO Waiting for the WNBD dispatchers. 14:59:54.185 libwnbd.dll!RecvExact INFO Request canceled. 14:59:54.185 libwnbd.dll!WnbdWaitDispatcher INFO Dispatcher stopped, waiting for any remaining dispatcher threads. 14:59:54.195 libwnbd.dll!NbdDaemon::NbdReplyWorker INFO Connection closed. 14:59:54.195 libwnbd.dll!NbdDaemon::Wait INFO WNBD dispatchers stopped. 14:59:54.195 libwnbd.dll!NbdDaemon::~NbdDaemon INFO Waiting for NBD reply dispatcher thread. 14:59:54.195 libwnbd.dll!NbdDaemon::~NbdDaemon INFO NBD reply dispatcher stopped.

C:\Users\support>wnbd-client map data 192.168.0.133 --port 2000 15:00:09.969 libwnbd.dll!NbdDaemon::ConnectNbdServer INFO Initializing NBD connection. 15:00:09.969 libwnbd.dll!NbdDaemon::ConnectNbdServer INFO Successfully connected to NBD server: 192.168.0.133. 15:00:10.019 libwnbd.dll!NbdNegotiate WARNING Ignoring unsupported NBD reply info type: 3 15:00:10.019 libwnbd.dll!NbdNegotiate INFO NBD negotiation successful. 15:00:10.019 libwnbd.dll!NbdDaemon::TryStart INFO Retrieved NBD flags: 3181. Read-only: 0, TRIM enabled: 1, FLUSH enabled: 1, FUA enabled: 1. 15:00:10.019 libwnbd.dll!WnbdCreate INFO Mapped device. Connection id: 5. Bus: 0, target: 0, lun: 0. 15:00:10.019 libwnbd.dll!NbdDaemon::TryStart INFO NBD mapping initialized successfully. 15:00:10.019 libwnbd.dll!WnbdRunNbdDaemon INFO NBD mapping created, running as a daemon. Press CTRL-C to stop or use 'wnbd-client unmap'. 15:00:10.019 libwnbd.dll!NbdDaemon::Wait INFO Waiting for the WNBD dispatchers.

vkornilyev1988 commented 1 year ago

PS C:\Windows\system32> wnbd-client show data Connection info InstanceName : data SerialNumber : data Owner : wnbd-client ReadOnly : 0 FlushSupported : 1 FUASupported : 1 UnmapSupported : 1 UnmapAnchorSupported : 0 UseUserspaceNbd : 1 BlockCount : 8589899776 BlockSize : 512 MaxUnmapDescCount : 1 Pid : 3004 DiskNumber : 2 PNPDeviceID : SCSI\Disk&Ven_WNBD&Prod_WNBD_DISK\1&2afd7d61&0&000000 BusNumber : 0 TargetId : 0 Lun : 0 MaxIOReqPerLun : 255 MaxIOReqPerAdapter : 1000

Nbd properties Hostname : 192.168.0.133 PortNumber : 2000 ExportName : data SkipNegotiation : 0

vkornilyev1988 commented 1 year ago

PS C:\Windows\system32> wnbd-client list Pid DiskNumber Nbd Owner InstanceName 3004 2 true wnbd-client data

vkornilyev1988 commented 1 year ago

image He really sees that there are two partitions from GPT, but this is false information because he confuses the MBR labels with data about NTFS, as I think

vkornilyev1988 commented 1 year ago

trying to read it as a table beginning of the disk it gets similar labels but not that it describes the tables and they are similar to the MBR table, which is why it displays it as an MBRtype again that's what I think but it's not necessarily true

vkornilyev1988 commented 1 year ago

image image I think you are right here, this in my opinion means that iscsi produces a table because there is a correct partition there

vkornilyev1988 commented 1 year ago

image This iscsi target partition

vkornilyev1988 commented 1 year ago

in general, I reattached the disk without formatting, funny this way: wnbd-client map 192.168.0.133 --port 2000 next dosdev G: \.\physicaldrive2 then I get an error about the device not being connected I do chkdsk G: and the disk is connected. It's as if some kind of disk activation is simply missing

vkornilyev1988 commented 1 year ago

Surprisingly, it works, I just checked a disk with a capacity of 10TB and it also writes MBR. It turns out that Windows can work with a direct volume without partitions. Of course it's awesome. MBR :))))

vkornilyev1988 commented 1 year ago

image

vkornilyev1988 commented 1 year ago

Thanks thanks petrutlucian94