defect: mknod fails to add additional nbd devices to system

[tarndt]

Summary: mknod command (or system call sys_mknodat) fails to add additional nbd devices to system.

Steps to reproduce:

1. Load nbd: sudo modprobe nbd
2. If loaded default way (nbds_max defaults to 16) you can test and observe the ndb devices created at module load work functionally and are present at both /dev/nbd* as well as /sys/block/nbd*
3. Use mknod to create a new device (ex. sudo mknod /dev/nbd500 b 43 500)
4. Observe the device file has been created at /dev/nbd* and ls/stat to see it has the correct major (43) and minor (ex. 500)
5. Observe the device instance has not been created at /sys/block/nbd*. The other devices (if nbds_max > 0) are present in both both `/dev/nbdas well as/sys/block/nbd, but the new device is **only** in/dev/nbd`.*
6. If you go on to attempt to open the new device /dev/nbd* (ex. /dev/nbd500 above) you will get an error: "no such device or address". My understanding is this means the device file and the driver are not truly connected, a hypothesis which is supported by the lack of anything in /sys/block/ for the new device.

System: Linux XXX 5.4.0-92-generic #103-Ubuntu SMP XXX x86_64 x86_64 x86_64 GNU/Linux

I am happy to contribute a fix or enhancement, but I need help understanding if this is a defect, or a limitation that the nbd module only supports devices created at module load time. I have found many scripts (random example) where users have used mknod to create nbd devices. Even the readme of this repo suggests dynamic (after module load time) device instantiation should work, as it uses MAKEDEV which in turn uses mknod. One tangential issue to note (esp. if trying to use MAKEDEV rather than mknod directly to reproduce; MAKEDEV on Ubuntu 20.04 claims it does not know how to make nbd devices:

$ sudo MAKEDEV /dev/nbd600
/sbin/MAKEDEV: don't know how to make device "/dev/nbd600"

Would one of you ( @yoe @ebblake @abligh @axboe or anyone else) be willing to work with me further to debug this issue? Thanks!

Appendix:

1. Historical nbd mailing list thread which may be the same or similar issue in action. But looks to have never been concluded.

[yoe]

Yes, this is working as designed.

NBD (and the userland tools) is older than udev, older than DevFS even. So in the old days (or if you choose not to use udev, not sure if that's even possible anymore these days) you had to run MAKEDEV/mknod manually to create the device nodes so you could talk to the kernel at all. The paragraph before the bit that talks about MAKEDEV in the readme does say "if you're not using udev".

Nowadays that's not longer necessary. I guess it's time to drop the references to MAKEDEV from the README file.

[tarndt]

Thanks. Follow up question then, if I want to create new nbd devices on-demand from my application it sounds like I can't just ignore udev and call the mknod syscall.... How does one create a new nbd device using udev?

[yoe]

You can't, the kernel doesn't support that currently. You need to create all the device nodes you need at module load time.

[tarndt]

Thank you for that important clarification.

My understanding is that the Linux kernel itself can dynamically create devices since this happens with other devices in the system.

Am I correct and understanding that the enhancement to achieve dynamic NDB device provisioning would need to be made to the NBD module in the kernel source tree?

The correct place for submitting such a patch would be the kernel mailing list correct? If so I will follow up there with a patch.

Thanks!

[yoe]

Am I correct and understanding that the enhancement to achieve dynamic NDB device provisioning would need to be made to the NBD module in the kernel source tree?

Yes.

It might require more than just that -- most user space code expects the current behavior, so you may not be able change that without breaking things.

The correct place for submitting such a patch would be the kernel mailing list correct? If so I will follow up there with a patch.

Amongst others; check the MAINTAINERS file. There's a script in the kernel which automates sending patches to the correct location(s).

[tarndt]

It might require more than just that -- most user space code expects the current behavior, so you may not be able change that without breaking things.

That's a good point and something I was considering as well. I'd like to propose the exiting device pre-allocation (controlled by the module load parameter nbds_max and defaulting to 16) would continue to function as it does today to ensure continuity. But that the driver would be augmented by my patch with the additional logic to allow udev (or mknod callers for that matter) to dynamically request further nbdX...n devices to be created on demand. Many of the physical device drivers (that support hot-plug, for example) have the required kernel module code to enable this kind of behavior today. Does this seem reasonable to you as a starting point?

[yoe]

I don't think that should be done by udev. If anything, configuring a new device through the netlink interface when all devices are already in use should trigger that.

That said, I'm the maintainer of the NBD userspace code, but I'm not involved with the kernel, do I suggest you talk to the kernel maintainer of the NBD driver.