piscsi: Expose protobuf interface via SCSI

[uweseimet]

The host services device shall support two vendor-specific SCSI commands for executing protobuf interface operations. One command shall process serialized protobuf data (see #1348) with the operaton code in JSON, protobuf text or binary format and send it to piscsi. The other shall return the operation result as serialized protobuf data in JSON, protobuf text or binary format. Input and output format shall not have to be identical.

With this feature clients can execute the same operations as supported by the socket-based protobuf interface by sending SCSI commands.

A use case for this feature is a user who on the connected computer creates a new disk image file, then inserts it into a removable media drive and uses it, changes the log level etc With this feature you can even use piscsi when there is no network connection, because you can control it with SCSi commands ;-).

With this implementation #1346 becomes obsolete.

See host_services.cpp for details on the implemented vendor-specific commands.

• [x] Implement sending and receiving of JSON data
• [x] Implement sending and receiving of binary data
• [x] Implement sending and receiving of text format data
• [x] Share command execution code between the piscsi core and the host services
• [x] Improve the SCSI controller error handling
• [x] Make SCSI controller more robust against buffer overflows. A side effect of this is that the transfer buffer is now dynamically allocated. This very likely saves almost 16 MB (!!!) of memory when using the DaynaPort emulation. The regular buffer size without DaynaPort is between 4096 and 65536 bytes.
• [x] piscsi shall permit only a single SCHD device. Same for SCBR and SCDP
• [x] Testing with a connected computer

[uweseimet]

This approach works in practice.

JSON command sent via SCSI:

{
 "operation": "NETWORK_INTERFACES_INFO",
 "devices": [
  {
   "id": -1
  }
 ],
 "params": {
  "locale": "en_US.UTF-8"
 }
}

JSON result received via SCSI:

{"status":true,"networkInterfacesInfo":{"name":["eth0","wlan0"]}}

The statistics are another example:

{"status":true,"statisticsInfo":{"statistics":[{"category":"CATEGORY_INFO","id":2,"key":"byte_read_count"},{"category":"CATEGORY_INFO","id":2,"key":"byte_write_count"}]}}

By abusing the new scsiexec tool (#1362) I could also test how the SCSI controller deals with unexpected bus phases or malformed SCSI commands. This helped with improving the controller's error handling in cases where the computer connected does something strange. I even managed to send broken commands freezing the Pi or crashing piscsi. With the changes in this ticket the controller is more robust.

[uweseimet]

@rdmark The SCHS device represents the piscsi process, i.e. there should not be more than one. This ticket enforces this. Would you agree that there can also only be a single SCBR and SCDP device?

[rdmark]

Yes I think allowing only one network device to be attached at a time is appropriate. In fact, I think this is already the de-facto situation, because the code for initializing the piscsi_bridge will throw an error when you attempt to double initialize a bridge.

[uweseimet]

@rdmark Yes, there already is an error. By enforcing only a single instance we get a better error handling, because the problem is detected before trying to initialize the network a second time.

A new localized error message is needed for that. The "issue_1349" branch already contains the required changes (ERROR_UNIQUE_DEVICE_TYPE). If anybody wants to add the missing translations, please go ahead and just add them to this branch.

[uweseimet]

Successfully tested with an Atari TT as client computer.

[uweseimet]

Yes I think allowing only one network device to be attached at a time is appropriate. In

By the way, the DaynaPort emulation reserves 16 MB of buffer space per instance. It's the device that consumes the most memory by far. If you want to bring a small Pi to its knees, just create several of them ;-). I doubt that wasting that much memory is required. It's just network packets that are transferrred after all. The SCSi controller robustness changes of this ticket now allocate a bigger buffer only when required. I guess this will help a lot.

[rdmark]

@uweseimet Regarding the intention of this feature: I understand the technical concept of being able to send protobuf commands from the host over the SCSI bus to the PiSCSI.

If you can help me understand your vision for this feature, however: Would you then develop a native application on, say the vintage Atari or Mac computer that makes use of this new interface? Do you already have such an application for Atari that could demonstrate the end-user feature set that you have in mind?

I'm just trying to understand the justification and tradeoff for the adding of a large amount of code (and the scsiexec tool) to an already complex software package, that takes hours to compile on an RPi Zero, as an example.

So in the end, while I see the potential utility, I'm just not sure about the benefit to the broader PiSCSI user base right now!

[uweseimet]

@rdmark For the Atari a simple tool already exists on https://github.com/uweseimet/atari_public/tree/pi_exec/PISCSI. See PI_EXEC.C, just a few lines of code. With MagiCMac (Atari emulation for old Macs) it even runs on a Mac: https://www.application-systems.de/magicmac/, and with MagiCPC on Windows: https://www.application-systems.de/magicpc/. In both cases the native SCSI driver of the Mac/PC is used by the emulation. The Linux SG driver should also work. I mention all this because it shows that the new functionality is something usable by any platform.

Remember https://github.com/PiSCSI/piscsi/issues/1206: In order to implement something like this on the Mac you need this new functionality in piscsi.

scsiexec, which uses this functionality, will also be useful when testing with the new in process bus. It has already helped me to identify issues in the existing low level code. By being able to send PiSCSI any SCSI command and set any bus phase as you want, you can simulate some of the behavior of hardware (computers, samplers) you do not own. And you can send SCSI commands to piscsi within the same process, which is similar to integration testing without needing any special hardware. This is similar to what I have already done by running piscsi and piscsidump against each other in the same process as part of https://github.com/PiSCSI/piscsi/issues/1295. The new in process bus of #1295 most likely makes the current virtual bus obsolete and replaces it with a more versatile approach. This means less code elsewhere.

[uweseimet]

@rdmark One more thing about which I have already been thinking, regarding compile times. I don't know whether this would already work yet, but the code is getting more modular all the time, and sooner or later it might be possible to ask the user before compiling which devices she/he wans to use. Then you would only compile what you need.

Why do we compile scsimon and scsiloop unconditionally by the way? Is the community interested in those? From that perspective, by default easyinstall might just run "make piscsi scsictl". And even scsictl is not used by that many users I guess.

And yet one more thing: I ensured that one can compile with clang++, which is faster than g++. I added notes on the Wiki how to link faster by using an alternative linker. And for more than two years I have been identified and removed a lot of unused code, which has taken time to compile. So I should also be allowed to use the time saved for new features, shouldn't I? ;-)

Just checked: When compiling only piscsi and scsictl the compile time is about 15% less. This may vary depending on Pi and compilation with or without optimization. When I compile on my Pi Zero I always only compile the binary I need.

I created https://github.com/PiSCSI/piscsi/issues/1383 dealing with compiling less by default in order to save time.

Sorry for writing a lot of comments these days ;-). But I constantly have new ideas what you can do with the initiator mode (which is related to scsiexec and thus also to this ticket). Of all the other SCSI solutions only BlueSCSI V2 also has this mode. But BlueSCSI does not have a command line where you can really exploit this feature. I assume that BlueSCSI has quite strict memory limitations as well, which also limits the practical use of initiator mode. But this is just a strength of PiSCSI: You do not have these hard limitations. This means there are much more potential useful features. Some of them might even attract new users. https://github.com/PiSCSI/piscsi/issues/1375 may help with recovering data from broken hard drives, for instance. By increasing (build) modulary one can ensure that compile times are acceptable or become even better than they were before.

But let's face it: You should not compile PiSCSI on a Pi Zero, unless you are a developer or as a regular user who needs to build a branch in order to verify a bugfix. Ordinary users with a Pi Zero may be better off with the binary distribution. Resolving https://github.com/PiSCSI/piscsi/issues/740 would probably be the best approach.