Issue with properties files for images in sub-directories

[nucleogenic]

Issue

Properties files within CFG_DIR are currently matched based on the image file name, i.e. {image_file}.properties.

When an image is in a sub-directory, the check no longer works, as the image contains the path.

If we strip the path, the 1:1 relationship between image file name and properties file name becomes ambiguous, for example:

Example:

~/images/machine_1/disk1.hds
~/images/machine_2/disk1.hds

~/.config/rascsi/disk1.hds
                   ^-- but, which machine?

Proposal

An alternative would be to store the properties files adjacent to the image files, for example:

~/images/machine_1/disk1.hds
~/images/machine_1/disk1.hds.properties
~/images/machine_2/disk1.hds
~/images/machine_2/disk1.hds.properties

These could then continue to be rendered in the web UI in the same way:

[rdmark]

This is a good catch. I didn't consider how properties files would be treated when the nested images dir feature was implemented on the backend.

An alternative would be to store the properties files adjacent to the image files, for example:

This is actually how it used to work. However, we decided to separate properties and image files, when the rascsi backend got logic to scan files in the images dir and perform various actions on them. For instance, it expects to find only valid image files in the images dir. Keeping the properties files in the images dir will cause the rascsi backend to continuously throw warnings.

Anyhow, we can certainly revisit this design, but it would require work on both the backend and frontend.

[uweseimet]

@rdmark Since these property files are only used by the web-ui it would not be a good idea to change anything in the backend. The backend should never have any knowledge about the frontend. As solution would be to change the web-ui, so that it it is aware of the folder structures. Not of the full path, but of the relative path. This means that if you have a nested structure of images, you would have a corresponding nested structure of property files. A completely different solution would be to just have a single property file in the .config/rascsi folder, containing the data for all images. This file can be in JSON format, for instance, and would essentially be kind of a device property database. This would also provide for having more than one set of properties for the same image file. There might be cases where this is useful. By the way, what happens whe you rename/copy/delete an image file with the web-ui? Does the web-ui also rename/copy/delete the corresponding property file? When you move an image file the web-ui also would have to move the property file. Dealing with such a scenario is less complicated when there is a single property file only.

All in all the single property file approach is similar to what other frontends (not located in the same filesystem, e.g. the RaSCSI Control app) do: They store all their meta data locally (on the computer/device they are running on), e.g. in a database. There would only be a single database with all properties, not a separate database for each image, which is similar to the single file approach.

I don't think that this is a high priority issue, though, because a user can simple ensure (maybe she/he should anyway) that there are no image files with the same name. But the web-ui should still ensure that copying/moving/deleting deals correctly with the property files, in case it does not already do so. And in case somebody renames a file with a different client than the web-ui (rasctl) there is a consistency anyway. This is a problem no client can resolve. In theory adding the complete property management to the backend is the only way to resolve this. But in practice this is not worth the effort, and it would have to be client-agnostic.

[rdmark]

@uweseimet

By the way, what happens whe you rename/copy/delete an image file with the web-ui? Does the web-ui also rename/copy/delete the corresponding property file? When you move an image file the web-ui also would have to move the property file. Dealing with such a scenario is less complicated when there is a single property file only.

Yes, the Web Interfaces does indeed rename/copy/delete the properties files when you rename/copy/delete an image file through the Web Interface.

And in case somebody renames a file with a different client than the web-ui (rasctl) there is a consistency anyway. This is a problem no client can resolve. In theory adding the complete property management to the backend is the only way to resolve this. But in practice this is not worth the effort, and it would have to be client-agnostic.

I agree with your conclusion that the only way to resolve the consistency across clients is for the rascsi backend to control this metadata. However, I would argue that it is a critical feature for a subset of RaSCSI users at this point, since RaSCSI operation on f.e. DEC, Apple Mac, or certain samplers like Akai, often depend on specific INQUIRY and block size being forced for native drivers to treat the emulated devices properly.

[uweseimet]

@rdmark I don't see anything critical here. It is sufficient to use the right -b option for the respective drive, just as advised by the warning displayed when you try to change the sector size with MODE SELECT. If the sector size is already correct (i.e. the size that formatting the drive is supposed to create) MODE SELECT will succeed. This is something I successfully tested when resolving https://github.com/akuker/RASCSI/issues/818. If there are users having this problem they are probably using a pre-22.10.01 release. If they are using 22.10.01 they should create a ticket with logs (provided the current develop branch also does not work) and I will have a look at it. This is why my other ticket is just some kind of funny feature, because effectively it does not resolve anything that cannot be resolved by other means. Edit: @rdmark I just noticed (just jumped directly to this page from the link in the github email I got) that I have not commented my own ticket https://github.com/akuker/RASCSI/issues/965 but this one ;-). Anyway, I think my comment and the resolution (-b and 22.10.01) is nevertheless adequate.

[rdmark]

@uweseimet You might misunderstand (probably due to the wrong ticket context): I'm talking in general about the ability to associate certain meta data with an image file and then have rascsi consistently use that meta data when attaching the image file: Vendor, Product, etc. Of course all of this is possible to do on a case by case basis with command line parameters, but it's cumbersome to do repeatedly and at scale.

My motivation for implementing the current *.properties scheme for the Web Interface was very much me wanting to automate that process after getting tired of juggling the complex command line parameters for attaching devices with custom Vendor/Product/block size data etc.

I think having a client agnostic capabilities for both configurations and image properties would be highly useful. It doesn't have to be done in the rascsi backend, but perhaps as a common Python client that provides an API to the other Python clients, for instance. This came up when the Control Board client was being developed, which led to https://github.com/akuker/RASCSI/issues/589

[uweseimet]

@rdmark In theory having these meta data in the image file itself is a solution that avoids issues related to distributing information across several files. The special NEC image files (hdi, nhd) do it this way and store the meta data in their header. In practice the disadvantage is that an image file is more than just a regular drive image. All in all there are probably more cons than pros when using this approach.

[rdmark]

@uweseimet Or, you could take your idea in https://github.com/akuker/RASCSI/issues/965 to its logical conclusion and optionally read Vendor/Product from the file name as well.

[uweseimet]

@rdmark And the SCSI ID and the LUN and the data for mode page 0 and for the SCSI level and ... ;-) Better not. Using the filename at all for meta data might have been a weak point of my ticket, but the sector size data are a different kind of data than INQUIRY data etc. They are about the physical characteristics of the drive, i.e. its format. And when thinking of the NEC drive images again, this is just the data they contain: Geometry data. But not other data like LUN, vendor etc.

[rdmark]

@uweseimet Not to argue too strongly for the idea, but I'm pretty certain that Vendor/Product/Version and block size are the four RaSCSI runtime managed properties that ought to be permanently tied to how RaSCSI interprets a particular image file.

SCSI ID, LUN, etc. are dynamic in a different sense, I would argue. They may be tied to a particular saved configuration, but the configuration itself is mutable.

[uweseimet]

@rdmark Don't take what I say next too seriously. I would like to use "?:-;*" as vendor name, and the version should contain a non-printable character. This will give a funny filename. Regarding ID and LUN you should argue with @erichelgeson, because these are encoded in the BlueSCSI image names ;-). Did you know that some QUANTUM drives support changing the SCSI ID with MODE SELECT, just like you can change the sector size this way? No joke, and in rare cases that can even be convenient. By the way, I also have a BlueSCSI now - otherwise I would not have known its filename encoding scheme ;-) - and this is the first device that immediately passes all the tests of my SCSI firmware test suite. All other "modern" (i.e. non hard disk) devices I tried before (also Atari-specific ones) always failed and needed several firmware fixes. Good job @erichelgeson! I wonder why neither RaSCSI nor BlueSCSI can transfer more than about 1.1 MB/s. The SCSI port of a Mac supports much more, and even my TT reaches about 1.8 MB/s with a real hard disk drive. With RaSCSI it does hardly make a difference whether you use a Pi Zero or a Pi 4. Looks as if the hardware is a limitation, not the CPU speed.

[erichelgeson]

Regarding ID and LUN you should argue with @erichelgeson, because these are encoded in the BlueSCSI image names ;-).

A goal of BlueSCSI is to never have the user have to edit a config file. Hence we pack as much as we can in the filename, but yes in some cases a user does need to edit a vendor/etc - we do that globally with scsi-info.txt. Not perfect but what we have.

is the first device that immediately passes all the tests of my SCSI firmware test suite ... Good job @erichelgeson!

Thanks! It's really a group effort and thanks to all the contributors! I've spent lot of time looking and trying to follow the SCSI commands as close as I could.

Back on topic, my 2c is i was against the entire properties files to begin with here - I much prefer things like .hda to do apple quirks, etc, but understand advanced users may want more. Maybe a compromise is to make a single json file key'd off path to image - you could then look it up. Would still allow advanced users to edit, while potentially addressing this issue. Just a random thought.

{
  "/some/path/disk1.hda": { ... },
  "/some/other/disk1.hda": { ... },
etc
}

[uweseimet]

@erichelgeson My remark on encoding information in the filename was not meant as criticism. Since there is no BlueSCSI remote access or client tools or a binary you can pass options to there are not that many ways to configure the devices. What I like about BlueSCSI is that you don't need much configuration, but essentially just drop the image onto the SD card. On the other hand, what I like about RaSCSI is the flexibility, because you can configure a lot, and the potential of having remote access. IMO both projects have their strengths, which is a nice thing. Regarding spending time looking at how SCSI commands have to work I know exactly what you mean ;-). But it pays off. I have had contact with authors of other SD card solutions (for the Atari's ACSI bus), and unfortunately some of them thought they would essentially just have to implement READ(6) and WRITE(6), and that would be it. Looking at the SCSI specification would not be required. The results are not convincing at all ...

[rdmark]

Fixing with https://github.com/PiSCSI/piscsi/pull/1082