[CHANGE] Align the Data Model with Wikidata Items like "file system object"

[dla-kramski]

Wikidata is playing an increasingly important role in digital preservation.

FileTrove should align its data model with this de facto standard (see https://www.wikidata.org/wiki/Q37787110 and related pages):

file system object (Q37787110)
    part of
        file system (Q174989)
    has characteristic: 
       path ((Q817765))
           has part(s):
                path separator (Q64826685)
                filename (Q1144928)
                    has part(s):
                        filename extension (Q186157)

This could be implemented with the following changes:

Session table

• New Column: filesystem
  • WD description: "concrete format or program for storing files and directories on a data storage device" (https://www.wikidata.org/wiki/Q174989)
  • Example value(s): "ntfs", "ext4", "fat32"
  • Remarks: The Wikidata description may not be the best possible... Need a controlled vocabulary for FS values.
• New Column: pathseparator
  • WD description: "character used to delimit components of a file path" (https://www.wikidata.org/wiki/Q64826685)
  • Example value(s): "/", "\"

Files and directories tables

• New Column: filepath | dirpath
  • WD description: "general form of the name of a file or directory; resources can be represented by either absolute or relative paths" (https://www.wikidata.org/wiki/Q817765)
  • Example value(s): "logs/filetrove.log"
  • Remarks: This should be relative to the sessions's mountpoint. This is identical to the existing columns filename | dirname
• New Column: filename | dirname
  • WD description: "text string used to uniquely identify a computer file" (https://www.wikidata.org/wiki/Q1144928)
  • Example value(s): "filetrove.log"
  • Remarks: We should add to the description: "Without the leading path". This is different from the existing columns filename | dirname.
• New Column: filenameextension | dirnameextension
  • WD description: "suffix to the name of a computer file" (https://www.wikidata.org/wiki/Q186157)
  • Example value(s): "log"
  • Remarks: Of course, the extension is only a weak indicator of the file format, but it does play a role. Less common for directories, but still may be useful.

This information could also be obtained by parsing the existing "filename" column afterwards, but ftrove has it all to hand on run time and can easily record it. The filename without the path may be particularly useful for tracking files with the same name across several sessions.

[steffenfritz]

This is a bigger change but it makes sense and this will be implemented. Added as a milestone for v1.0.0

[steffenfritz]

In regards of the filesystem Ross adds +1 in #43 (just don't want to lose it due to closing of that issue)

[steffenfritz]

@dla-kramski I am not sure about the extension of directory. As there is no definition of it and it really is just part of the name, how would you work with it? How would you define the boundaries, e.g. would "." separate the dirname and the extension? How should we handle if a dirname has more than one (arbitrarily) chosen separator? I understand that there are extensions like "SYSTEM" and such but these are more semantics for the user and do not serve a technical purpose like MIME connotations.

So, I'd like to not add dirname extensions.

[steffenfritz]

Added, resp. aligned with Wikidata

• filepath | dirpath
• filename | dirname
• filenameextension
• pathseparator (depends on the OS FileTrove is running on)

ToDo / To be discussed

• dirnameextension: see above
• filesystem: As this has to be done for all supported OS that FT is running on seperately, this takes some tme and testing.

[steffenfritz]

Go's filepath.Ext() is not working as expected:

filename: .hiddenfile
filepath: ../../testdata/.hiddenfile
filenameextension: .hiddenfile

The doc says: Ext returns the file name extension used by path. The extension is the suffix beginning at the final dot in the final element of path; it is empty if there is no dot.: https://pkg.go.dev/path/filepath#Ext

As the example shows, this is not a perfect approach as every hidden file on Linux/Unix without a dot extension will have it's own name as extension. There must also be the condition that the dot is not the first element in the string.

[steffenfritz]

Go's filepath.Ext() is not working as expected:

filename: .hiddenfile
filepath: ../../testdata/.hiddenfile
filenameextension: .hiddenfile

The doc says: Ext returns the file name extension used by path. The extension is the suffix beginning at the final dot in the final element of path; it is empty if there is no dot.: https://pkg.go.dev/path/filepath#Ext

As the example shows, this is not a perfect approach as every hidden file on Linux/Unix without a dot extension will have it's own name as extension. There must also be the condition that the dot is not the first element in the string.

https://github.com/golang/go/issues/66814

[steffenfritz]

golang/go#66814

As Golang will not change filepath.Ext(), I added a workaround.

[steffenfritz]

Regarding filesystem:

Significant changes would have to be made to automatically determine the file system. In addition, the execution might have to be carried out as root/SYSTEM, which is not desirable. I am considering introducing a flag that allows users to add the file system manually.

[dla-kramski]

@dla-kramski I am not sure about the extension of directory. As there is no definition of it and it really is just part of the name, how would you work with it? How would you define the boundaries, e.g. would "." separate the dirname and the extension? How should we handle if a dirname has more than one (arbitrarily) chosen separator? I understand that there are extensions like "SYSTEM" and such but these are more semantics for the user and do not serve a technical purpose like MIME connotations.

So, I'd like to not add dirname extensions.

On second thought, I'm inclined to agree.

[steffenfritz]

• dirnameextension dropped
• For v1.0.0 there will also be no filesystem detection. In a next version an automatic detection will be discussed again. I am not sure if filesystem detection is in the scope of FileTrove as it works on files, independent from the filesystem.