add file type flag (symbolic link, regular file, directory)

petersilva commented 5 years ago

In Sarracenia v02 (the protocol on which this work is based) there is a use case involving mirroring of trees of files. For that case to work, in addition to new versions of files being published, there is a need to publish events such as creation of symbolic links, and file removals as well.

In v02, the format uses the sum header to include that information. There are checksum algorithms called "L" (link), and "R" (remove) for these cases. the checksum values are just for the file name and add little to no value. There is also a weakness in Sarracenia currently, where there is no event to post for directories/folder creation, and so the permissions of those items are not mirrored. Rather, default permissions are used to fill in this information.

for v03, the sum header is now called integrity in v03, and it may become overloaded in v03 because people want to use it to validate the origin of the post via a signature of some kind. Currently in Sarracenia, v02 'L' is implemented as the v03 integrity method link, and v02 'R' is implemented as 'remove'. With this transliteration, it is likely difficult to have a signed removal or link event.

It would probably be good to:

separate Link, remove and directory events from the integrity header.

It isn't clear if we need a unified way to represent all of them, or if some combination is OK. The other problem with this is that thes are special cases for the signature algorithm anyways, which is why it was done this way in the first place. for a symbolic link, it does not make sense to checksum the file at the other end, because it might not point to the same data on both ends (depending on settings) so it needs to be a signature based on the link value itself.

Similarly, for directory, it makes no sense to digitally sign the directory, because that requires identical contents at the time of the check, and it is too dynamic for that to be worthwhile, and on different file systems the directory content will not be binary comparable anyways.

Similarly, for file 'removal', it is another special case, since the file data is not available to be checksumed.

In all these events, the meaning of the signature isn't so much to validate the content of the file, but more to

Option: lntegrity Header

Leave it as in Sarracenia v02 version, the 'integrity' header includes link, and remove algorithms, perhaps tweak the values to be more of a signature, rather than the current SHA512 checksum based on the name. add a 'folder' method to fill the small gap.

Option: event header

one could add an event header, value is one of: 'modify' (file), 'folder' (directory creation), 'link', 'remove'. This would add the 'event' header to every message, about 12 bytes. It is the most obvious/clearest thing to do, but it adds bytes.

What happens to the integrity header in this case. It needs to know that these items are special

Option: Separate Headers

Sarracenia already has a link header to store the value of the symbolic link. One could use a 'link' header, whose value is where the link points to, and that one is taken care of.
similarly, add a 'directory' header, to indicate the given file is a directory.
similarly, add an 'remove' header to flag this event as a removal.

What happens to the integrity header in this case. It needs to know that these items are special

Any other ideas? suggestions?

petersilva commented 5 years ago

Clarification: the current WMO_Sketch feed does not send any of these special events. They needed to be implemented in Sarracenia in order to get through internal validation, but have not been added to wmo_mesh demo yet.

petersilva commented 5 years ago

I added what Sarracenia does to the message-schema.json so that people know what is currently functional. I'm not really sure whether it should stay that way or if there is a better idea, but I think it is good for all the materials to describe the same baseline, and then iterate from there. (code and schema should march forward together.)

petersilva commented 4 years ago

Canadian implementation already optionally transports the mode field which includes 0777 permission bits traditional on Linux. It would be kind of normal to add bits to this field corresponding to ( S_IFDIR, S_IFLNK, S_IFREG ) ( man 2 stat on a linux system. )

Would have to ensure the bits are portable, or explicitly define them portably. would save defining another field.

MetPX / wmo_mesh