Improve Format of Directory Listings

[waldheinz]

I think the JSON format of directory listings can (and should) be improved. From the current spec, a directory is listed like this:

{
  "abc": "DEADBEEFDEADBEEFDEADBEEF",
  "def/": "1337ABCD1337ABCD1337ABCD"
}

The problems I see with this are, in no particular order:

• It is impossible to specify a schema for this object. While this is not a problem per-se, it feels strange that in other places remotestorage encourages the usage of schemas, but uses such an anomaly in a central place.
• It cannot be easily extended in future versions, because there is no obvious way to add new properties to this object which would not look like files or folders. And even if we can come up with an convention to do so, it would likely be error prone to implement client-side. The lack of extension points has already been criticized in issue #4.
• It is cumbersome to parse using any language / JSON library I'm aware of (except JavaScript, where using objects as associative arrays is quite standard).

Therefore I'd like to suggest a response format like this:

{ "contents" :
  [ { "name" : "abc",  "tag" : "DEADBEEFDEADBEEFDEADBEEF" }
  , { "name" : "def/", "tag" : "1337ABCD1337ABCD1337ABCD" }
  ]
}

The proposed format

• is easy enough to translate to the current format client-side in JavaScipt (where it's likely beneficial having an assoc. array of the "old" format),
• offers obvious extension points for the directory listing as a whole (the top-level object),
• offers obvious extension points for the listed items (the objects in the list); see also issue #21 which opts for supporting HEAD requests for determining the file size. While I strongly support this proposal, I think issuing numerous HEAD requests purely for determining the file sizes would, because of the necessary server roundtrips, be slower than it should be.
• there can be a schema for this format, and because of this
• highly efficient parsers can be built using standard tools.

I don't consider my proposal "final", e.g. if it's called contents or items or whatever is not the scope of this post, it's just about the general structure. Maybe it's a good idea to add a version property to the top-level object, I don't really have an opinion about this.

[michielbdejong]

• switching from associative (js: object) to numerical (js: array) does not imho increase parser efficiency nor does it imho decrease cumbersomeness. such a big change would only be merited by a blocking reason, that is, a noticeable effect on the user experience.
• the spec is versioned, so extensibility would only affect future versions and not the current one. switching now just because we may have to switch in the future is silly - if in the future there is a need to switch from 'tag' to {tag: 'tag'}, then we can still do that in the version where that is necessary for the first time
• specifying the schema in the link is not common practice in linked data. you link to something, and it determines its own mime type and schema. also, schemas only apply to json (and possibly xml) documents, and this protocol acts at a lower level, where both the Content-Type header and the document body are opaque octet-streams. if anything, you would add the content-type in the directory listing, not the schema. the reason schemas are not mentioned in the spec is a separation of concerns between the "data layer" and the "transport layer" (i'm probably using the wrong words here, but hopefully you understand me nonetheless ;)
• adding the Content-Length in the directory listing could be a performance increase, yes, if you want to know the Content-Length of at least, say, 5% of the documents contained in a given dir. but by the opposite reasoning, people have asked for the HEAD request because they believe that there are cases where you want to know the Content-Length of less than say 5% of the documents in a directory, and so having to retrieve the parent directory listing (to which we could add Content-Length info) would be prohibitively expensive. :)

i think the driving argument here should be performance issues in the fuse client. i'm reluctant to make any changes at all to the spec unless they come from a real-world impact on user experience. sorry to be such a [insert historict strict person] about this

[waldheinz]

I did not mean that the directory listings should contain schemas for the listed entries, but I meant that there is (and cannot be) a schema for the directory listing itself. Otherwise I agree that there is no immediate need for action here, but I think it's worth keeping an eye on this because of the limitations of the current format.

So, if for some reason a breaking change has to be done anyway, this could be slipped in with no additional effort.

[michielbdejong]

ah right, yeah that makes more sense. so then it would become something like

{
  "@context": '"http://remotestorage.io/2013/directory-listing",
   "items": {
    "foo": "1",
    "bar/": "8"
  }
}

good point!

[waldheinz]

While we're at it, I'd rather go for something like

{ "@context" : "http://remotestorage.io/2013/directory-listing",
  "items" :
    [ { "name" : "abc",  "tag" : "DEADBEEFDEADBEEFDEADBEEF" }
    , { "name" : "def/", "tag" : "1337ABCD1337ABCD1337ABCD" }
    ]
}

But otherwise, yes, that's what I meant.

[nilclass]

And possibly also:

{ "@context" : "http://remotestorage.io/2013/directory-listing",
  "items" :
    [ { "name" : "abc",  "tag" : "DEADBEEFDEADBEEFDEADBEEF", "type": "text/plain" }
    , { "name" : "def/", "tag" : "1337ABCD1337ABCD1337ABCD", "type": "application/json" }
    ]
}

[steventebrinke]

The proposed structure looks fine to me, it certainly is more extensible than the current structure. It seems that the purpose is mainly to give more information in the directory listing, instead of using additional HEAD requests. If that's the purpose, IMO, it would be easiest if all property names are the same as the HTTP headers. (Since HTTP headers are case insensitive, requiring property names to be lowercase might be good). For example:

{ "@context" : "http://remotestorage.io/2013/directory-listing",
  "items" :
    [ { "location" : "abc",  "etag" : "DEADBEEFDEADBEEFDEADBEEF", "content-type": "text/plain" }
    , { "location" : "def/", "etag" : "1337ABCD1337ABCD1337ABCD", "content-type": "application/json" }
    ]
}

This makes it very extensible, since any HTTP header can be used. A schema would not really be required, if specified that the meaning of each property is equivalent to that of the HTTP header, but a schema could, for example, specify that both location and etag are required and all other properties are optional.

[waldheinz]

I would rather not bind the format of the directory listing to the underlying transport (i.e. HTTP right now). This might cause trouble if another transport is used, like local file access or WebSockets or whatever might come.

[steventebrinke]

I don't see why that would be a problem, since the whole protocol is built on top of HTTP, so for another transport, a major revision is required anyway. Besides that, even when another transport is used, names can still be mapped to the ones used by HTTP.

[michielbdejong]

let's do it like https://github.com/remotestorage/spec/issues/26#issuecomment-22315722 in -02, that is the minimal change that makes us able to say we're part of the linked data revolution

[michielbdejong]

oh, and combine it with https://github.com/remotestorage/spec/issues/16

[michielbdejong]

postponing until -03, due to lack of people to who would have time to implement this right now.

[raucao]

I would do it. And I badly need it for 2 apps I'm working on. And I guess I'm not the only one, since basically everybody except you wanted to have it during the unconf, including a core maintainer.

[raucao]

Also, does anybody have any problem with @steventebrinke's proposed format? I think it would be a great solution.

[michielbdejong]

done