Create JSon interface

[daviddurand]

All metadata properties of content objects that you manipulate within Tizra: collections, books, excerpts, pages, etc.

Notes:

• Each object already has a TizraId and maybe custom URL. JSON support will enable scriptable download of metadata about the object based on it's TizraId, perhaps custom URL as well.
• For all metadata access, fields not marked "show to user" or "include in search" will not be sent.
• Site-level metadata will not be accessible via JSON (includes advanced account settings).
• Collection contents will be accessible ++ list information and other costly properties will not be delivered on every access -- some field results will be JSon URLs that must be accessed to get further information.
• List of attachments for an object.
• List of Excerpts for a book.
• Metadata properties of user accounts. (will not include secure information about passwords, etc.) Will require that an account be logged in to access account information.
  Will include account name, email, etc.
• Check if a user can access a given piece of content.
• List of user purchases.
• List of licenses owned by the logged-in user, or groups that they are members of.
• (probably) JSON results for most forms, including search, account updates, registration, etc.

[daviddurand]

This first batch of questions has already gotten larger than I expected. It's definitely helpful to me if you can answer what you can, what excites you, and what takes your fancy. I'll be proceeding in parallel with gathering your input, and as this is the foundation for what I expect will eventually be an important API, remember that I'm evaluating options, not simply taking orders for features.

While I have to be the final judge on the technical decisions, the result should meet as many of your use cases as we can in the available time. So try to explain not just what you prefer, but why you prefer it. I'm also especially interested in knowing about libraries that you are using that have expectations about APIs, as conforming to existing (popular) library expectations increases the reach of whatever we may do.

In particular I want to move a lot of Tizra stuff to jQuery as soon as it's practical, so I think we are in rough agreement about tooling.

Questions

It seems like it might make sense to filter metadata properties so that you can request exactly the properties that you want, and reduce data traffic -- also we know that list properties and some full-text items, like abstracts, homepage descriptions, and HTML fields can tend to be large. I'm thinking of a few possible options, which would be most helpful?

1. default request mode is based on an explicit list of desired properties -- i.e. you have to ask for something to get it. Asking for "everything" is a special operation.
2. default request mode is to return a list of the most-likely useful information (some properties will be JSon pointers for further information).

Independent of these are a number of global questions about how the API might work:

• we could deliver even large information groups in a single response if and only if that information is explicitly requested. This can't be a completely universal policy, because some things (like Tables of Contents) can be so large that they don't ever make sense to transmit in one message (ASPPA is a great example of this). Static page descriptions were an instance that we saw in the problems with the Amia site some time ago. This is an easy mistake to make, so I will be trying to be cautious.
• I'm still thinking about strategies for extra-long return values. Short arrays could come in directly:

["value1", "value2" ... ]

Long arrays would return an object representing a continuable group:

{"paginated-list" : [ "value1" ... ],
   "next" :  { "JSon URL"} }

• How necessary would information on total "page" length and current position be? Would you need to control size of the "page", and the current "page" number explicitly? I'd rather keep this as simple as possible, but not simpler...
• Alternatively we could have every JSon request always return an object with a "type" field of some sort. Other fields on the object would be present depending on what type was returned. Each return value would be slightly more complex, but the javascript access would be more uniform, as you would always switch on the type. You might even be able to share Json reception functions between requests in that case.
• Since the JSon stuff is based on some internal features that are already in play, there will be some detail changes as I build it out for public consumption -- the approaches above are overviews of issues, not final proposals.
• I am still thinking about packaging for parameters on data fetching operations. URL-based parameters require escaping, and present some real length limitations that bit me in the past in the admin UI -- but I'd like to stick to real REST for this kind of API, so I'd rather not use the POST HTTP method for something that is not creating an object on the server. I'm doing some reading on best practice here, so wait for news on this.
• I am also investigating the ways the people can easily generate HTML from JSon results. Are there particular approaches/libraries that you are already using or would be especially interested in?

[daviddurand]

Should we also have a variation of the JSon Calls that returns simple HTML fragments instead of JSon?

I'm looking at JSon->HTML packages, and a lot of them are pretty considerable templating packages. PURE is one option that seems pretty nice, and Mustache is also intriguing. But these are relatively large solutions.

[orangedrink]

Thanks David.

To your first question on metadata my preference would be for the default request mode to return a list of the most-likely useful information based on which fields are marked as shown to users or maybe even an "expose to API" option that would show up with "Shown to users?" and "Include in Advanced Search?" on the document properties screen.

To the more general question I'm usually in favor of getting the big pieces of data in one big chunk. As long as good visual feedback is provided to the user a longer load time doesn't bother me and actually cuts down on bandwidth since you have less overhead for http requests/headers. it also cuts down on the work you and I have to do to split this up and put it back together. If we did need to paginate something we would definitely need to have control over page size and position but I don't think we would need to get that information passed back from you.

It would probably come in handy at some point to have a "Type" field to switch on but I'm not 100% sure I would use it. I tend to just use a separate anonymous function for each request type.

I do like URL based parameters in this kind of application too. I can't think of a reason we would need to send very long values but I'm sure there are a lot of things I can't think of now. Either way its easy for me to use post too on my end so just let me know how it will work and It shouldn't be a big deal.

As for tools that I use JQuery pretty much covers it. I would probably use a templating plugin like PureJS or Mustache. jsRender isn't quite in beta yet but I'm a big fan of Boris' work: https://github.com/BorisMoore/jsrender.

I hope this feedback is helpful!

[daviddurand]

I've finally got the doc up. https://github.com/Tizra/Tizra-Customer-Tracker/wiki/JSon

I've got to add one more thing (notes about how paging through results works), but it should be pretty usable.

[orangedrink]

I'm looking into implementing this for a replacement for the collection list block. I'm going to extract the 'url-id' from window.location.href for now but in the future it might be useful to have a macro that would return this. Maybe something like this exists already but I'm not aware?

[orangedrink]

Is there a mechanism to sort collection listings? That's important for us.

[orangedrink]

We need property sort values to be respected (or at the very least exposed so I can sort them) as well

[daviddurand]

• Sorting: There isn't a way right now, but there should be. I'll see about adding it. Should be pretty easy so maybe later today or tomorrow.
• Url-id of the current page is actually not currently available. I've just added it as a meta element in the "head" name="tizra-id" content=the id. The value would be blak on a page where there is not an identifiable object, but I don't think you will ever see that. What you will see is the "site" object id, which is the same one that shows up in webDAV URLs. You won't get access to useful properties on that object however.

[orangedrink]

OK. Thanks David.

[daviddurand]

Sorting turns out to be hard, at the moment, because of some of the optimizations that make it fast, so you should probably sort in Javascript -- It'll be a while before sorting gets into the JSon stuff.

Because the field visibility thing was turning out to be a pain, (and would be worse for sort fields), I've added a new JSon visibility flag to properties, so you should be able to make any property visible without the hacking around we were doing.

This should make it easy for you to send JSon info for sorting even if you don't want people viewing and searching it.

You should use the field filtering parameters on the JSon stuff to send just the fields you actually need -- it should make a big performance difference.

[orangedrink]

If you can't sort on your end then the paging we were promised is useless because the first load won't be loading documents that belong on the first page.

In fact isn't paging mandatory on long lists? How is that going to work?

I would have to do multiple ajax requests, aggregate all of the documents into a huge array and sort that? I don't think we would see a performance increase doing things that way.

[daviddurand]

I was not thinking of the ajax queries as a replacement for the list block -- though it is one way to handle paging with less work on the server side than what I've been thinking of. I agree that paging of unsorted lists is pretty useless, although you can specify your own paging limit using the limit parameter. It will be respected up to 1000 items, which should meet any JSon needs for page display. For data export and the like, where the lists may be very long, limits really do make sense.

I do intend to look into the sorting for these reasons, but we are working on the static export fixes now (actually interrupting the reader work), and so it can't be immediate. Same for looking at the header macros. (issue #67)

[orangedrink]

Yes. Those static export items are definitely a higher priority. I'll just hold off on JSON dev tasks for now.

Thanks David!

[daviddurand]

Since I was looking at the api anyway, I've added sorting. It's live now.

api version is now 1.1.0. The /api/ result gives the basic documentation. Wiki updates will follow.

[orangedrink]

This doesn't seem to be listing collection contents properly now. This URL

http://omniplayground.tizrapublisher.com/api/query/contents?tizra-id=2btkr8&limit=100

Should be listing 10 to 15 documents but all I'm seeing are the sites collections.

After a few tests sort-prop seems to be working.

Thanks David.

[daviddurand]

Collection contents are fixed, and can be sorted. Change is already live.

[orangedrink]

Thank you David. Now I need a way to sort the properties under each document. Would it be hard to order those by the property sort value?

[orangedrink]

Also I'm not seeing how to get the display names of any of these document properties.