search

wet-boew / wet-boew-api-standards

Possible requirements for Government of Canada APIs based on the White House standards
Other
11 stars 11 forks source link

better offsets into frequently changing data #11

Closed wardi closed 10 years ago

wardi commented 10 years ago

This solves a real problem we had with our site. We needed to collect all the activities since a given date, but we had to page through them with an API

Since the list of activites is continually being added to, an offset parameter isn't very good: we would get the same activities returned multiple times, making several wasted API calls. There was also the possibility of missing activities completely when an earlier activity gets removed from the list by an administrator.

Using a "following" parameter based on the sort order of the items returned solved these problems nicely, and was very easy to use. It's probably also gentler on the database at the back end too.

chrismajewski commented 10 years ago

Is this a standard or part of a standard? If so I'd like to reference the source.

I wonder if this would be better described as a section on limiting results in general. If it's a time offset it's best described by "following"+, per record by "offset"+limit or series by "page"+limit.

Also, what time format is that? GoC is mandated to ISO8601 for output, I'd prefer the same input if it's human readable. To be accurate you need a full description with a GMT offset so December 7th 2012 at 15:00:00 with a time zone offset of -4 GMT ( EST ) is: 2012-12-07T15:00:00-04:00

A case could be made for epochal input date time, as google uses for the timezone API, but that's worth a debate.

We have a running discussion here: https://github.com/wet-boew/wet-boew/issues/2834

I've made a request for input from the community, lets give them a few days and if there's nothing we'll sort it out here.

wardi commented 10 years ago

I thought the example was valid iso 8601. http://en.wikipedia.org/wiki/ISO_8601#Times says the last element can have decimal fractional values following a ',' or '.' I used the comma because that's what date -Ins gave me.

chrismajewski commented 10 years ago

To denote "14 hours, 30 and one half minutes", do not include a seconds figure. Represent it as "14:30,5", "1430,5", "14:30.5", or "1430.5".

2013-10-30T15:06:22,556645642Z

.556645642 is a really small fraction of a second. Many of the conversion functions I saw that willl accept 12 but not use anything more precise than a millisecond.

date -Ins

The nanoseconds really threw me.

I'll check it when I'm back tonight.

chrismajewski commented 10 years ago

A bit late, if not tomorrow I'll get this in wednesday. I'll pull this in and then update the language to accommodate the incrementing logic. Doesn't hurt to give people another day or so.

You now have me reconsidering any date/time example offer. Do we stay simple ( which people are already finding complicated ) or do we go complete.

chrismajewski commented 10 years ago

Thanks again Ian, the language will change for the Languages and Offset sections but they will be better by your pointing out gaps.

I look forward to your next change.