Sort speeches according to Akoma Ntoso order

[jpmckinney]

The default view seems to be a table of contents. But people don't want a table of contents; they want content! When you usually read Shakespeare, for example, you don't drill down to Act 1, Scene 1, and then start reading. You open a book, and there it is, Act 1 Scene 1 on the first page, and you can start reading the first speeches of Shakespeare's play, right then and there. If you want to jump to Act 4, that's when you look at a table of contents. I think it's very confusing to click on a parliamentary debate, expecting to see some speeches, and instead you see a big table of contents (with some speeches appearing at the bottom if this section has speeches in addition to subsections).

On that last parenthetical note, the current views seem to assume that speeches only appear in leaves. If Mr. Speaker says a few words of introduction before the first section of the debate, his speeches appear at the bottom of the page, after the full table of contents. These speeches should appear in sequence with the sections, not under them - going back to the point that the default view should be a flat view with section headings and speeches appearing in sequence, with the table of contents appearing on a different reference page.

[jpmckinney]

@dracos How does this all_speeches keyword argument get set? If I can set it, it's possible the view will look as I expect it to be.

[dracos]

The links on http://shakespeare.sayit.mysociety.org/about could go directly to the first scenes of the plays, that's a good idea, but I don't think Act/Scene numbers are comparable to the variously titled subsections of e.g. a day's debates in Parliament. As you say, there could be introductory/other speeches in a section, which might be important; if I click the link for a day's debate on TheyWorkForYou, I do expect to see the overview of debates on that day, not go straight to the first debate. Perhaps it could display snippets from any first speech present in the subsection, as http://www.theyworkforyou.com/debates/?d=2014-07-03 does, and remove the links when there are no immediate subspeeches, only subsections that are already displayed on that page.

On the display of speeches on a section page - if the speeches (or some of them) have timestamps, then a sections page tries to order its contents appropriately from that information. For example, introductory speeches before the first subsection of a debate do appear at the top correctly on e.g. http://leveson.sayit.mysociety.org/hearing-19-december-2011 (and if there were final or interleaved timestamped speeches, they'd appear in the right places too - better display of subsection links is #135).

Currently if speeches don't have timestamps of any sort, they sort with a maximum datetime, which is why then those speeches appear at the bottom of the subsection list (at one point they did appear at the top, but that was also confusing in a different way). The all_speeches variable is set in the Akoma Ntoso view in order to display its contents; I'm not sure of any performance implications and I'm not sure it would display how you want (I think it would display the entire hierarchy of a section on one page, but still not with untimestamped speeches in the right points as you want them, they'd appear within their section at the bottom).

Perhaps with sections contaning untimestamped speeches, it should fall back sorting internally by earliest speech ID - assuming insertion order is okay (which in the absence of timestamps I guess you have to assume without a manual ordering field), that might produce a better default view with untimestamped data?

[jpmckinney]

Indeed, most hansard data is irregularly timestamped in Canada, with the prominent exception of the House of Commons. Sorting on insertion order (i.e. the order of the speeches in the Akoma Ntoso file) sounds like the best option.

Edit 2: Untimestamped speeches, if originating from Akoma Ntoso files, should be treated differently from other untimestamped speeches. AN files give a definite, real order to speeches, which should be respected in the UI. Updated issue title to reflect this focus.

Edit 1: As an example of "just show me speeches", Open Parliament first displays a condensed list of topics, but once you scroll to the speeches, it's just pages of speeches. You only see the hierarchy of sections by consulting the metadata in the left sidebar of each speech.

[jpmckinney]

As mentioned in #334, an hpos-like field should be added to Speech. Another use case which requires a sequence field is when speeches are paginated, and you want to link to a speech in context. The easiest way to determine which page to link to is if you know the index of the speech you're linking to.

[jpmckinney]

@dracos In terms of priority, adding a hpos-like field is our next most important issue in SayIt.