What does a WPub URL resolve to?

[prototypo]

Problem Statement

We have decided that a WPub with be identified with a URL. What should that URL resolve to?

This issue suggests four scenarios and evaluates their ramifications in the hopes of driving toward consensus. A conclusion section at the bottom of this post suggests an answer to the question posed.

Scenarios

The working group has considered several possible answers to this question. They mostly fall into one of four scenarios:

1. A WPub URL resolves to a JSON manifest file
2. A WPub URL resolves to an HTML file that contains both a table of contents (TOC) and other metadata
3. A WPub URL resolves to an HTML file containing a TOC and which links to a JSON manifest file
4. A WPub URL resolves to a binary package (e.g. a ZIP file or SQLite database file)

Each of these scenarios are considered in turn as they would be viewed by four types of clients:

• An existing search engine bot
• A WPub aware search engine bot
• A reader using an existing user agent
• A reader using a WPub aware user agent

Scenario 1: JSON Manifest

If a WPub URL resolves to a JSON manifest file, the four clients may be expected to act like this:

• An existing search engine bot (path 1A in the image below) will not index JSON files.
• A WPub aware search engine bot (path 1B in the image below) will need to index all JSON files on the Web to determine which ones represent WPubs.
• A reader using an existing user agent (path 1C in the image below) will be presented with a raw JSON representation.
• A reader using a WPub aware user agent (path 1D in the image below) will be able to see and operate upon the WPub as intended.

Scenario 1 seems inconvenient for both old and new clients with the exception of new WPub aware user agents.

Scenario 2: HTML TOC & Metadata

If a WPub URL resolves to an HTML file that contains both a table of contents (TOC) and other metadata, the four clients may be expected to act like this:

• An existing search engine bot (path 2A in the image below) will encounter and index the HTML page without modification.
• A WPub aware search engine bot (path 2B in the image below) will be able to determine that the HTML page represents a WPub, but will need to parse the entire file in a WPub-specified manner to extract metadata.
• A reader using an existing user agent (path 2C in the image below) will see a TOC containing links to the components of the WPub.
• A reader using a WPub aware user agent (path 2D in the image below) will be able to see and operate upon the WPub as intended.

Scenario 2 would be handled cleanly for old and new clients, but could cause difficulties when old clients are presented with metadata they cannot understand. There is also some danger of overloading or stretching the use of HTML to define the necessary metadata within an HTML document. New clients would need to parse the metadata out of the HTML to operate upon it.

Scenario 3: HTML TOC & JSON Manifest

If a WPub URL resolves to an HTML file containing a TOC and which links to a JSON manifest file, the four clients may be expected to act like this:

• An existing search engine bot (path 3A in the image below) will encounter and index the HTML page without modification.
• A WPub aware search engine bot (path 3B in the image below) will be able to determine that the HTML page represents a WPub, and follow the provided link to the metadata if it wishes.
• A reader using an existing user agent (path 3C in the image below) will see a TOC containing links to the components of the WPub.
• A reader using a WPub aware user agent (path 3D in the image below) will be able to see and operate upon the WPub as intended.

Scenario 3 seems to cleanly handle old and new clients in appropriate ways. Old clients could follow their noses to the components of a WPub, and new clients could easily load the JSON object to efficiently access metadata.

Scenario 4: Binary WPub

If a WPub URL resolves to a binary file, the four clients may be expected to act like this:

• An existing search engine bot (path 4A in the image below) will either be aware of the MIME type for the binary file and attempt to handle it in an inappropriate manner, or it will need to be modified to handle a new MIME type.
• A WPub aware search engine bot (path 4B in the image below) will either be confused by the overloading of an existing MIME type, or handle a new MIME type correctly.
• A reader using an existing user agent (path 4C in the image below) will probably be asked to download the binary.
• A reader using a WPub aware user agent (path 4D in the image below) will either be confused by the overloading of an existing MIME type, or handle a new MIME type correctly.

Scenario 4 could lead to confusion for both old and new clients unless a new MIME type is registered.

My Conclusion

Given the various pros and cons, Scenario 3 (the URL to a WPub resolves to an HTML TOC document, which in turn links to a JSON manifest) seems to most easily interoperate with the existing Web and provides a clean upgrade path to WPub-aware clients.

Acknowledgements

Thanks to @BigBlueHat @TzviyaSiegman @iherman @tcole3 @bdugas @GarthConboy and others from the Publishing WG for discussions which led to this issue and its discussion.

[mattgarrish]

Do we need to require toc in this, or do we only need to require that the link resolve to a resource that is considered the primary entry point for the publication, and that must include a link to the manifest?

It sounds like we're mandating the presence of a table of contents, when not every publication will need one. Using the table of contents, or ensuring the landing document has a clear link to one, is perhaps only best practice in the case of multi-document publications?

[dauwhe]

Yeah, this is one of my fundamental questions. If I point my browser at the URL of a WP, what happens?

> GET /MobyDick/ HTTP/1.1
> Host: www.example.com
> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36 OPR/38.0.2220.41
> Accept: */*

Strongly agree that the URL must resolve to an HTML document, and that this HTML document must contain a link to the manifest (or contain the manifest; I think that's still an open question).

Like @mattgarrish, I'm a little less certain about requiring this document to contain a TOC. Perhaps if it doesn't contain a TOC, and it is a multiple-document publication, it must link to a TOC? Sadly rel=contents isn't super-standard, but rel=index might be appropriate? Or is it enough that the manifest include a link to a TOC?

Do we require that this be the first document in the default reading order? I think yes. In retrospect, the whole "begin reading" thing in EPUB felt like an evasion. Saying that [1] front matter is so important that it must come at the front of the book, and [2] that it is so unimportant that we don't want the reader to actually see it, is just avoiding responsibility for designing your own content appropriately.

[iherman]

To repeat some arguments, just for the records, against option 2:

• If the metadata used the HTML elements like <meta>, <link>, or even <title>, that would be, in my view, in violation to the HTML spec. That spec clearly says "The meta element can represent document-level metadata". I.e., using that element for the Publication, instead of the for the containing document would be misusing those for a different purpose. We should not do that.
• It would be possible, to include the whole metadata in a <script> with the type set for javascript but, in practice, this may be a stretch. For example, no HTML authoring tool that I know would make that easy to do. Also, the JSON metadata should then me made in a way (using JSON-LD tricks) that the subject for the metadata in question is the Publication and the the enclosing document itself.

My conclusion on option 2 that it should not be adopted.

[dauwhe]

[edited] Ivan, are you talking about option 2? That seems to be the one where some publication metadata is expressed in HTML.

[iherman]

Oh bugger, I wanted to say Option 2! I will edit the comment...

(Lesson: never say anything serious before breakfast!)

[rdeltour]

If the metadata used the HTML elements like , , or even