Should the entry page be optional in the package?

[llemeurfr]

The current consensus is that a WP MUST have a Primary Entry Page, i.e. an HTML page.

I has been argued by the Audio TF that audio books don't have HTML pages and adding a dummy page to a package would be a burden. Note that in this situation (no HTML), a JSON ToC should exist in the manifest or the ToC must be inferred from the track listing (i.e. the reading order)... this is discussed in https://github.com/w3c/wpub/issues/369.

On the other side, I was made aware of a particular situation (Audiolib in France) where a book had a beautiful graphical ToC and the audio publisher integrated an image of this illustration in the audiobook as supplementary content and it would have been great to be able to use it as a real ToC. In such a situation, having an HTML entry page makes great sense.

When a package is exposed as a Web Publication, it is easy for a processor to create on the fly an entry page if there is none in the package. This HTML page will have a link to the manifest (or will embed the manifest) and its content will be created from the metadata found in the manifest. This is a tiny development.

We could therefore choose to have the HTML entry page optional in the package. It would offer simplicity for basic use cases and guarantee great results for advanced use cases.

Note: The other solution is to conclude that a JSON ToC is not an option and that an HTML Toc is imposed to audiobook publishers.

[GarthConboy]

I'm somewhat torn here. I view the entry page as unneeded fluff in a packaged audiobook, and thus can see disallowing or discouraging it. On the other hand, if we're going to have an HTML ToC (which I don't object to), there is no reason that couldn't be basically the only content of an entry page.

So, I guess, for me, it comes down to a JSON ToC decision -- if that's the path we go, I'd obviate the entry page (for packaged audiobooks).

[iherman]

Regardless of the ToC in JSON issue, I would not want to disallow the HTML page. I think we all agreed that the packaging format we define would not only be for audio books but, possibly, for other profiles, too, like visual narratives. Having an exception for audio books is a bad idea from that standpoint...

[HadrienGardeur]

I find this discussion quite confusing: the requirement for having an entry page in a packaged publication has nothing to do with the ToC.

As a reminder:

• the ToC is optional, it's never a requirement
• it doesn't have to be in the entry page, it can be in a different HTML document linked in resources

Regardless of the ToC in JSON issue, I would not want to disallow the HTML page

There's also a massive difference between not requiring an HTML entry page and disallowing the entry page.

The entry page is completely useless for the packaged audiobook use case, it was mostly added to WP in order to always have a fallback option.

For packaged audiobooks:

• most of them won't have a URL that returns HTML as their main identifier
• none of them will be consumed by a "non WP aware" UA, since it'll require the UA to at least know about the packaging format

Based on this, I would highly recommend to:

• require an external manifest at a well-known location (manifest.jsonld) for packaged audiobooks
• allow but not require an entry page at a well-known location (index.html)

Forcing something (entry page) for the sake of consistency is IMO a bad approach (EPUB did a very similar mistake by forcing an HTML wrapper for FXL instead of allowing images in spine, which was a foolish decision that the whole ecosystem still have to struggle with today).

[llemeurfr]

I agree with @HadrienGardeur 's conclusion, which has also some implication on #34.

[iherman]

See https://github.com/w3c/pwpub/issues/32#issuecomment-45474530.

I would modify this by saying that allow but not require manifest.jsonld _or index.html, but require that at least one of the two should be available. This should be the case, imho, for the general case.

Whether a particular profile, like audio books, would make further restrictions, that is another matter.

[dauwhe]

I would modify this by saying that allow but not require manifest.jsonld _or index.html, but require that at least one of the two should be available. This should be the case, imho, for the general case.

Whether a particular profile, like audio books, would make further restrictions, that is another matter.

Packaging seems to divide neatly into two aspects: how to package, and what to package. Would there be utility in leaving the "what to package" to the profiles, and just use a "packaging" spec to define the restrictions on ZIP?

[iherman]

@dauwhe this for me is the last resort (but is doable). I would like to maintain as much unity among profiles as possible; putting another way, minimize the compartmentalization of these.

[TzviyaSiegman]

I think it is worth considering @dauwhe's proposal seriously. Why should a packaging spec go beyond HOW to package? I believe that we are beginning to define details of the HOW based on requirements for specific file types. As we look at different modules, we will likely encounter different requirements. Perhaps restricting the definition of the package to HOW will enable the modules to coexist more peacefully.

[GarthConboy]

I can agree with Hadrien's last two bullets above.

I will push back a little on

the requirement for having an entry page in a packaged publication has nothing to do with the ToC

my linkage of the two was driven by the still undecided serialization of the ToC -- if it's only HTML, it might as well be in the entry page (though, said wouldn't be required). That's why I'd like to resolve this issue at least with eye out to where/how the ToC is encoded.

[iherman]

This is just making a note. If the decision is that the primary entry page is not necessary, then the canonicalization section must be updated. Indeed, that algorithm makes use of the Document DOM Node of the primary entry page. The algorithm text must be modified allowing that parameter to be undefined.

[iherman]

The canonicalization referred to https://github.com/w3c/pwpub/issues/33#issuecomment-456734392 ensures the default settings of the 'name' (ie, the title) and the reading order (consisting of the single entry of the primary entry page); these would be required to change in the spec.

[HadrienGardeur]

I think that in general, if the scope and the name of the spec is changed to "Web Publication Manifest" instead of "Web Publications", most of the text about the entry page should be changed.

The first use of WP will be for audiobooks: they don't need an entry page. Visual narratives (comics, mangas & BDs) won't need an entry page either.

This goes beyond the canonicalization, it also affects the core requirements since a packaged audiobook won't have an address and therefore won't have an entry page at that address either.

It also affects the title:

If not included in the authored manifest, the user agent MUST use the value of the title element [html] of the Web Publication’s primary entry page, if present, when generating the canonical manifest.

[iherman]

@HadrienGardeur I disagree. I think we should minimize these changes. The current agreement is that the WPM is, as much as possible, general and not audio book specific only. That is true even if the current rec track work is on audio books only.

[HadrienGardeur]

Well, if you go down that road, each profile will need to deviate from the core spec. I don't think that's a very good strategy.

The current WPM is not general at this point, it's a relic of how this spec was designed (as "Web Publications" instead of "Web Publication Manifest") when affordances were still considered to be within the scope of the core spec.

[llemeurfr]

To sum up the current state of the discussion:

• the WG has decided to create a packaging mechanism based on the ISO zip specification, which defines a well-know location for the manifest and the html entry page.

• such container should be usable for any variant of web publication one decides to package, i.e. audio publications, digital visual narratives or academic papers.

• html entry pages and perceived as a burden for the distribution of packaged audio publications (same for comics). But they are perceived as necessary once the audio publication is exposed on the Web.

• an html ToC is rare in the audio publication sector. If it exists it is referenced from the manifest. The possibility to create a json ToC belongs to another issue.

• problem: the Web Publication specification is (still) about both the json manifest AND the html entry page, therefore an entry page made optional in the package will impact sections like manifest canonicalization (see #issuecomment-456734392 and following).

• proposed solution: if the entry page is missing in the container, it can be generated on the fly when the content of the container is processed to become a Web Publication (either dynamically using a "streamer" or via a simple batch script). It means that to create a WP from a package, a basic unzip is not enough and a simple pre-processing is required. Let's call it "WP-setter".

[iherman]

@llemeurfr,

my proposed solution is a bit different (actually, I am actually not sure it is very different, maybe just more detailed), taking into account that the lightweight package is for WP in general and not only for audio books:

• Either the HTML entry page or the separate JSON-LD Manifest MUST be in the package. It is not required to have both, though.
• The user agent MUST first look for the entry page with the name index.html
  • if found, it MUST follow the rules as described in the WP spec to extract the Web Publication Manifest;
  • otherwise, the user agent looks for the Web Publication Manifest with the name manifest.json to extract the Web Publication Manifest.

This means that, e.g., audiobook publishers have the possibility to ignore the entry page when creating the package, whereas a scholarly publisher can use the (more natural) embedded manifest. And anything in between, with mixtures of HTML and other media.

I do realize that this is a bit of an extra load on user agents, because they must be prepared to go the extra mile to parse the HTML entry page and extract the manifest. On the grand scale of things I do not think this is a really a big deal, however, compared to the overall job of creating a decent user agent (this is also based on my experimentation which has proven to be a pretty straightforward).

This also allows for your "WP-Setter" approach if a packaged publication is to be turned into a bona fide WP.

[HadrienGardeur]

@iherman that approach is fine (except that we've been using manifest.jsonld instead of manifest.json, which is more unique to WP).

Specific profiles can then further restrict things, for example the audiobook profile could require the presence of a separate manifest.

That said, while the packaging spec would be lightweight enough to accommodate the needs of dedicated profiles, that's not the case of the WP spec itself. Including a url in the manifest and responding to that URL with an entry page are still requirements in WP, which is problematic when the entry page is not required in the package.

[iherman]

@iherman that approach is fine (except that we've been using manifest.jsonld instead of manifest.json, which is more unique to WP).

Yep, you're right, it should be .jsonld

Including a url in the manifest and responding to that URL with an entry page are still requirements in WP, which is problematic when the entry page is not required in the package.

Sure, if we go down that line, we will have to review the WP spec with this special goggle on...

[llemeurfr]

@iherman I can buy this "entry page OR manifest" in the container. The overhead is of treating both cases is already there for WP user agents.

[GarthConboy]

I can buy this too. Almost kinda like it. :-)

[iherman]

If the final decision is that we MUST have an entry page to make the content directly usable on the Web, here is the minimal PEP that is needed:

<html>
    <head>
        <link rel="publication" href="manifest.jsonld">
    </head>
</html>

By adding this, a (unpacked) package is a bona fide Web Publication (in case the name of the JSON-LD manifest file is fixed for a package).

[geoffjukes]

@iherman to my mind the entry page and manifest serve two different purposes; The Manifest describes the package, the entry page describes how to display it.

It seems to me that the manifest is required in both audio-only and 'web publication' packages. The entry page is not.

Therefore, what would be the implications of making the manifest required, and the landing page optional?

If a publisher desires to control the presentation of the audio, they can include a 'landing page resource' in the manifest. The reference could then be to package-local resources OR web-based resources.

For example a distributor may have a customized web-based audio player that is linked by reference, rather than packaged.

I'm not a web developer, so I won't pretend to understand if this is sensible.

[iherman]

@geoffjukes

@iherman to my mind the entry page and manifest serve two different purposes; The Manifest describes the package, the entry page describes how to display it.

on a very high level, that can be said (although in the current set up some items, like the title, MAY come from the PEP (ie, the primary entry page, ie, index.html).

Therefore, what would be the implications of making the manifest required, and the landing page optional?

That would not work on the Web. A usual user agent cat do something with any HTML file; it the only thing it sees is a bare JSON file, it can (possibly) display it as a text, but that is as far it goes. The PEP, as HTML, can serve (beyond the display proper) a bunch or roles: set up the 'origin' for the publication that will be important for security reasons if any kind of JS file is used, create an environment (almost a minor operating system) to run those JS files, etc. HTML has turned into the starting point for just about everything:-)

Hence the PEP is a MUST for the publication on the Web as some sort of a starting point. That also means that if a packaged publication ever wants to be used (unpacked) on the Web, then the PEP must be present.

The other way round... well, that is the issue.

[geoffjukes]

@iherman Thanks for the clarification, that certainly makes sense.

In essence, the PEP becomes the 'is web enabled' indicator, and would therefore be required in the end-user package. It would not be present in the business-to-business packages, and would not be required (but there is no need to call out that distinction in the specification).

I would be amenable to generating the minimal PEP when we are "packaging for the web", with the caveat that I would like the manifest be required, as it is the most useful component from a business to business perspective.

[mattgarrish]

Perhaps part of the problem is that we're trying to base everything on Web publications instead of a broader concept of a publication that is inherited by Web publications, audiobooks, etc.

A cleaner separation of concerns might lead to a model like:

Publication manifest |--> Web Publication
                     |          ^
                     |          |
                     |          v
                     |--> Audiobook

where all our outputs are considered modules that inherit from a common publication manifest, with perhaps no required metadata and certainly no mention of structure. That way we can introduce different required metadata, other ways of harvesting metadata, different manifest/pep discovery models, etc. etc. on a per-model basis.

There should always be paths between the formats, but it won't always be true that one format is necessarily the other all the time.

I think our last revision to take out the affordances got us halfway to this model, but we'd still need to separate all the Web-specific structures and metadata requirements.

It's not what we set out to define, of course, but maybe is a more pragmatic model?? (I could be out to lunch on this comment, too, as I haven't thought it through all that deeply. :)

[TzviyaSiegman]

I am having emoji issues today, but i very much agree with @mattgarrish.

[geoffjukes]

@mattgarrish I believe this concept is compatible with the work that @dauwhe has been doing (and to a lesser degree myself).

Speaking for myself, everything starts with a manifest. That being said [and assuming a minimum set of requirements such playback order, audio file name/size/length/checksum, supplemental file name/size/checksum, etc] we should be able to generate a manifest based on the current delivery packages that we receive from publishers, simply by analyzing the files.

In one of our more complex products, we package audio, html, and media overlays, to provide a 'read along' experience. We do this with a custom player, but I throw it out there as an example of "There should always be paths between the formats" - as it is more accurately a hybrid of two :)

[mattgarrish]

Right, the manifest is the key to every format because otherwise you just have html/audio/etc. It's always been what has bound all the resources together. So long as it is common, there will be various ways of translating content, and not strictly from one to the other as you say.

I think the problem has been that we started by analyzing one module without realizing it was just one module. As we discussed on the call today, I don't think it's imperative that everything derive from web publications, but it's taken moving on to another module, audiobooks, to realize that we've arguably been modeling too much on one possible rendering.

That said, I still think there's going to be some complexity in remodeling what we have. What is a canonical manifest, for example, if data harvesting may not apply to every module? These are eminently solvable problems, of course, but might take some creative thinking to avoid needless duplication. I'm going to see what I can make of splitting out the manifest from wpub, as a first step.

[geoffjukes]

@mattgarrish As a pure audio publisher (primarily) I have struggled with the term 'readingOrder' to describe what is more appropriately a 'listeningOrder'. How much separation do you foresee in the modular approach?

Metadata (book title etc) is common (potentially) but 'runtime' would only be relevant to the audio component of an audio-enabled package. So would that live in an 'audio' segment? or would a 'listeningOrder' imply audio, and a 'readingOrder' imply HTML etc? What about the media overlay, would that rely on the existing SMIL? or could we consider a future datapoint 'mediaSync' or something?

Rhetorical questions really.

[mattgarrish]

I have struggled with the term 'readingOrder' to describe what is more appropriately a 'listeningOrder'. How much separation do you foresee in the modular approach?

That's an interesting question. My gut says we should find a single term to avoid over-complicating the process, as both identify the same basic concept - the ordered progression to follow in the absence of user divergence. There's seduction in defining a model that anything can be up-translated to, but the downside is that this can be terribly confusing to authors, as well as developers.

But this is the sort of case where we need to find consensus for all stakeholders; it'll be part of harmonizing the formats. Perhaps there is a more neutral term than reading order, or maybe people decide my view is wrong. :)

Assuming a single property, though, the reading/listening aspect would be influenced by a combination of the type of resource the manifest identifies itself as (e.g., via its schema.org type) and the media types of the resources referenced in the reading order.

I'm not sure about the future of SMIL, though. @danielweck and @marisademeglio have been spearheading how SMIL/media overlays will be made manifest in web publications and audio books.

[dauwhe]

As a pure audio publisher (primarily) I have struggled with the term 'readingOrder' to describe what is more appropriately a 'listeningOrder'.

I always liked "sequence" but got outvoted.

[mattgarrish]

I always liked "sequence" but got outvoted.

Might be time to start a new issue in the wpub tracker given this feedback about audio, otherwise we could hijack this issue. :)

[iherman]

This issue was discussed in a meeting.

• No actions or resolutions
  View the transcript

  primary entry page in a package
  Wendy Reid: https://github.com/w3c/pwpub/issues/33
  Wendy Reid: Next item - the primary entry page. If anyone is not prepped up on this, it’s because the github is in the Packaged web publication github - so I don’t blame you for missing this one.
  … picking up on the github comment, we might have come to a mild consensus. Ivan had a great comment: The question is about the primary entry page - and whether it should exist or not.
  Wendy Reid: The current consensus proposal: Either the HTML entry page or the separate JSON-LD Manifest MUST be in the package. It is not required to have both, though. The user agent MUST first look for the entry page with the name index.html if found, it MUST follow the rules as described in the WP spec to extract the Web Publication Manifest; otherwise, the user agent looks for the Web Publication Manifest with the name manifest.json to extract the Web Publication Manifest.
  Wendy Reid: Ivan do you want to elaborate?
  Ivan Herman: Are there any questions?
  Benjamin Young: If we disallow the index.html or make it optional - we’re going to make some sorta-works-on-the-web and sorta-not - so I’m finding it confusing. The more we diverge from ‘unzip this and put it on the web’ VS the naming conventions of a zip file…
  … so some of this is related to our vision of packaging - if this is a web package, or a package for moving around pieces of the web, that are themselves webby and have a 1:1 mapping to something on the web. Or if something has webby pieces but isn’t meant for the web, like epub…
  … so defining which one we are after defines if the landing page is browser ready is required or not.
  Deborah Kaplan: To speak to the same concerns as Benjamin - there are - if we say that index.html is optional - then we are … by doing so - we are saying that either it won’t natively work - or be openable in a browser - or we have strong expectations that the native browsers will become WP aware…
  … because this is a standard… OR - that we are saying - they are webby as long as you have a polyfill or some sort of extension in your browser. Those are the things we are saying if index.html are optional, but only if we’re trying to say one of those things.
  George Kerscher: I think I have the same question as Deborah - but stated my own way - if there is no index.html, and the JSON-LD is referenced, the expectation is that the index.html will be generated from the information in the manifest. Is that correct?
  Laurent Le Meur: Yes
  George Kerscher: would that be a script/polyfill that would ship with the manifest? Or would we expect the browser to fill that.
  Wendy Reid: It could be either-or
  Laurent Le Meur: As you said, it could be either or. We’re speaking first about the package. So the package would not be consumed by the browser. It would need to be translated first. The conversion will involve the generation of a landing page.
  … it can be done statically or dynamically. By having the entry page option, we care keep it mandatory and generate automatically.
  George Kerscher: If the script is transferred in the package, that would auto-generate the index. Would a browser who is aware - would they know not to do that and run their own?
  Bill Kasdorf: this is sounding messy
  Deborah Kaplan: Bill_Kasdorf agreed
  Laurent Le Meur: it’s open - and it’s outside the packaging mechanism itself. If we set the scope is ‘what do we put in the package’ then it’s out of scope - if it’s ‘what will the user agent do’ then it is a WP aware agent? Then it could even process something that has no index.html. I don’t think it’s the best, but it could. If not WP aware, it must have an embedded reader, and this reader could be inside…
  … the package - but it depends what we do when we explode the package. It comes down to what we do within the package.
  Ivan Herman: Lets look at it from the history - It’s easy to get lost in the details. As an aside, we are not talking of making the primary entry page optional - so something that is on the web MUST have a primary entry page. Lets put that issue aside.
  Deborah Kaplan: thank you for clarifying that Ivan; that was unclear to me!
  Ivan Herman: The issue is - if you look back - the whole discussion that has been going on - is if you have a publisher who publishes an audiobook, and the only thing they want to do is publish it as a package thing and carry it from one place to another, and they don’t care about the web - for those, having the primary entry page is completely unnecessary burden…
  Geoff Jukes: +1 Ivan
  Ivan Herman: we can say we don’t care about that use case, and we care when everything is on the web, but I don’t think we should. We should have a situation where we say: “if a publisher cares about publishing their content on the web, they should zip it up and it can be a perfectly valid audiobook…
  George Kerscher: Based on Ivan’s explanation, I am a happy guy.
  Ivan Herman: and the UA should be able to render it. But if you have a publisher who doesn’t care about the web, then they should be able to create a WP without the index.html. If you take a package without it, then the primary entry page can be done pretty easily. And that whole thing is orthagonal if the browser works with a polyfill or not.
  Bill Kasdorf: so we are saying you can distribute a publication on the web that is not a web publication. I buy that; e.g. a PDF can be distributed on the web.
  Laurent Le Meur: great explanation Ivan
  Ivan Herman: that is not what we’re talking about. We want to give an option to publishers who do not want to publish to the web. That’s the sort of consensus solution we came up with - no more no less
  Tzviya Siegman: Thank you for clarifying Ivan. I think that when we put this in the spec, we need to make it much more explicit what the UA does when it encounters these things. What happens if you encounter both. I expect this is where the concern is coming from - right now our primary instance is audiobooks…
  … in the world we’re living in, it’s just something that lives in the package. We don’t have the use case of the audiobook written on the web or living on the web. I think that we’re hearing - I’m concerned that if we have other types of publications - this duality is going to create problems.
  Mateus Teixeira: +1
  Tzviya Siegman: so if I want scholarly packaged material, having these options will create problems down the line. So the concern is about tunnel vision when it comes to audiobooks - so we’re doing things in a way that is too closely tied to audiobooks.
  Bill Kasdorf: +1 to Tzviya
  Bill Kasdorf: if we do this we just need to be very clear that that audiobook is not a Web Publication
  Dave Cramer: I think that variety can help us. We have an industry need for packaged audiobook format that seems largely to industry publications. There seems no need for an index.html. I fear that trying to use the full machinery for web publications on this use case will prevent adoption or scare publishers away. We don’t have to use every piece everywhere. We’ll have…
  … more flexibility if everything has to use everything - such as having an HTML page where we don’t need it out of theoretical purity. We try to solve our audio use-case as best as we can. It’s best if we can find simple pieces that work in large spots of our use cases, but we won’t find one solution for everything…
  Ivan Herman: +1 to dave
  Dave Cramer: i don’t want cross pollination making things worse.
  Benjamin Young: A bit off what Dave said - and Laurent’s point of polyfills, if it’s for audiobooks not on the web, eschew the index.html - but since it’s for audiobooks, and possibly on the web, we’ll need the HTML page to polyfill…
  … to assume something somewhere will generate an HTML file - it won’t compute. The only way to polyfill is via a javascript, and you do that through an HTML file, otherwise it’ll just show you the JS code - so html is the conduit…
  … including things that read JSON. Sure - browsers display HTML files for images, but images have been here forever. So to think that browsers will display HTML files for the custom JSON we’re creating… that’s a bit far…
  … so when it comes to audiobooks, if we just want something that reads audio files, in a package, then lets create something new with a different name, that handles those things properly… But if we want to focus
  … on browsers and make things work on the web, then lets focus on that use case.
  Nick Ruffilo: I like bigbluehat’s comment, we’re focusing on web publications, we not only have to think of the now and the past, but also the future
  … nothing is stopping a publisher from using things against spec, if someone is only using it to package audio, and may not have a index.html
  … anyone who is new, and wants to develop an app that reads audiobooks in the browsers, they would have an index.html to reference to and it allows for availability
  … maybe it is too much for implementors, outputting an HTML file does not seem like a stretch for software, it is not rocket science, I think as much as we can keep the web in mind, there is value in that
  Deborah Kaplan: - I will not -1 on Ivan’s language - it’s not a hill I’m going to die on. I think it’s a bad idea to reopen the issues at writ large. We have resolved some things and I don’t think they are broken…
  Ivan Herman: +1 to dkaplan3
  Tzviya Siegman: +1000 to dkaplan3
  Deborah Kaplan: what I do think is true - in my experience - working with usually small publishers. One thing that is most confusing is anything that is optional. In practice, reading systems will not deal the same way with things that are optional…
  Mateus Teixeira: +1 dkaplan3, bigbluehat, NickRuffilo
  Ric Wright: +1 tzviya
  Benjamin Young: +1 there is no SHOULD or MAY there is only MUST
  Deborah Kaplan: perhaps if we say something like: “this is optional ONLY if there are no HTML pages in the document…” because anytime anything is optional, publishers will get it wrong and reading systems will not get it consistently - take for example covers in epub…
  Matt Garrish: A lot of what I was going to say has been said. The problem that we’re trying to solve is the distribution of audiobooks. The web is a nice to have. We’re trying to enforce that everything must be a web publication. We should be looking at harmony of the two.
  … if the primary is distributing audiobooks, great, but if the publisher wants to publish to the web, then we should make an easy path for that - but I would rather find a harmony in terms of allowing the author decide what it is they want to do.
  Ivan Herman: +10000 to matt
  Luc Audrain: +1 to Matt
  Matt Garrish: let it work for audio publishing, but not disallow the index, but don’t push everything into one box because then there is too much compromise for requiring the index.html
  Avneesh Singh: the entry page has another important function - it gives access to the non-WP aware browser. It was not the original plan, but it became the gateway for a non-WP aware agent to actually display the content to the user
  … if we say it’s optional, in a way, it looks OK, but it means we’re saying that the entry page can be neglected forever. So the question is - how much effort are we asking them to make…
  Benjamin Young: +1 to Avneesh
  Avneesh Singh: it can be easily automated by a production tool. Slowly we can encourage them to make a TOC and have it more valuable to a non-WP aware browser, but it’s hard to get people to come back to that later.
  Mateus Teixeira: +1
  Avneesh Singh: I would like to discuss again - and all of these problems are focused that we’re focused on audiobooks. We don’t want things skewed because of this.
  Deborah Kaplan: +1 that it can be easily automated in production
  Wendy Reid: +1 Avneesh
  Ivan Herman: A number of things - We are influenced by the Audiobook profile, but I don’t think that it’s true to say that we’re working exclusively with the audiobook profile. For example manga may want to be distributed only via packages, which hit only the same problems as audiobooks.
  … Nick said some things I don’t agree with - the text that I propose made it very clear - and it must be part of the document - a user agent MUST understand a primary entry page and handle it accordingly. A compliant user agent can understand a WP. The choice is up to the publisher and the publisher only…
  … it’s up to the publisher if they want to address a specific area or not. A scholarly publisher will have a primary page and it will handle it. But the publisher does not want to go that way because the only constituency only have special vendors, then they won’t have to do it… This is the only goal…
  … you might say that adding that extra primary entry page is no big deal - but on the other hand, if there seems to be no consensus in having that, why forcing it?
  Geoff Jukes: Speaking as an audiobook publishers, aggregator, and distributor. From our perspective - the index would be an indicator if the publisher has desires on how the audiobook is presented.
  … I could well imagine a publisher wanting to theme all their books with the same font. Certainly, home publishers will never have the warewithall for making index.html for all their books - they will rely on aggregators to “package” their books in the way they see fit.
  … From my perspective - i came from the data driven point of view - the manifest is the most important part of the publication. From a data perspective, how to deal with it. What you call reading order…
  … any additional items like bonus materials - it’s described in the manifest. The index.html would be this desire for themeing of how to present. As a distributor, we have our web sales. We read in a JSON file now - then render the content themed for our site…
  … we use HTML5 to pull in the audio as needed - it’s a thin client. In summary - the manifest is the thing that describes everything. The index, which seems optional to me, would be a way for publishers to define their theme
  Bill: I hope a simple question - What we’re calling the audiobook format - the simple format without the index.html, is there anything about it that would prevent us from distributing any proprietary not webby content.
  Wendy Reid: If you’re referring to the packaging format - that’s not audio specific
  Bill: so it could be any non-web content. This is why I’m wondering how it can be part of our web-publication work. Unless we specifically say ‘we’re addressing non-web publications in our work’
  Nick Ruffilo: +1
  Bill: Two common formats that I want to get on the table. PDF - a perfectly fine way to distribute PDFs. Something more amenable to this group - is it a way to distribute a 3.N that is not a web-publication. It’s still an XML, etc. You see where I’m going
  Laurent Le Meur: ok that’s what I wanted to say also
  Ivan Herman: Reacting to what bill said - if I take a web publication today - if there’s a resource that’s accessible to the web - like PDF, that’s already a web publication, it doesn’t matter what resources you refer to.
  … so we can have a web publication today that refers to a PDF or an epub. I don’t think that’s such an issue in this case. Let me go back to a question to Geoff. If I take the position that we MUST have an index.html, the minimal index.html is, as I said, 5 lines with no body and a link to the manifest file…
  … We could, and that was part of the discuss, require the minimal index.html be part of the package and then close it - the feedback I always got was that the audiobook publishers will not want to do that, they’ll run away screaming…
  Geoff Jukes: I cannot speak for the publishers, but for blackstone, the idea of injecting an index.html is trivial. We have audio delivery systems that injest 10s of thousands of books, and we repackage them for our distribution partners.
  … if the index.html becomes required - because it doesn’t describe the book directly, it will be difficult. What we end up doing is that we end up generating the file itself…
  Ivan Herman: The manifest, as we define it, there’s just an index.html that refers to the manifest and allows the browser to read and find that page.
  Geoff Jukes: I agree with that and from my perspective, I would turn it on it’s head - if there’s an index… For me, I would treat the manifest as - if there was an index or multiple index, they would be linked within the manifest…
  … so the manifest would be mandatory, not the index.html and all browsers and such look for the manifest, and then figure out the landing page from there. Are you following?
  

[iherman]

I like what @mattgarrish described in https://github.com/w3c/pwpub/issues/33#issuecomment-460369726, though I am a bit concerned about the complexity thereof, as well as the possible proliferation of mutually incompatible modules. The original consensus proposal included the fact that a user agent remains compatible with a packaged WPUB without further ado, and the choice of adding (or not) a PEP is exclusively in the hands of the publishers.

I would also note that, as showed in https://github.com/w3c/pwpub/issues/33#issuecomment-460354871, the minimal PEP to be added to make everything compatible is ridiculously simple, and that index.html file could be used as an indication of the nature of the package. A bit like EPUB's mime type file. If there was an agreement among audio publishers that it is fine to add that to the package, we would be done without a major damage...

[geoffjukes]

@iherman I am warming to the idea of making the PEP required based on the minimum of linking/referencing a manifest, and I concede to your point on compatibility.

that index.html file could be used as an indication of the nature of the package. A bit like EPUB's mime type file.

I would like to combine this thought with #34 and particularly the reference to @dauwhe comment on embedded manifests.

By requiring an embedded PEP, the publisher is effectively saying “this is a web publication”. They then have the opportunity to define the manifest to use for the ‘Web Publication’ version of the package. This would be extremely useful for adoption, I believe, as the package itself could have multiple manifests for different applications. It would also allow manifests to be hosted external to the package itself.

On the subject of #34 (as it is a companion ticket to this) I would say that the PEP is required inside the package, but the manifest (which is also required) must be accessible via the reference in the PEP.

[edit: I’m having a difficult time keeping track of the myriad conversations, so i apologize if I am making statements that have already been made/assumed.]

[iherman]

@iherman I am warming to the idea of making the PEP required based on the minimum of linking/referencing a manifest, and I concede to your point on compatibility.

Wow. This may be a way out of the current deadlock...

On the subject of #34 (as it is a companion ticket to this) I would say that the PEP is required inside the package, but the manifest (which is also required) must be accessible via the reference in the PEP.

That is already the case. This is what makes a WPUB, in fact: the manifest MUST be either referenced or included in the PEP:

The primary entry page is the only resource in which a manifest can be embedded. To ensure discovery of the manifest, the primary entry page MUST provide a link to the manifest, regardless of whether the manifest is embedded within the page or external to it.

At the moment the manifest may either be embedded in the PEP (in JSON-LD) or referenced from it if the manifest is a separate JSON-LD file. #34 is based on the assumption that if the WPUB is packaged, the manifest MUST be a separate file; on the other hand, there is another open issue (https://github.com/w3c/wpub/issues/327) which would require just about the opposite: the a manifest may appear within the PEP only. At the moment we do not have consensus on neither of these two:-(

[HadrienGardeur]

If the final decision is that we MUST have an entry page to make the content directly usable on the Web, here is the minimal PEP that is needed:

<html>
   <head>
       <link rel="publication" href="manifest.jsonld">
   </head>
</html>

For me this perfectly illustrates why we shouldn't have the entry page as a requirement in any package:

• the only requirement that makes sense for the content of the entry page is the link to the manifest (or an embedded manifest), this is better achieved with a well-known location to a manifest (manifest.jsonld) in a package
• while we've discussed in the past the idea of requiring the entry page to provide an entry point in the publication, I believe that spec-wise this is extremely difficult to have as a requirement (it's too vague), plus a link to an audio resource (in the case of an audiobook) is not enough to go through the entire publication
• the entry page serves two primary goals: discovering the manifest and as a fallback for non WP-aware UAs, but non WP-aware UAs won't be able to open packaged publications anyway and we need a lot more than a link for a fallback

This minimal entry page would bring zero value to the table, it just makes a packaged audiobook more complex to produce.

This group has widely over-estimated the usefulness of the entry page as its defined today and too many of these requirements only make sense in the context of publications that are primarily meant to be distributed on the Web. For EPUB and audiobooks, these requirements only make things more complicated than they should be, at the cost of simplicity for both authors and UAs.

[HadrienGardeur]

where all our outputs are considered modules that inherit from a common publication manifest, with perhaps no required metadata and certainly no mention of structure. That way we can introduce different required metadata, other ways of harvesting metadata, different manifest/pep discovery models, etc. etc. on a per-model basis.

I mostly agree with your comment @mattgarrish but with a few tweaks:

• the current set of required metadata is really minimal, aside from dropping the requirement for url (also called the address) I don't think that it needs to be tweaked
• discoverability could also remain in the document that defines the manifest, but it wouldn't be a requirement and there wouldn't be any concept of an entry page
• a separate Web Publications specification would introduce the concept of an entry page, require url in the manifest and require discoverability from the entry page
• the Web Publication specification would also define a fallback when the default reading order is missing (use the entry page)
• the Lightweight Package specification would define a container (ZIP based on ISO spec) as well as well-known locations (manifest.jsonld and index.html)
• the Audiobook specification would define a profile of the Lightweight Package specification, with additional requirements for the manifest (reading order should only reference audio resources) and maybe additional recommendations or requirements for metadata

[mattgarrish]

The original consensus proposal included the fact that a user agent remains compatible with a packaged WPUB without further ado

And this would remain true for Web Publications that are packaged.

But what I see is that we're moving off into a different realm, where we have formats that predominantly live only in their packaged form, with only some desire to be able to make them also deployable as web publications. The path from exploded web content back to these specific formats is probably even smaller. This would encompass audiobooks, epubs and no doubt other forms as well.

The PEP makes it easier to explode these packaged formats onto the web, but it's not a critical part of them in their packaged form. That's why the formats should all inherit the same common manifest format, but whether they need the additional trappings to be web publications remains optional.

I would also say that a requirement of any packaged format is that it must retain the ability to be produced as a conformant web publication, not that it must simultaneously be a conformant web publication.

[geoffjukes]

@mattgarrish @iherman I think I am confusing the wpub and pwpub projects.

Is the scope of this project is to define how to package (i.e. contain in a single redistributable file) a collection of files that conform to the standard laid out in the wpub project?

[edit: w3c/wpub#400 Yep. I am]

[geoffjukes]

@iherman @HadrienGardeur Given the edit above, "If the final decision is that we MUST have an entry page" then from the perspective of an audiobook and epub Publisher, and an Audiobook aggregator/distrubutor, we would have no issue including a reference to a manifest inside the index.html to facilitate manifest discovery.

For publications that already have an index.html, the rel link would be injected. For publications that do not, we would include a bare PEP.

The name of the rel should be established so as to minimize the risk of a clash inside an existing index.html. Alternatively choose a well-known filename that is unlikely to clash with a file that is part of the publication (or make index.html a reserved filename in the wpub spec)

[mattgarrish]

Is the scope of this project is to define how to package (i.e. contain in a single redistributable file) a collection of files that conform to the standard laid out in the wpub project?

We have a bit of overlapping scopes here, to be honest, as we're also looking at whether audiobook = web pub in terms of what has to be packaged and how to discover it. That might be contributing to the confusion.

In general, though, the packaging defined here should work for web pub, audiobook and whatever else we define.

[geoffjukes]

@mattgarrish Thanks for the response.

This may be better suited to the wpub project....

We (Blackstone) have 3 high-level product types in this space.

• Audiobooks
• Print/ePub
• Hybrid of the two (an ePub with audio).

All 3 products have the potential for supplemental material which may need to be embedded (a PDF for example) or linked (a DVD ISO for example)

For us it would make sense to ensure the two can be sensibly co-mingled. We do not produce Manga, or ePubs with Video assets, but I would assume the complexity of such would match our 'hybrid' product.

• Audio-only products do not dictate how the files are displayed, only what order they should be played in.
• ePub-only products strictly dictate how the content should be displayed, and what order they should be consumed in.
• Hybrid products are more complicated. There is no guarantee that 1 audio file will match 1 html file, and often does not (and I am ignoring the media overlays for a moment).

I know I seem to be flip-flopping here, but it could be argued that a Packaged Web Publication that is missing a PEP, is Audio only, and therefore intended for consumption by an application running on a device, rather than inside a web browser, and as such the manifest should be consumable directly without requiring discovery.

The only utility I can see in a more 'complete' PEP for audio-only, would be if a publisher had specific display requirements for that title. However, I could counter-argue that it would then fall under the loose heading of the 'hybrid' product, and is not strictly 'audio only' (in essence, it's a 1-page ePub that is a playlist).

Apologies if some of this is naive. My background is operations/infrastructure, not application design. I am involved here because I developed the automation systems we use at Blackstone for asset ingestion and packaging.

[mattgarrish]

The name of the rel should be established so as to minimize the risk of a clash inside an existing index.html.

If we have to resort to specially named files, it would be good to only have one, and I'd lean to that being the manifest given its general utility.

It feels like we could just add a link relationship to identify the pep in whichever of the reading order or resource list it appears in.

[geoffjukes]

@mattgarrish For clarity, is the manifest.jsonld here intended to be the same one as the one defined by the wpub spec, but with non-local resources locations converted to local ones?

[mattgarrish]

For clarity, is the manifest.jsonld here intended to be the same one as the one defined by the wpub spec, but with non-local resources locations converted to local ones?

I haven't seen the issue of non-local resources raised yet, since this was borne out of audiobooks and their not living on the web first.

It would be a problem that needs solving to package web publications, but perhaps that doesn't get handled before the web packaging specification (i.e., there are limits on what can be packaged by this LPF container).

[geoffjukes]

@wareid was kind enough to provide me with clarity.

My opinion here would be that a well-defined manifest, that includes a sequence for media playback, would render a PEP redundant. I would therefore advocate for making the PEP optional but the manifest required in all publications.

From an application perspective, a missing PEP would indicate the publication was 'media only' and render an application-themed media-appropriate playlist based on the sequence.

The presence of a PEP would indicate the publication has specific display requirements, and should be rendered appropriately.

If the consensus is to make the PEP required, I would be OK with including a minimum PEP as described by @iherman

[mattgarrish]

a missing PEP would indicate the publication was 'media only'

That's true right now, but does it hold for future packaged publications? What if an EPUB 4 also doesn't require a primary entry page?

[geoffjukes]

@mattgarrish I’m not sure I follow. If EPUB4 is HTML based, it would have an index. If it is manifest based, it would not need a PEP.

The specifics of how an manifest-based EPUB4 might be interpreted and rendered is not relevant to this ticket :)

I’ll happily edit or remove the paragraph on what the absence of a PEP implies, as I feel the paragraph on what it’s presence implies is still appropriate.

[mattgarrish]

If EPUB4 is HTML based, it would have an index.

The PEP isn't (or hasn't been) about media type, but about deployment on the web. Whether an EPUB is destined for the Web is also a publisher choice, so at this time I can't see why it would be required -- except in the same scenario where we require publishers to produce unnecessary files.