3.3 Constituent Resources

[marcoscaceres]

Req. 16: The information regarding the constituent resources of a PWP must be easily discovered

The case for this requirement is extremely weak.

In particular, no clear justification is given for:

It is therefore necessary for the reading system to have an easy access to the list of constituent resources, and some of their characteristics like their media types or sizes.

It just gives some hand-wavy non-justification in the sentence before, about things being "prohibitive".

It almost feels like this requirement has been snuck in to justify the inclusion of some manifest file format for the packaging format (which seems pretty clear is trying to underpin/undermine? the requirements of this document).

[marcoscaceres]

The size of the components must be known in advance.

This also doesn't make sense... resources could be generated dynamically, on the device or on the server.

[marcoscaceres]

The reading systems needs to know if a publication contains/requires support for a specified media type (without processing the complete PWP).

Again, this is not how the Web works. We generally sniff stuff on the web. You don't need to know the type.

[sideshowbarker]

It almost feels like this requirement has been snuck in to justify the inclusion of some manifest file format for the packaging format (which seems pretty clear is trying to underpin/undermine? the requirements of this document).

Yeah it also seems to me like this requirement is here only to justify an assertion that’s been made previously elsewhere that a manifest is needed (which itself seems to be made based on the assumption that some sort of new packaging format is needed…).

I think the comment at https://github.com/w3c/webpayments-payment-apps-api/issues/33#issuecomment-246609553 is probably relevant here as well:

If you can avoid having an external JSON resource, please do so. … Web Manifest predates the extensible web and service workers, which is why we didn't have an API in the first place. If we were to do web manifest today, we would not do it as JSON - but as an API in Service Workers (hence w3c/manifest#472).

In other words, this is another section that would probably not be relevant if the document were (re)written with an assumption that Service Workers are use (just as it is already written with an assumption that HTML and CSS are used). Or more broadly, with an assumption that existing standard features of the Web platform are used—rather than the current core assumption is seems to have instead, that a new packaging format of some kind is absolutely necessary.

[lrosenthol]

@marcoscaceres @sideshowbarker While I agree with you that there are some requirements listed here that are indeed problematic in certain workflows (eg. sizes for dynamic content), there are other clear use cases where that could/would be useful - especially when considering an RS/UA for a device with limitations.

[GarthConboy]

This requirement is included to drive the inclusion of a manifest for processing purposes, especially in the case of offline use and also for eventual packaging.

So, snuck in through the front door.

[marcoscaceres]

@lrosenthol

especially when considering an RS/UA for a device with limitations.

With all due respect, this would be a premature optimization. This is no different from any webpage running on a crappy phone from a couple of years ago. Browsers are pretty good a running on limited devices.

@GarthConboy

This requirement is included to drive the inclusion of a manifest for processing purposes, especially in the case of offline use and also for eventual packaging.

Correct. It presupposes a solution. That's not good.

[GarthConboy]

@marcoscaceres

I can accept that, as long as the requirements for offline and optional/eventual packaging for addressing the first "P" in PWP is codified.

[marcoscaceres]

I can accept that, as long as the requirements for offline and optional/eventual packaging for addressing the first "P" in PWP is codified.

"You keep using that word..." :) But seriously, we could dump PWP and just replace it with web application. The packaging thing is leading people a stray.

[lrosenthol]

But these are not applications - at not as most users think about what an application is. (and many may not even contain any JavaScript).

And even if we were to consider that route - it doesn't help. because you can't transfer web apps via ad-hoc distribution mechanisms.

[iherman]

Let us come back to the original issue, and consider, for the sake of this discussion, a SW based set up. Let us also consider the scenario that I get the URI of a publication but I know that I will need it offline (because, say, I have to leave for the airport to flight to Lisbon in about half an hour:-). Let us also consider that the book consists of around 100 various resources or more (which is realistic; think of the proceedings of a large conference). How do I tell the system to put everything into the browser's 'offline storage' (forgive me if I do not have the right terminology) right away?

It strikes me that if (via some source, eg, a manifest) the list of constituent resources are known, then this step becomes straightforward. However, if not, then I do not see any other approach then let the browser+SW parse all the constituents, find the references that are essential part of the publication (let us say all the references with a relative URI) and use that information to download everything that is needed. It is of course doable but I would expect that to be quite prohibitive; by listing those resources in advance (and this information is known to the author of the publication, obviously) that step becomes fairly simple. (Note that using the ToC, that we already agreed is needed, is not enough; that does not provide all the information.)

I could see that listing the constituents for small publications is not really necessary, ie, this hint may be optional. (Eg, for a publication consisting of a single HTML page with a few CSS files and maybe images.) But for the aforementioned proceedings (with, say, over a 100 papers, each with several images, videos, companion JS file, you-name-it) the situation seems to be quite different.

[marcoscaceres]

But these are not applications - at not as most users think about what an application is. (and many may not even contain any JavaScript).

All HTML things are applications and are treated the same by browsers. How a user conceptualizes the difference is UX - but that's not relevant to the reality that they are applications, irrespective of them having any JavaScript.

And even if we were to consider that route - it doesn't help. because you can't transfer web apps via ad-hoc distribution mechanisms.

You might be right, but there might be other formats we can use: for example, a printed physical book. The point being, we should work within the constraints of the medium/Web Platform, and see where those constraints are unacceptable. There are some things that physical books will be better at then web books - but there are things where web books well be better.

[marcoscaceres]

Let us come back to the original issue, and consider, for the sake of this discussion, a SW based set up. Let us also consider the scenario that I get the URI of a publication but I know that I will need it offline (because, say, I have to leave for the airport to flight to Lisbon in about half an hour:-). Let us also consider that the book consists of around 100 various resources or more (which is realistic; think of the proceedings of a large conference). How do I tell the system to put everything into the browser's 'offline storage' (forgive me if I do not have the right terminology) right away?

That book/publication/application provides you the means to do that. The user does not tell the browser to do that. @iherman, I get a bit of a sense that you are viewing things as "declarative": where you want the browser to do smart things on behalf of the user. As browser vendors, we don't want to do that: we want web applications (in this case, this hypothetical conference proceedings application) to do that.

Browser vendors are ONLY interested in providing low-level primitives, privacy, and security. We want developers to do the "intelligently make these things available for me offline" for the user part.

Please see: https://extensiblewebmanifesto.org

It strikes me that if (via some source, eg, a manifest) the list of constituent resources are known, then this step becomes straightforward.

For who? Again, be careful not to fall into the trap of using passive voice here. And yes, it becomes straight forward for the developer of the book. They just use whatever format they want and put the things into the cache via a negotiation with the end-user (through a HTML-based interface, and not through the browser).

Again: the browser is not involved here (apart from rendering stuff and being told what to cache). This is a negotiation between the web application and its end user.

However, if not, then I do not see any other approach then let the browser+SW parse all the constituents, find the references that are essential part of the publication (let us say all the references with a relative URI) and use that information to download everything that is needed.

The browser doesn't care. It's up to the app to do clever things. The browser only cares that they app respects the users privacy, and security, and that it doesn't take up too much space or otherwise annoy the user.

It's critical to stop thinking that the browser is going to do clever things. It's not. Where you say, "let the browser+SW parse all the constituents," you need to replace "browser" with "web developer": and web developers are pretty clever.

As a web developer, I can think of lots of clever ways of overcoming this problem.

It is of course doable but I would expect that to be quite prohibitive; by listing those resources in advance (and this information is known to the author of the publication, obviously) that step becomes fairly simple. (Note that using the ToC, that we already agreed is needed, is not enough; that does not provide all the information.)

As above. The developer would pick their favorite language, tools, whatever, and make it work.

I could see that listing the constituents for small publications is not really necessary, ie, this hint may be optional. (Eg, for a publication consisting of a single HTML page with a few CSS files and maybe images.) But for the aforementioned proceedings (with, say, over a 100 papers, each with several images, videos, companion JS file, you-name-it) the situation seems to be quite different.

Not really. It scales the same. But it's up to the developer to negotiate with the user what they want to download. It would be silly to d/l all the things, because it could take up gigabytes of space. So the application might ask the end-user: "What proceedings do you want? Please select and I'll d/l them for you."

Look at how the Economist native app on iOS handles this problem:

1. It only stores the last 3 publications (print equiv).
2. For each publication, I can choose exactly which audio articles to download. Even more awesome, I can select per section or per article!

That allows me, as a user, full control over what I d/l and when. That's exactly what we want.

[marcoscaceres]

Note that using the ToC, that we already agreed is needed, is not enough; that does not provide all the information.

Note that, I, at least, am still not 100 sold on the ToC part... Safari Books (screenshot below), for instance, displays the TOC using HTML just fine. It's more of a "SHOULD" or "MAY" requirement.

[marcoscaceres]

Note also that this TOC is customized really nicely. It shows the publisher, some details about the book (e.g., publication date), etc.

[iherman]

@marcoscaceres just a small note (running to the TPAC site): I do not believe there is a disagreement between you and me. In the requirement document we do not at this moment separate what would be done by the browser directly and what would be done by an application built on top of it. Actually, we were using, tentatively, the concept of a PWP processor as a layer on top of a browser (plugin, extension, whatever) that would do things; it becomes then a next step to find out which part of the PWP processing steps is essentially done by the browser and which has to be done on top of it. And I actually do not expect the browser to understand things like the ToC or the list of resources out of the box, for me this is clearly for a layer on top of it.

Note that this is the way the Readium already works; it is an extension on top of a browser engine doing EPUB specific things. One mode of operation of Readium is an application on top of Chrome (more exactly, blink, because it also works in Vivaldi). This has always been the model I had in mind, personally...

[iherman]

Per safari books: I do not know how the internals of Safari work exactly but, AFAIK, they use EPUB extensively. They display the ToC nicely for a book, but that content comes from the (declarative) ToC content they retrieve from the individual EPUB instance. It is the same as what you showed using an iBook...

[marcoscaceres]

PWP processor as a layer on top of a browser (plugin, extension, whatever) that would do things;

We (browser vendors) don't want this either, please! These kinds of extensions just lead to more security issues. We should be working for the benefit of the web as a whole, not with the aim of making more specialized readers.

[lrosenthol]

We (browser vendors) don't want this either, please! These kinds of extensions just lead to more security issues. We should be working for the benefit of the web as a whole, not with the aim of making more specialized readers.

Now I am really confused, @marcoscaceres.

First you tell us that the browser should only do very simple things ("low-level primitives, privacy, and security") and that developers should provide the intelligence. And so we give you an example of how we are going to provide that intelligence (via a "PWP processor"), and you tell us that you don't want that either :(.

You can't have it both ways....

[marcoscaceres]

First you tell us that the browser should only do very simple things ("low-level primitives, privacy, and security") and that developers should provide the intelligence. And so we give you an example of how we are going to provide that intelligence (via a "PWP processor"), and you tell us that you don't want that either :(.

You can't have it both ways....

I can. What I want is Requirements that benefit the web as a whole. I want to be able to go back to Mozilla and say, "if we add these little bits, then developers can deliver book-like experiences to our users".

Or, even better: "turns out that the web platform right now can do 'books' without needing any new APIs or new features. We are awesome! Long live the Web!". :)

[dauwhe]

A few random thoughts as I read through this, mostly to provide some history on why the ebook tribe acts the way we do.

Paperwork

EPUB folks are used to doing a lot of "paperwork." Every content document in an EPUB is listed in the manifest (a list of every file), the spine (which order do they go in?), the nav doc (like the spine, but in html with links and link text), and most likely in the not-quite-dead NCX (which is like a nav doc only in a custom XML vocab with extra annoying features). So we start to think all this enumeration and duplication is essential.

Part of the reason we had to do this is that early ebook reading devices were insanely underpowered. Look through every EPUB that exists in the world, and you will probably find only a handful of HTML files that are larger than 300k. This is because old devices would choke on files that big. So we would split them up into tiny little pieces, and we still do, because reasons.

The reading systems also wanted to avoid as much HTML parsing as possible, and so wanted a declarative way to find out anything they wanted to know about the EPUB/OEB.

Do I like doing all this paperwork? No. I would love for a computer to sort it out for me. I'm told that computers can identify all the files in a folder, find out what resources they're using, figure out what kinds of files they are... So why am I doing all that work?

The less "manifest" the better.

To Code or Not To Code

To a first approximation, no one in the entire publishing industry can write JavaScript. Publishers usually outsource any work requiring coding to outside (meaning offshore) suppliers. Most ebook reading systems either explicitly forbid JS, or the support is so crippled, buggy, and undocumented that it's just not worth it. Ebook developers don't even get dev tools.

Simple declarative solutions are very appealing to us.

We lose money on every book we sell, but we make up for it with volume

I work for a rather large publisher. We publish roughly a thousand distinct titles a year, in multiple formats and editions. But that's nothing like doing a thousand different apps in a year. Every book is the same, except for the words. We don't want the user experience to vary drastically. We don't want anyone to have to learn how to use our book. In some ways it's very much like web pages, where we rely on the browser to provide a standard interface.

I don't want to write a thousand different service workers each year. I don't even want to write one. I want to point the browser at my book, and have it Do The Right Thing. Which is why I think the display modes idea being thrown around here is really important :)

And so my ultimate question is, what is the minimal amount of "stuff" beyond the raw HTML/CSS/JS that I need to provide to the browser? EPUB requires a lot of stuff. It would be lovely if "none" were the answer. But I don't know that yet.

[marcoscaceres]

And so my ultimate question is, what is the minimal amount of "stuff" beyond the raw HTML/CSS/JS that I need to provide to the browser? EPUB requires a lot of stuff. It would be lovely if "none" were the answer. But I don't know that yet.

Depends... for some tiers of content, it might be nothing. For example, Safari will trigger the ability to enable reader mode on content without you having to do anything:

For more sophisticated publications, it might be you need:

1. Manifest with display mode ("I'm a publication, yo!")
2. TLS (all new features need TLS - not negotiable because privacy)

For super duper, it might be you need the above and:

1. service Worker - offline

For money makers, the above:

1. push notifications ("New content! Wow! look here!")
2. Payments API, etc. ("You wanna buy the next chapter? of course you do!")
3. ... and so on... how fancy is up to you... imagine a travel guide that leverages Geolocation... so cool.

[marcoscaceres]

(fixed markdown fail above)

[marcoscaceres]

Just want to clarify:

Safari will trigger the ability to enable reader mode on content without you having to do anything

By "anything", I mean no JS. You still need to structure your content a bit to trigger the magic.

[lrosenthol]

I will also add that the group also wishes to extend beyond the bounds of professionally curated "books and publications" to the ad-hoc world of documents. Memos, letters, reports, etc. Things that are today authored by anyone (and everyone) who know even less than the publishers - but can figure out "Save as PDF" or "Save as EPUB" in their authoring tools.

On Wed, Sep 28, 2016 at 6:07 AM, Marcos Cáceres notifications@github.com wrote:

Just want to clarify:

Safari will trigger the ability to enable reader mode on content without you having to do anything

By "anything", I mean no JS. You still need to structure your content a bit to trigger the magic.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/w3c/dpub-pwp-ucr/issues/110#issuecomment-250072759, or mute the thread https://github.com/notifications/unsubscribe-auth/AE1vNZJV0KflwNa0G6bi4m2iCHpMufqzks5qufX-gaJpZM4J8XbJ .

[GarthConboy]

This issue too may well benefit from the bifurcation of Portable/Packaged from Web Publication that will be forthcoming.

[HadrienGardeur]

Thanks @marcoscaceres for your list of requirements, I think that this is very good to have.

I also tend to agree, that for a simple use case of reading in a browser, you don't really need anything more than HTML/CSS/JS.

That said, the goal of this group is to push the envelope a bit and figure out if we can cover more complex use cases.

Manifest with display mode ("I'm a publication, yo!")

Which display modes do you have in mind? Something similar to the current Web App Manifest?

Also, part of saying "I'm a publication, yo!" is to provide metadata about the publication. So at a very minimum this is metadata + display mode.

Among the other things that we've seen come up in the discussions, I could also list:

• table of contents (one or more)
• reading order (one or more)
• a list of resources that are part of the publication

TLS (all new features need TLS - not negotiable because privacy)

Fully agree, and the security model in general is quite important.

service Worker - offline

Sure and a number of questions come up too, for instance:

• should the browser offer a default reading mode that will rely on service worker to cache resources of the publication?
• or is this strictly the responsibility of the publication?
• do we rely on the manifest to identify resources that should be cached?

Payments API, etc. ("You wanna buy the next chapter? of course you do!")

So the content is potentially behind a payment gateway, with some sort of authentication requirement? If so, is there a standard way to handle authentication and authorization?

[marcoscaceres]

On Fri, Sep 30, 2016 at 11:10 PM, Hadrien Gardeur notifications@github.com wrote:

Which display modes do you have in mind? Something similar to the current Web App Manifest?

Correct. Though there might be other means to invoke a display mode as needed.

Also, part of saying "I'm a publication, yo!" is to provide metadata about the publication. So at a very minimum this is metadata + display mode.

Sure. To be clear, the two need not reside in the same place.

Sure and a number of questions come up too, for instance:

• should the browser offer a default reading mode that will rely on service worker to cache resources of the publication ?

• or is this strictly the responsibility of the publication ?

  Where possible, always the responsibility of the publication.

  • do we rely on the manifest to identify resources that should be cached ?

No. That would duplicate what the cache API already does. That list could just be bundled with the service worker (a common practice today).

Payments API, etc. ("You wanna buy the next chapter? of course you do!")

So the content is potentially behind a payment gateway, with some sort of authentication requirement?

No, it's up to the author.

If so, is there a standard way to handle authentication and authorization?

Soon. Google for Credential Management API. I think there were videos shown at last Google I/O. Pretty amazing stuff... at the very least, good food for thought there.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/w3c/dpub-pwp-ucr/issues/110#issuecomment-250740380, or mute the thread https://github.com/notifications/unsubscribe-auth/AA1HCpf3ttWEIGXDDFP4hlywGGwJPwd0ks5qvQpNgaJpZM4J8XbJ .

[HadrienGardeur]

@marcoscaceres

• should the browser offer a default reading mode that will rely on service worker to cache resources of the publication ?
• or is this strictly the responsibility of the publication ?

Where possible, always the responsibility of the publication.

But in this case, it puts the burden of providing offline capability entirely on the author/publisher.

To quote @dauwhe:

I don't want to write a thousand different service workers each year. I don't even want to write one. I want to point the browser at my book, and have it Do The Right Thing.

If the browser/browser engine doesn't want to handle that part, then we should at least unify our efforts and create a shared JS script among publications (all pointing to the same URI) that would handle all things Service Worker related.

• do we rely on the manifest to identify resources that should be cached ?

No. That would duplicate what the cache API already does. That list could just be bundled with the service worker (a common practice today).

But as far as I can tell, there's nothing standard or even true best practices for that. Most examples that I could find, simply use an array of strings and that's it. Am I missing something?

Also what's the harm about extracting that info from a manifest instead of embedding them with the service worker? I don't see the benefit.

Our shared script could simply look for a manifest on a page, extract the list of resources, and handle caching and priorities (cache first, network first) on behalf of the author/publisher.

[marcoscaceres]

On 1 Oct. 2016, at 12:25 am, Hadrien Gardeur notifications@github.com wrote:

@marcoscaceres

should the browser offer a default reading mode that will rely on service worker to cache resources of the publication ? or is this strictly the responsibility of the publication ? Where possible, always the responsibility of the publication.

But in this case, it puts the burden of providing offline capability entirely on the author/publisher.

Yes. It puts the publisher in control. That's not a "burden": that's a prevailed responsibility. It also allows people to create libraries that can make the process easier: see Google's SW toolbox or Mozilla's SW cookbook.

To quote @dauwhe:

I don't want to write a thousand different service workers each year. I don't even want to write one. I want to point the browser at my book, and have it Do The Right Thing. Which is why I think the display modes idea being thrown around here is really important :)

If the browser/browser engine doesn't want to handle that part, then we should at least unify our efforts and create a shared JS script among publications (all pointing to the same URI) that would handle all things Service Worker related.

Now you are talking! Yes should absolutely do this.

do we rely on the manifest to identify resources that should be cached ? No. That would duplicate what the cache API already does. That list could just be bundled with the service worker (a common practice today).

But as far as I can tell, there's nothing standard or even true best practices for that. Most examples that I could find, simply use an array of strings and that's it. Am I missing something?

No. that's what is so beautiful about it: It's super simple.

Also what's the harm about extracting that info from a manifest instead of embedding them with the service worker? I don't see the benefit.

Manifest things are instructions related to installation and application startup.

Our shared script could simply look for a manifest on a page, extract the list of resources, and handle caching and priorities (cache first, network first) on behalf of the author/publisher.

Browsers don't want to do that for authors/publishers: they want to treat everyone equally and give authors and publishers full control.

That way, anyone can create a JSON file or whatever format they want, and with some js, they can make their publications behave however they wish.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[HadrienGardeur]

@marcoscaceres good, this feels like we're heading somewhere.

Manifest things are instructions related to installation and application startup.

That's the definition of "manifest" for Web Applications maybe, but not the only one. In EPUB the "manifest" is the list of resources used in the publication, and in AppCache the manifest also provides a list of resources to cache (among other things).

Now here's what I'd like to propose:

• a manifest format for publications that handles metadata, display mode and a list of resources to cache: https://github.com/HadrienGardeur/webpub-manifest
• a shared JS script among Web Publications that will get the list of resources from the manifest, cache them and handle network requests using Service Workers

At this point, it's a little unclear what we should or shouldn't do regarding reading order, but it's worth exploring.

With these two pieces, we should have something that works for browsers.

In addition, for dedicated reading systems and for the publishing industry (where packages are sold and borrowed) we'll also define a simple package format that can contain both the manifest and associated resources.

[marcoscaceres]

On 1 Oct. 2016, at 2:38 am, Hadrien Gardeur notifications@github.com wrote:

@marcoscaceres good, this feels like we're heading somewhere.

Manifest things are instructions related to installation and application startup.

That's the definition of "manifest" for Web Applications maybe, but not the only one. In EPUB the "manifest" is the list of resources used in the publication, and in AppCache the manifest also provides a list of resources to cache (among other things).

AppCache illustrates my point: browsers royally screwed that up - and it's why we don't want to do declarative solutions. Hence the extensible web manifesto, service workers, etc.

We don't even want to do HTML Elements: we rather people define their own using Custom Elements.

Now here's what I'd like to propose:

a manifest format for publications that handles metadata, display mode and a list of resources to cache: https://github.com/dauwhe/epub31-bff

Sure, but we don't need a "web standard" (I.e., a standard that dictate browser behavior) for that. We can create some industry standard, but then just process it with JS. The browser won't give it any special treatment tho.

a shared JS script among Web Publications that will get the list of resources from the manifest, cache them and handle network requests using Service Workers

That be great. Like a jQuery, but for publications.

At this point, it's a little unclear what we should or shouldn't do regarding reading order, but it's worth exploring.

Absolutely. I think we can do something there in the browser. That feels like it needs an API because it affects (potentially) the browsers UI.

With these two pieces, we should have something that works for browsers.

I'd say you only need the second, but sure.

In addition, for dedicated reading systems and for the publishing industry (where packages are sold and borrowed) we'll also define a simple package format that can contain both the manifest and associated resources.

Sounds good to me!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[lrosenthol]

@HadrienGardeur - thanks for putting up the strawman of the two proposals, unfortunately neither one works as stated

a manifest format for publications that handles metadata, display mode and a list of resources to cache

I believe there is full agreement that we need a way to handle metadata (for some definition thereof) and a list of resources used in the publication. Whether these go in a "manifest" or even in the same place is quite up for debate.

I don't even know what a 'display mode' is - and why that's something declarative.

a shared JS script among Web Publications that will get the list of resources from the manifest, cache them and handle network requests using Service Workers

If someone wants to write that - great! However, I wouldn't consider that the only (or even best) direction for publication handling. As previously mentioned, there are considerations in the use of Service Workers in a variety of valid use cases posed here - so they may turn out to not be the panacea that some believe. (hopefully some of the problems could be fixed with adaptations to the SW spec..)

[marcoscaceres]

On 1 Oct. 2016, at 2:55 am, Leonard Rosenthol notifications@github.com wrote:

@HadrienGardeur - thanks for putting up the strawman of the two proposals, unfortunately neither one works as stated

a manifest format for publications that handles metadata, display mode and a list of resources to cache

I believe there is full agreement that we need a way to handle metadata (for some definition thereof) and a list of resources used in the publication. Whether these go in a "manifest" or even in the same place is quite up for debate.

I don't even know what a 'display mode' is - and why that's something declarative.

a shared JS script among Web Publications that will get the list of resources from the manifest, cache them and handle network requests using Service Workers

If someone wants to write that - great! However, I wouldn't consider that the only (or even best) direction for publication handling. As previously mentioned, there are considerations in the use of Service Workers in a variety of valid use cases posed here - so they may turn out to not be the panacea that some believe. (hopefully some of the problems could be fixed with adaptations to the SW spec..)

I agree. Would be great to fix any limitations we find.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[HadrienGardeur]

@marcoscaceres

AppCache illustrates my point: browsers royally screwed that up - and it's why we don't want to do declarative solutions. Hence the extensible web manifesto, service workers, etc.

Yes, I get that part. Service workers are very low level and powerful, which is quite different from a declarative solution.

Sure, but we don't need a "web standard" (I.e., a standard that dictate browser behavior) for that. We can create some industry standard, but then just process it with JS. The browser won't give it any special treatment tho.

Sounds reasonable to me, we can't dictate what browsers will and will not implement anyway. If this is good enough and seen as useful, then browsers could still implement it down the road (in the reader mode for example on Firefox).

Absolutely. I think we can do something there in the browser. That feels like it needs an API because it affects (potentially) the browsers UI.

That's a very interesting idea. At a browser level this could simply be an API, and at a publication level we would leverage that API.

Do you have something more specific in mind on the browser side?

[HadrienGardeur]

@lrosenthol

thanks for putting up the strawman of the two proposals, unfortunately neither one works as stated

As you've said yourself, these are "proposals" so I don't see what you mean by "neither one works as stated". At this point everything is still undecided, but IMO proposals are helpful to such discussions.

I don't even know what a 'display mode' is - and why that's something declarative.

Take a look at the Web App Manifest specification: https://www.w3.org/TR/appmanifest/#the-display-mode-media-feature

If someone wants to write that - great! However, I wouldn't consider that the only (or even best) direction for publication handling.

It's the Web, even if we adopt both a manifest and a script, you're still free to do whatever you want and roll out your own thing if you prefer.

[iherman]

Just a terminological side-remark, to avoid future misunderstandings...

Now here's what I'd like to propose:

a manifest format for publications that handles metadata, display mode and a list of resources to cache: https://github.com/dauwhe/epub31-bff

Sure, but we don't need a "web standard" (I.e., a standard that dictate browser behavior) for that. We can create some industry standard, but then just process it with JS. The browser won't give it any special treatment tho.

Not all W3C standards are directly reflected in a browser behavior. Having a W3C standard (if necessary!) for a manifest format or, possibly and more exactly, extending a core manifest with data that is relevant for publishing, and used by that separate JS library only (ie, ignored by the core browser), is perfectly fine. We should find the maximal common grounds with browsers that they think belongs to the "core", and anything else that happens in that separate JS should not be in contradiction with the browser requirements (e.g., as far as I know, an extension cannot change the browser chrome) but, I believe, the publishing world needs more than a vague "industry standard"...

Again, I believe this is simply a terminological question.

[HadrienGardeur]

@marcoscaceres @iherman also, I'd like to have an easy way to detect if a browser has direct support for Web Publications or not.

This way could we could test if the browser has support for Web Publications, and only provide additional features using JS if it doesn't.

[marcoscaceres]

On 2 Oct. 2016, at 12:30 pm, Hadrien Gardeur notifications@github.com wrote:

@marcoscaceres @iherman also, I'd like to have an easy way to detect if a browser has direct support for Web Publications or not.

This way could we could test if the browser provides supports for Web Publications, and only provide additional features using JS if it doesn't.

Sure, but you need to see a WP as a progressive enhancement - and we need to be super clear as to what aspects would change based on support (in trying to say it might not affect JS at all): for instance, the display-mode media feature can be detected as being supported by the browser, allowing authors control over the publication's UI using only CSS.

Alternatively, when a publication is "installed", a particular service worker can be run (via the manifest). Alternatively, the "start url" can be used to control which resources are loaded. Plenty of options :)

At the end of the day, these publications should work on both supporting and non-supporting user agents - and it should be up to users if they trigger the web publication UI (if a special one is provided by the browser).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[iherman]

At the end of the day, these publications should work on both supporting and non-supporting user agents

I think that is an important design principle we should indeed adopt. It may affect some of our decisions, and we will have to keep it mind.

and it should be up to users if they trigger the web publication UI (if a special one is provided by the browser).

... or provided by a special JS library running, say, in an extension

[iherman]

(We may have to add this as an explicit UCR item, b.t.w.)

[lrosenthol]

While I agree with the concept - the problem is how we will define "work". The minimum viable product (MVP) is going to be very difficult to define, IMO...

On Sun, Oct 2, 2016 at 3:10 AM, Ivan Herman notifications@github.com wrote:

At the end of the day, these publications should work on both supporting and non-supporting user agents

I think that is an important design principle we should indeed adopt. It may affect some of our decisions, and we will have to keep it mind.

and it should be up to users if they trigger the web publication UI (if a special one is provided by the browser).

... or provided by a special JS library running, say, in an extension

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/w3c/dpub-pwp-ucr/issues/110#issuecomment-250957637, or mute the thread https://github.com/notifications/unsubscribe-auth/AE1vNVEucPQ4FERb2TvgvkpFNanwcgrPks5qv1jwgaJpZM4J8XbJ .

[HadrienGardeur]

I'm going to start a new document with a list of what the Web Publication JS could handle.

[HadrienGardeur]

Here's a first draft: https://github.com/HadrienGardeur/webpub-manifest/wiki/Web-Publication-JS

[marcoscaceres]

While I agree with the concept - the problem is how we will define "work".

The publication would be readable in a legacy or non-supporting user agent, meaning that a user could still read the text, navigate from one chapter to another, etc. What ever features stem from the requirements, should enhance the publication.

Requirement: Features of WP's MUST be designed in such a way that they work as progressive enhancements to the Web Platform - allowing these publications to degrade gracefully on user agents with limited, or no, support for WP features.

The use case: < Name > is traveling abroad and needs to access her publications from an out-of-date web browser in a hotel lobby. The browser has limited support for new web features, but < Name > is still able to read and navigate around the publication to get the information she needs.

< Name >'s owns a phone and tablet that run different OSs. The browser on one of the devices is updated regularly with new features (including new WP features), while the other is updated once a year. < Name > regularly uses both devices to access his publications - and despite the difference in supported features, he still has a productive reading experience.

[iherman]

I have tried to go a little bit deeper into how Service Workers work and how they could be used/adapted to publications needs, too. I have therefore looked at the latest spec and two "tutorial"-like pages on MDN and on Google. I must admit that reading the spec itself turned out to be daunting, and I essentially failed understand it; I am simply not familiar with all the cross references to some of the latest API specs, and the SW spec itself does not really give the big picture. The two tutorials were helpful (to me).

I think that the requirements of Dave, and the more explicit descriptions of Hadrien, lead to a model whereby:

• I have my Web Publication somewhere on the Web, which contains various declarative data on whatever is necessary to have the Web Publication properly rendered
• I have a generic, "standard" JS file, ie, a JS application (a "shared JS script among Web Publications" per @HadrienGardeur), to do everything that is necessary for publishing and that browsers would not do by default.

The questionis is whether that shared JS file must be physically part of the Web Publication (ie, essentially, replicated for each and every Web Publication ever produced), or whether the JS file is not necessarily on the Web site of the Web Publication itself (ie, could be shared among WP-s if it is some sort of a standard). I believe forcing the publishers into the former setup would be very difficult. "Simple declarative solutions are very appealing to us" said @dauwhe…

My reading of the documents listed above seems to indicate that if that shared JS application uses Service Workers, i.e., it is up to that application to register and initialize a service worker for a specific book, then… well, I am not sure:-(. Although, per spec it is possible to assign a different "scope" to a service worker than the default scope generated by the service worker file's URL, I have hit further problems:

• How would that shared JS's event listener for install populate the cache with the right files? In other words, does that application have an access (as part of the event structure, for example) to the "root" URL of the Web Publication? If so, it could use the information made available declaratively (and what this whole github issue is all about), otherwise I am not sure… But I am actually confident that this should be possible.
• However, I also found a remark in the MDN saying: "The service worker being pointed to is on a different origin to that of your app. This is also not allowed." There are terminology differences ("app", I believe, corresponds to our Web Publication) but if my understanding is correct the service worker application, i.e., the "shared JS" must reside on the Web Publication's site. In other words, it should be delivered by publishers alongside the rest of the content.

The situation may become different if the offline/online handling is done within the browser, ie, the shared JS takes the online/offline divide as, functionally, irrelevant, because it is handled by the browser. This would be an ideal solution but then raises another issue: is it possibly to, sort of, polyfill such a facility on top of a browser, waiting for those functionalities to become part of a browser? I must admit I simply do not know but, if the answer is "no" on that question then… I am not sure where we should be heading.

Maybe (I hope!) I am worried for nothing (in which case sorry for the noise). I know that Readium has experimented with Service Workers on top of Chrome, albeit in a separate Chrome "application". @danielweck, could you share your experience on this?

(Sorry for the long text...)

Cc: @HadrienGardeur @marcoscaceres @dauwhe @danielweck

[HadrienGardeur]

@iherman

I've been down that same road these last few days and I've explicitly built a live demo for that purpose: https://hadriengardeur.github.io/webpub-manifest/examples/MobyDick/

It essentially does two things for now:

• cache the resources for offline viewing
• use a Service Worker to create and serve a Web App Manifest

If you test it in Chrome on Android, the second time that you open the publication it'll pop up an install banner to install the publication as an app (you can also do the same thing at any time from the options menu). Based on my experience, the Web App Manifest should only appear on the "homepage" of the publication, otherwise you end up with the install banner popping up everywhere, including in the middle of the publication after an install.

To answer your question @iherman, based on my current understanding:

• the shared JS library could be hosted anywhere (for example on a CDN operated by the W3C)
• but the ServiceWorker (a separate JS) itself has to be on the domain, and its location defines its scope
• I haven't figured out yet a good way to pass the URL of the Web Publication Manifest to the Service Worker

That said, I don't think that one ServiceWorker per publication is necessary, just one per domain with that approach. This means that to make my domain compatible with Web Publications' offline mode, I would have to also include the Service Worker JS.

I haven't explored or read much about cross origin service workers, maybe they would come into play here but @marcoscaceres knows a lot more about this.

I've also started a separate document listing what the shared JS could do at https://github.com/HadrienGardeur/webpub-manifest/wiki/Web-Publication-JS

[BillKasdorf]

Thank you so much for this thorough overview, Ivan!

One comment, in a use case vein, re:

whether the JS file is not necessarily on the Web site of the Web Publication itself (ie, could be shared among WP-s if it is some sort of a standard)

I think that "shared JS" would not need to be, and most often would not be, a sort of "standard." Instead, I think it would be very common for a publisher to say "this shared JS applies to all my publications" or "this shared JS applies to this set of publications."

Bill Kasdorf

VP and Principal Consultant | Apex CoVantage

p:

734-904-6252 m: 734-904-6252

ISNI: http://isni.org/isni/0000000116490786 ORCiD: https://orcid.org/0000-0001-7002-4786https://orcid.org/0000-0001-7002-4786?lang=en

From: Ivan Herman [mailto:notifications@github.com] Sent: Wednesday, October 05, 2016 9:34 AM To: w3c/dpub-pwp-ucr Subject: Re: [w3c/dpub-pwp-ucr] 3.3 Constituent Resources (#110)

I have tried to go a little bit deeper into how Service Workers work and how they could be used/adapted to publications needs, too. I have therefore looked at the latest spechttps://w3c.github.io/ServiceWorker/ and two "tutorial"-like pages on MDNhttps://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API/Using_Service_Workers and on Googlehttps://developers.google.com/web/fundamentals/getting-started/primers/service-workers. I must admit that reading the spec itself turned out to be daunting, and I essentially failed understand it; I am simply not familiar with all the cross references to some of the latest API specs, and the SW spec itself does not really give the big picture. The two tutorials were helpful (to me).

I think that the requirements of Davehttps://github.com/w3c/dpub-pwp-ucr/issues/110#issuecomment-249888579, and the more explicit descriptions of Hadrienhttps://github.com/w3c/dpub-pwp-ucr/issues/110#issuecomment-250757809, lead to a model whereby:

• I have my Web Publication somewhere on the Web, which contains various declarative data on whatever is necessary to have the Web Publication properly rendered
• I have a generic, "standard" JS file, ie, a JS application (a "shared JS script among Web Publications" per @HadrienGardeurhttps://github.com/HadrienGardeur), to do everything that is necessary for publishing and that browsers would not do by default.

The questionis is whether that shared JS file must be physically part of the Web Publication (ie, essentially, replicated for each and every Web Publication ever produced), or whether the JS file is not necessarily on the Web site of the Web Publication itself (ie, could be shared among WP-s if it is some sort of a standard). I believe forcing the publishers into the former setup would be very difficult. "Simple declarative solutions are very appealing to us" said @dauwhehttps://github.com/dauwhe…

My reading of the documents listed above seems to indicate that if that shared JS application uses Service Workers, i.e., it is up to that application to register and initialize a service worker for a specific book, then… well, I am not sure:-(. Although, per spechttps://w3c.github.io/ServiceWorker/#navigator-service-worker-register it is possible to assign a different "scope" to a service worker than the default scope generated by the service worker file's URL, I have hit further problems:

• How would that shared JS's event listener for install populate the cache with the right files? In other words, does that application have an access (as part of the event structure, for example) to the "root" URL of the Web Publication? If so, it could use the information made available declaratively (and what this whole github issue is all about), otherwise I am not sure… But I am actually confident that this should be possible.
• However, I also found a remark in the MDNhttps://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API/Using_Service_Workers#Why_is_my_service_worker_failing_to_register saying: "The service worker being pointed to is on a different origin to that of your app. This is also not allowed." There are terminology differences ("app", I believe, corresponds to our Web Publication) but if my understanding is correct the service worker application, i.e., the "shared JS" must reside on the Web Publication's site. In other words, it should be delivered by publishers alongside the rest of the content.

The situation may become different if the offline/online handling is done within the browser, ie, the shared JS takes the online/offline divide as, functionally, irrelevant, because it is handled by the browser. This would be an ideal solution but then raises another issue: is it possibly to, sort of, polyfill such a facility on top of a browser, waiting for those functionalities to become part of a browser? I must admit I simply do not know but, if the answer is "no" on that question then… I am not sure where we should be heading.

Maybe (I hope!) I am worried for nothing (in which case sorry for the noise). I know that Readium has experimented with Service Workers on top of Chrome, albeit in a separate Chrome "application". @danielweckhttps://github.com/danielweck, could you share your experience on this?

(Sorry for the long text...)

Cc: @HadrienGardeurhttps://github.com/HadrienGardeur @marcoscacereshttps://github.com/marcoscaceres @dauwhehttps://github.com/dauwhe @danielweckhttps://github.com/danielweck

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/w3c/dpub-pwp-ucr/issues/110#issuecomment-251675698, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AIYxNd6KmhDp3--_Wtb2NbBMCE-DURr1ks5qw6dRgaJpZM4J8XbJ.

[dauwhe]

In my experiments, I have a single SW for multiple books that reads each book's (extended) manifest to know what files to cache for offline.

[HadrienGardeur]

In my experiments, I have a single SW for multiple books that reads each book's (extended) manifest to know what files to cache for offline.

But that SW is on the same domain, and its location (https://dauwhe.github.io/epub-zero/acme-publishing/sw.js) means that its scope can cover the publications that you use it for (https://dauwhe.github.io/epub-zero/acme-publishing/MobyDick/index.html).

[mac2net]

The cache lives for 10 minutes and then puff? There's no UI to know if a set of related documents has actually been cached? What if I wanted to take @iherman last presentation including video on the plane for a flight from Europe to SF? How would I know the 10mb presentation was downloaded and if the cache settings were for a long enough duration? What if I wanted to annotate the presentation on the plane? If the cache expires before I get to my hotel do I lose the presentation and the annotations? Or should I print each page to PDF an annotate it in Preview?

Given that the cache duration and therefore the actual offline persistence is defined by the publisher, I don't understand the point of an ephemeral offline standard except as quasi-DRM.