Packaging for WP - Requirements and Success Criteria

[wareid]

The packaging requirements of WP and Audiobooks are still unclear. To better help us steer our decision, we need to know what we need to do to be "successful". Let's define what this means to us.

From the wiki:

• Simple - the option we choose should be straightforward and understandable to the industry
• Distributable
• Streamable

Comment on this issue in the next two weeks, issue will be closed on January 18

[dauwhe]

I want to be able to create the format in question using a text editor and a command-line, without having to install new programming libraries or languages.

[llemeurfr]

As a producer, I want to be able to exchange in-progress Publications between different individuals and/or different organizations.

As a publisher or conversion house, I want to provide Publications to distribution platforms or sales channels.

As a user, I want to be able to download a file containing a Publication from a distribution platform.

[llemeurfr]

@dauwhe even for creating a zip file you need a zip utility. Some OSes don't have it pre-loaded.

[ghost]

As a Reading System vendor, I hope

1. the package file is not too big or could be split considering downloading performance.
2. have some definition about playback order of audio files.
3. have some definition similar as TOC that we could show user where they are.

[GarthConboy]

I like #2 and #3 in @rakutenjeff 's above. I would not want to add split-ability as that would add complexity, and then we'd need to add some sort of higher level manifest or a package for the parts. :-)

I'm on board with OCF-Lite as proposed for audiobooks in our current WP-influenced world. That could also, likely, be used for a (near) future comics effort. I would not worry about packaging yet of general purpose WP's -- as we have yet to really find that constituency and we expect Web Packaging to be impactful if/when that constituency is identified.

[iherman]

@rakutenjeff, while I agree with your points (2) and (3), I think the present issue is only on the packaging (file) format itself. (2) and (3) are covered (or should be covered) by the WP Manifest, and that is not really in question. The question is, should we use zip or not zip, ocf or not ocf, an MPEG variant or not, etc.

(1) ("the package file is not too big or could be split") is very relevant, though.

[HadrienGardeur]

Streamable The package file is not too big or could be split considering downloading performance

This is not strictly tied to the packaging format itself, the transfer protocol plays a huge role in both of these points.

Let's take ZIP/OCF:

• it's streamable but not optimized for streaming
• you can split download using HTTP byte ranges if enabled on the remote HTTP server

have some definition about playback order of audio files have some definition similar as TOC that we could show user where they are

IMO these two points are not tied to the package but to the manifest instead. As long as we require the package to include a manifest, they should be covered.

[llemeurfr]

Streamable

a/ The term streamable is not precise enough to be used in success criteria. Many would advocate that streaming media is associated with a streaming protocol like RTSP or RTP. It will be much clearer to speak about progressive download.

b/ I disagree that progressive download compatibility is needed for the container format we design here. Progressive download, like streaming, means that a client end-user can use a media player to start listening to digital audio content before the entire file has been transmitted. We don't need this requirement for our container because Web Publications are already made for direct consumption of the Web and we don't want to confuse the two types of delivery. Plus, asking our container to be progressive download-friendly would impose the manifest to be the first file in the zip. And it's better if we can avoid requiring a file order in the container.

[llemeurfr]

the package file is not too big or could be split considering downloading performance.

We can't impose a size limit. Ten years ago a 1 Gb file would have been considered crazy. Today direct download of HD movies is kids routine. Nevertheless, as Hadrien said, a well written download manager software (which will use HTTP byte range as much as possible) is useful for huge files.

[wareid]

Success Criteria (updated), based on the comments here:

• Simple - the option we choose should be straightforward and understandable to our audience, requiring standard tools (i.e. a text editor, command line), not complicated libraries or proprietary software
• Distributable - the option we choose should allow for our stakeholders (producers, publishers, distributors, users) to exchange/distribute the audiobook as needed throughout the process of production, sale, and consumption
• Progressive Download-"friendly" - the option we choose should not interfere with an implementer's efforts to introduce progressive download functionality, and in fact should encourage it considering the average size of this content type

[iherman]

This issue was discussed in a meeting.

• RESOLVED: PWG will adopt a light-weight version of Zip, based on <a href="https://www.iso.org/standard/60101.html,">https://www.iso.org/standard/60101.html,</a> with some restrictions and additions for WP
  View the transcript

  packaging
  Tzviya Siegman: https://github.com/w3c/wpub/issues/390
  Tzviya Siegman: The next item is packaging. Wendy did a nice summary — but since the last meeting we’ve defined success criteria. Here is the issue with success criteria. Those who attended the AB publishing meeting, we got to see how things stand with packaging.
  … we got info about the HEIF format — and Dave did a presentation on OCF-lite. We heard from the Chrome team about the current incubating format that google proposed. All that said, we have the success criteria put out.
  … We are all but agreed agreed on OCF-Lite — but that’s a bad term for it because it’s possibly misleading…
  Dave Cramer: I have a question about what we are specifying — it came up in a packaging thread. There is an ISO standard that basically is very close to being a subset of ZIP that OCF uses. It mentions restricting the types of compression — it has an appendix on restricting filenames…
  … it’s also oddly, for an ISO standard, you can get the PDF without paying a bunch of Swiss franks. I think it would be good if we didn’t copy a whole bunch of text from OCF, but borrowed things that work like folder locations…
  Tzviya Siegman: Dave — so you’re in favor of using a restricted format of ZIP instead of rewriting a spec?
  Dave Cramer: I’m worried that we have an OCF spec, and we could be normatively adding more items. If we’re saying we want to use ZIP the ISO zip has some nice items in there.
  Dave Cramer: https://www.iso.org/standard/60101.html
  Dave Cramer: which is ISO/IEC 21320-1:2015
  Garth Conboy: I think we could probably make — I don’t disagree with Dave — I think we could make a resolution to do something like OCF-Lite. Whether built up from the ISO Zip or down from OCF — I think we’re in the same neighborhood.
  … One decision is about creating compatibility with meta-inf. We’re in the neighborhood that we want to use something zip based with minimal restrictions. We can probably resolve that now and there is some work to do if it’s build up or build down.
  Nick Ruffilo: .. but we could resolve if it will work for audiobooks, but maybe other profiles for different uses of WP…
  Ivan Herman: +1 to Garth
  Laurent Le Meur: I’m not against the ISO standard as a base, but there are several items to discuss. the ISO 21320 spec only prohibits some characters — the OCF original prohibits other characters — so it’s not 100% compatible. In the ISO standard, there is a table that says that digital signatures is not allowed…
  … but is not described in the OCF. It won’t be enough for a packaging mechanism, we need to specify what the name of the manifest file is — so we have some details of packaging that needs to be described that won’t be in the ISO spec. This is why in the draft I made before, I had chosen to copy the OCF spec and remove things.
  … even if I make reference to the ISO spec, most of the language that is there will still be there.
  Ivan Herman: First of all — accepting what Laurent had said — it will put our document/work on a more solid basis if it was based on ISO spec. It should be a starting position. Whether we want to be compatible with OCF on characters — we can go through that…
  … The starting position should be the ISO — it will help in talking with TAG and others. The original reason I was on the queue is that our position goes — whatever we define here is not to define specifically and exclusively for audiobooks.
  … whatever we do, we should try to be as generic as possible. If tomorrow, another profile comes up that is similar to audiobooks — manga came up — the starting position should be that the document Laurent creates is a lightweight format for publishing in general, which happens to be used by audiobooks.
  Nick Ruffilo: +1
  Tzviya Siegman: +1 to ISO
  Laurent Le Meur: +1 to a generic packaging format for WP
  Bill Kasdorf: +1 to Ivan
  Wolfgang Schindler: +1 to Ivan — generic packaging format for WP
  Garth Conboy: I would agree with that. I didn’t mean to imply it was only audiobooks, but that there could be more formats in the future. If we have to dream up another name, we can. We can get to what Laurent drafted by building up from the ISO spec…
  … we clearly will have to have a well-known place for the manifest. I hope that we can say we are building a lightweight zip format based upon the ISO spec with additional restrictions and rules for WP
  Laurent Le Meur: If agreed, I can modify the draft to reflect that…
  Proposed resolution: PWG will adopt a light-weight version of Zip, based on https://www.iso.org/standard/60101.html, with some restrictions and additions for WP (Tzviya Siegman)
  Tzviya Siegman: I think we should talk about if we need a stand-alone document
  Ivan Herman: There are things we need to describe somewhere, so we need to have a document for this. Where you find the manifest, etc. It may only be one page but it must be there.
  Ivan Herman: +1 to the proposal
  Garth Conboy: +1 to proposal (too)
  Wolfgang Schindler: +1 to the proposal
  Dave Cramer: Just thinking about Tzviya’s comment about what we need for a document. OCF is 2 specs — there is an OCF abstract container, and then the zip container. The later includes all the ISO stuff, the former is ‘everything has to be the same folder and you have to put a container.xml here’
  … not sure how we move the abstract container stuff into our spec…
  Laurent Le Meur: +1 to PWP…
  Laurent Le Meur: PPWP for Pragmatic PWP
  Luc Audrain: +1
  Tim Cole: +1
  Tzviya Siegman: we do have a document — that is a shell — PWP — packaged web publications — that way we get around writing a package document. We have the information associated with packaging — anything else — but Matt might get annoyed if we change the short title.
  Ivan Herman: +1 to dauwhe
  Dave Cramer: My concern about putting this under PWP — will this make it feel like we’re rejecting all other attempts at the web for doing packaging?
  Garth Conboy: When I typed PWP I almost typed “profile 1” but I am sympathetic to Dave’s comment. We can be clear that “this is what we’re doing now, we hope to use it in the future, but it’s not a hard decision that there won’t be another format in the future”
  Ivan Herman: That’s why there was an email where we set up the limits and milestones for the upcoming year as a “lightweight packaging format”. There might be a heavyweight coming in the future, but I agree with Dave. We should be careful. I’m cautious using PWP.
  … We have talked too much about it being all the solutions to the miseries of the world, so coming up with this will lead to ugly discussions.
  Tzviya Siegman: +1
  Laurent Le Meur: +1 to the vote
  Tzviya Siegman: the document we’re talking about is very short — Light Weight Packaging Format.
  Joshua Pyle: +1
  Bill Kasdorf: +1
  Wolfgang Schindler: +1
  Mateus Teixeira: +1 to proposal
  Nick Ruffilo: +1
  George Kerscher: +1
  Gregorio Pellegrino: +1
  Resolution #3: PWG will adopt a light-weight version of Zip, based on https://www.iso.org/standard/60101.html, with some restrictions and additions for WP
  Tzviya Siegman: Moving on — what are our next steps?
  Laurent Le Meur: I have to modify the draft, remove everything about the characters. Replace that will reference to the ISO zip format. Rename what is PWP inside this document — we can use something like LWPF until we choose a final name. I can keep it as ocf-lite but we can rename it to something else when we chose the name..
  … no one will see PWP, but next week would be good to chose a final name.
  Ivan Herman: Great! A question we will have to decide is whether this document is on a Rec track or not. The ISO part makes this much easier. That means the only thing we have to test as part of the CR procedures are the ones we add — not the packaging/unpackaging. Which is very helpful.
  … We will have to decide if it’s rec track or not. Personally I think it should be a rec track document.
  Tzviya Siegman: I think referencing the ISO document makes it much shorter. The only things needed in our document are adjustments — so it becomes very slim.
  Laurent Le Meur: but there’s no mention of font obfuscation, but still there is an issue where we have to discuss that…
  Tzviya Siegman: For now, leave it out, we can add it later…
  Dave Cramer: as someone who has spent too much time with the OCF spec, I hope we can write something clear about what is expected from authors and user-agents. OCF is a bit loose about what is happening with packaging
  Laurent Le Meur: as a first step for next week, we should discuss which kind of template or reading system behaviour. There are many ways to specify the user agents, so I would like some input from the group on which kinds of writing we should put.
  Tzviya Siegman: There will be a placeholder in the explainer — and we can work on getting the document drafted in the next few weeks