HPub definition

[svdgraaf]

Hi Guys!

Should we perhaps start with writing down the definition of the hpub standard, and then move on from there? Perhaps first setup a draft and get input from others? The Baker.app could serve as an an example implementation of the standard, and other people could start working on -for example- an Android implementation.

~

The current specification is here: https://github.com/Simbul/baker/wiki/hpub-specification

[folletto]

Yes, it's one of the biggest part I'm working on, I didn't open the issue because I wasn't expecting to discuss this part so soon and I need some time to synthesize what I already have, but surely we can start collecting feedback.

I'll mark this issue as discussion. :)

I can already say:

• the least number of mandatory parts
• no new syntaxes as much as possible

From those assumptions, hpub as you see today is already quite there: with alphabetical ordering and zipping it's complete imho, content wise.

We still need somewhere to specify a sort fo "meta" or "manifest" container. For the discussion I had up until now this is the current dratf:

• HTML syntax
• must contain title
• must contain author
• must contain URI identifier with specific "ebook" protocol
• should contain width (in whole pixels?)

[svdgraaf]

Another thing I thought of this afternoon (also in combination with the Footer bar/navigation), is thumbnails.

I can imagine an author would want to set an image per chapter (page), and one for the overall book.

We could set the preview image for the overall book in the meta file (or perhaps define a thumbnail.png, and the images could be set in a meta header or something, or look for an image with the same filename, but with a .png extension.

I would not define any image sizes, as the user can use the images in different ways.

[folletto]

Yes, but please always keep in mind that we need to strive for simplicity at least for version 1.0 of the HPub standard. If we start to add parameters, settings, thumbnails, etc. we'll make the life more difficult to book creators. Since HPub is not yet a "standard", we have first to aim for a good baseline, simple and clean, and on that build extensions (i.e. navigation, thumbnails).

For example: I was thinking if it was possible to dynamically generate the thumbnails from the HTML pages: that would allow the author to have ZERO effort and allows the user to get real 1:1 matching thumbnails.

However, I like the idea of having page thumbs that have the same name of the HTML with a PNG extension. That could be a good, minimal-effort, extension :)

[svdgraaf]

I would like to generate thumbs from the html pages as well, but I don't think that that is a option for now. The PNG extension is the best option for now I guess :)

[segundoseis]

hello! If the bottom navigation is contained in an HTML, I think that the authors will have more versatility. We could create a navigation as simple or complex as required by the design ... thumbnails, photo and text, just text, etc. Besides if we do not put the nav.html, we do not have navigation.

For the standard hpub, I think we could create a base template within nav.html (button text), then be customized if required by the author.

[segundoseis]

I think the good is that each author can customize the content to the maxim. Personally I'm starting to hate the adobe plugin. All magazines are equal! :-(

[folletto]

Exactly. Let's see that solution how could work out:

1. How would a user open the nav?
2. How could we make it so it's usable consistently (adapted, but consistent) no different page sizes (vertical, horizontal for example)?

[bru]

as for the thumbnails/icons, something along the lines of the html "favicon" seems simple enough. agreed that it probably belongs to a future version of the spec.

[svdgraaf]

I just had an idea. We can save the meta file as json file. Why? Because it's easy to parse (there is a good library for it), xml is a lot of unnecessary overhead, and the enduser can use the json file in their javascript in their pages itself to do additional stuff. And above all of that, it's easy to read, write and understand, and it's easy to add additional custom fields.

[segundoseis]

JSON +1 ;-). I like it

[frank06]

I asked in another issue why not use Zhook instead of creating yet another e-book format. Unfortunately all those comments are gone with Github's recent outage.

Basically the argument against Zhook was that it uses one single file instead of many, an is thus less capable or performant. (I'll explain below why I think this isn't really an issue, [1]).

In my opinion there is much more to gain from joining forces on a standard, than fragmenting:

• We're essentially doing the same thing
• Thought has been put into Zhook, and it's a format already being used (Monocle, Booki.sh, Widescript, ...)
• Interactivity is something that isn't limited by any of these formats. (As a matter of fact, I just finished authoring a really interactive Zhook -- more than anything I've seen out there)
• Tools: convert to and from ePub: https://github.com/joseph/peregrin ; also I should be releasing something very handy quite soon

Baker is a neat idea. It would be perfect if we could pull together towards the same goal, a solid, no-fuss alternative to ePub, making it easier for authors to decide where and how to distribute their content. What do you think?

[1] I don't remember who answered my last comment, it was said that it was non performant, heavy on memory consumption, etc. Whilst I don't think this is the case, it could be addressed by splitting files before bundling them into an app... we'll find some sort of solution? One of the longest books in history, "Les Misérables" by Victor Hugo weighs in at (only) 4mb of total HTML size; I'm sure there are other problems performance-wise -- which of course need to be measured first.

[folletto]

Yes, I completely agree that it would be better to join our effort, also because Zhook already has some degree of adoption. I tried to think about solutions to solve the differences, but so far I wasn't able to meet our needs using the Zhook format.

Basically the point, as of today, is: Zhook is almost a subset of HPub focused on a normal book experience. Because HPub and Zhook are both HTML5 zipped, but HPub supports multiple files, while Zhook only one.

Reasons for HPub:

1. It's not book-only. HPub targets everything from book to magazines.
2. It's based on a different interaction model made possible by digital readers: multiple chapters of unlimited length one on the side of the other.
3. We are focused on the simplest possible approach for the whole production chain of complete multimedia experiences.
4. It doesn't prescribe anything to the reader different from what's already obvious.
5. It's going to address the cross-referencing of books problem.

Why Zhook doesn't work for each one of those points:

1. It's book only. It's "the best of breed EPUB". HPub isn't, but allows a great book experience too.
2. Zhook is based on the old epub idea of a "whole page". HPub allows that, plus the new natural interaction emerged with the touch devices (Zhook is a subset of HPub).
3. With one big file, you are just allowing for one person working on it, or you require the editors/programmers/designers to use a versioning system to merge changes. If you split a book in chapters, then you can easily work on one file while another person works on another file. Faster to manage, easy to send, discuss. And since it's not pre-defined, you can organize the files in the best way you want to work with them. We don't even constrain the name of the "first" file: it's just alphabetical, exactly what you get from the filesystem. I can go on a lot on this, but as you might understand, it's really intuitive the way we are designing HPub.
4. Zhook leaves a lot in the hand of Reading Systems: "the RS must give the element an id that is unique", "the RS may modify the DOM", "the RS may split the DOM". When I'm building a book or a magazine, I want that my reader gets a consistent and determined experience. What if I build a wonderful JS/CSS interaction within my book and the reader "splits" it? How do I know? Have I to test every different implementation PLUS the HTML5 evolutions and quirks? Painful.
5. We are going to create the separated URI reference scheme and add to the HPub standard as a MUST. This is going to allow easy referencing, something that epub didn't get: they have an ID, but what could you do with that? Who would you query? Where's the file? And Zhook just misses it: "The unique book identifier is an EPUB concept that has no analogy in Zhook. I'm not really a fan of the idea — it seems like half a solution to a not particularly significant problem".

In the end, think about it: would you build a whole, complex, feature rich website inside a single HTML file? I'm sure not. :)

That's why we need to have multiple files. :)

That's why HPub and not Zhook.

But still: we could make Zhook and HPub ideas converge, if the author wants to (since Zhook to me looks like a single company invention, isn't it?). I also don't really care about the name, I don't really like HPub - and neither Zhook if I have to be clear - so I'm open to suggestions.

[joseph]

I'm going to append some thoughts to Francisco's comment first, and then get back to your specific Zhook questions, folletto.

First, yes, arguments about the performance of file formats are bonkers. They conflate two different problem domains. Here are the respective challenges:

The app: be fast The format: be easy to author

Zhook builds in zero requirements for the book creator to be mindful of app-level optimisations. But it does provide a simple algorithm for componentizing its single HTML file into multiple discrete chunks if the reading system desires it (that's what we do with Booki.sh, for instance).

Now I think there's more than enough room for multiple ebook formats to co-exist. The one-ring-to-bind-them-all can wait another decade at least. But various people have pointed out to me the similar goals between Zhook and hPub, and it seems worth covering how Zhook arrived at its present point. Francisco has done a terrific job, I'll back it up with a few points about why we went single-HTML-file.

Previewability. If you have multiple files, it's hard to preview the whole book in a web browser. If you're clever, you can install Firefox extensions that recognise rel links, and pop these in your <head>. Or you could get tricky with a meta HTML file that loads all the components in iframes. But if you want it to be dead-simple for web designers to make books, your best option is a single page.

Whole-book operations are common when editing. You've probably already discovered that what you tend to do when editing a book is perform batch operations. Some people like simple find-and-replace. Some get serious with regexes. Power developers will use a swiss army knife like Nokogiri. If you spread out across multiple files, things become a chore quickly, and destructive errors creep in.

You get structure for free (manifests suck). Obviously this is the most important point. You're looking at JSON to explain the structure of the book and capture its metadata. That's not bad. EPUB uses several byzantine XML files. That's sucky. But what are you really doing here? You're marking up an arbitrarily complex, single document: a book.

It turns out that's what HTML is built for. Want to chunk your book into chapters? Or nested sections within sections? Just wrap each discrete component in an <article> tag. Want to have a table of contents? Just give each section a heading: <h1>, <h2>, etc. Want metadata? Use the <meta> tag (or RDF-a for complex metadata).

Anyway, I had thought that if there was a point of difference between Zhook and hPub, it was reflowability. Zhook has to be reflowable (and paginatable) — hPub does not. Dismissing that requirement does potentially give hPub a number of affordances that would be worth exploiting. But I see there's been recent work to support multiple orientations and resolutions. Which arrives you at reflowability. At this point, if you'll excuse the pun, there might be some value in pooling our resources.

[joseph]

Different interaction models: I see this as a real point of difference. If we decide that HPub is for one interaction model, and Zhook is for another interaction model, it definitely makes sense to shake hands and wish each other well. Can we define these different interaction models? (If it's pagination vs scrolling, fair enough — my unshakeable opinion is that pagination is essential for long-form text.)

Book-only: yes, unashamedly. I wouldn't use the word 'subset', but it's not important: we can get together and compare Venn diagrams sometime.

Focussing on the simplest possible approach to production: then you'll want to get rid of manifests. :)

No prescription: that's fine as long as you've only got one reading system. Giving the <html> element a reader-specific ID simply allows book designers to do different things with different reading systems. You can leave it out, but soon enough book designers will beg for it. 100% of our focus with both Monocle and Zhook is on modifying the DOM as little as possible. But if you want to avoid it entirely, at the end of the day you've just built a web browser.

Cross-referencing books: that quote of mine about a "not particularly significant problem" is accurate. It hasn't bothered me so far. I think that there are definitely use cases of course, and they simply demand a central registry. Everyone making up their own IDs is just half a solution — and that's where EPUB currently stands. If you're going down this path, stand on the shoulders of giants and use Keith Fahlgren's work on the topic: http://blog.threepress.org/2010/10/21/developing-an-epub-linking-spec

Co-working: that "one big file" is the HTML. I don't really believe you would have multiple people working on that at the same time — it's not a workflow that makes sense to me. Now, you might have one person creating semantic HTML while another does CSS and even another does images/scripting/etc, but authoring the book content itself is more or less indivisible. In large-team cases, of course, you can write a 3-line shell script to stitch stuff together. I recommend you optimise for one-person book teams, not big teams. That's the most common case.

The difference between a book and a "whole, complex, feature rich website" is that a book is a single document — even when it has multiple sections with quite distinct personalities.

Zhook is no more a single-company invention than HPub. The official spec is a Gist — by all means fork away! Contributions very welcome.

Naming: I like HPub better than I like Zhook. Zhook is a ridiculous name! :)

[svdgraaf]

I see one big issue, which is my main point to prefer split files over one big file. I'm looking for ways to easily distribute magazines across different tablets. Every page could potentially have a completeley different stylesheet and have a lot of dynamic content (slideshows, videos, etc) which is not going to go well with one big file. Our HTML files are actually going to be built by multiple persons.

That being sad, it would be silly to both work on somewhat the same standard, and I like the idea of not having to use a manifest file at all.

Btw, the bigger players in the market (woodwind, adobe) do use multiple split files, with a manifest XML structure (last time I checked).

Oh, and hpub is actually a better name ;)

My 2 cents

[folletto]

From your comment, I'm thinking that you are referring a specific usage, both for authors and readers. As I've said before, HPub is born to be wide in scope and address a lot of different usage patterns.

Point-by-point, your first comment:

• Previewability: Since a reader is something external from a book, you can just make a simple JS engine that loads inside any div a book taken from a specific URL, doing exactly the same thing we do inside Baker Framework. I don't think it should be declared inside the book itself, like swf files or m4v files don't have the player embedded inside. No difference.
• Whole-book operations: you look like having a developer point of view, but even from that point of view the difference between a batch operation on a single file or multiple file is usually the difference between cmd-f and shift-cmd-f. But again, that's a developer's point of view. I'm talking about different chains, where authors deliver content in single articles or chapters. Where editors work on them. Where designers take it in Photoshop, InDesign or Illustrator. Where developers convert them to the right format. It's way more complex, and it's even more complex when you move from books to magazine or interactive books: multiple authors, multiple designers, maybe video experts for multimedia content. As I've said before: you would never do a website in a single HTML file. Plus for HPub.
• Structure for free: Zhook and HPub share the same structure: HTML5, so I can't see why we are taking about structure, we are on the same ground. We aren't talking of the manifest for structure, but for other things. About structure we're good with the current multifile, HTML5 base, exactly like Zhook. The manifest here is discussed for other reasons. No difference, with a plus for HPub if you need filenames to manage your structure. ;)
• Reflowability: it isn't a difference, because again, while Zhook inherits from epub and so it focuses on books, HPub don't. So if an author wants reflowability, he gets it. If not? Then it doesn't. HPub gives this choice to the author. Zhook doesn't. HPub is as simple as a website, as flexible as a website. Try doing our published book, "5x15 in Tokyo" in epub - you can't. Zhook? Probably, but it's going to be way more complex... unless it's extended to cover the same parts that HPub already does.

Your second comment:

1. Different interaction mode: from your comment it seems that you haven't tried the book. Please do. :) HPub is new because it allows an interaction mode that's what it's being used in many digital magazines: scrolling and pagination. With complete design control in the hands of the designers, pixel perfect. With again a choice for the author: if he wants to, he can paginate, otherwise, scroll, or any in between.
2. Book-only: I'm saying subset just because in some ways the standard is a "one file version" of the current, evolving, HPub definition. I'm sorry if it looked harsh, it was just a technical consideration.
3. Simple: Exactly, I'm trying to avoid the manifesta as much as possible. If HPub is going to use a manifest, will be to add MORE functionality than the one provided today by either epub or Zhook. That's what's boiling.
4. No prescription: Almost. We already have CSS for that and readers can simply load the CSS they require to work: it's the same. Instead of prescribing "inject an ID" the reader can just say "load this CSS". It's the same, without injection, and completely in the hands of the reader's hands.
5. Cross-referencing: I know that epubs position is wrong, and that's what we are planning to solve. I also think that Keith solution misses a big point (too focused on the book side again), but this is another discussion. ;)
6. Co-working: you're too developer oriented, and book oriented, as I was saying above. HPub solution includes book, but it's not limited at it. :)

We can discuss to merge the two standards in one, that would be wonderful, but HPub objectives need multiple files for all the reasons stated here and above. Otherwise you are just making another epub, and that's not something this project is for, that's not something we are receiving loads of enthusiast email, feedback (and more...) for.

From your second comment, I think that you built Zhook explicitly for books, while HPub has a wider approach. That's philosophically different, and leads to all the points we have commented before. But those points are a consequence of that choice

However, if you think about it, as of today the main differences are:

1. Must have one file -vs- multiple files (No difference).
2. Must have a specific name -vs- choose you name as long as it's alphabetically ordered (HPub is simpler).
3. Must include a cover -vs- no cover required (HPub is simpler).

After that, we are discussing a few other things that might require a manifest... or not. I'd like to avoid that, but well... let's settle one discussion before moving to another.

So: do you want to join forces? I'll be fine with a shake of hands and a good luck, but I'd really love to join forces, efforts and minds on that.

[joseph]

Of course I've tried Baker! I've had 5x15 on my iPad since your launch day. It's great, delightfully simple, and yes — seems to take interaction cues from the recent slew of magazine apps like Wired. I like the design quality. I hope the project isn't jeopardised by Apple's recent decision to ban 'templated' single-title book apps from the App Store.

My point about pagination vs scrolling is this: from a reader's perspective, as soon as you have any scrolling, you don't have pagination in a meaningful sense (you might still have article-ization or something). Pagination to a reader means "give me a screenful of text at a time". I like to apply the nose test: can I read this using only my nose? :)

Structure: no, this is really, really important. Of course all popular ebook formats use HTML for markup. Zhook uses it to describe the structure of the book to the reading system. The reading system optionally generates a table of contents from the headings. Components from the article tags. Footnotes from sup'ed links. Metadata from the head. All the meta-information about the book — everything you're planning to store in the manifest — is in the semantic structure itself. It's actually simply an outcome of using HTML5.

Well, here's what I think should happen next. I like Zhook's focus on books. I don't want to widen it to magazines, because I think they have very different problems to books. But I think something really terrific is happening here. Actual reading system developers are coming together and talking about the hard bits. You guys with Baker and HPub, Widescript (frank06) who are using EPUB and Zhook, and us with Monocle and Zhook. I'd really like to see this conversation continue. The environment at the moment is full of opportunites, and it's better to experiment than to try to consolidate things (if anything, this is my primary criticism of the current EPUB3 initiative).

EPUB3->HPub->Zhook all sit on a conservative/radical continuum of HTML5 ebooks, and it'd be good to share ideas and problems. Maybe some kind of ebook-dev Google Group? To the people here I would add Keith and Liza of Threepress (also reading system developers — Ibis Reader — and easily the smart and wisest people in our field), then blaine and augustl, who have both built interesting HTML5 ebook reading systems here on Github, and Hadrien Gardeur, the clever bastard behind Feedbooks.

[folletto]

• Scroll: yes, that's why we support also vertical single-page scrolling (tap once at the bottom, nose test passed!). I don't think it's "perfect", but seems working well so far. :)
• Structure: I agree. It's just slightly different with multiple files: you have to aggregate that structural parsing: since HTML5 have not anymore a "universal" h1, you can easily think that a group of files is in fact another top level article block, with the titles as h1. Zhook has surely a slight advantage here from my point of view, but not heavy if you want to build a single-html HPub. ;)

~

As I thought, you're surely way ahead of me on the book field. :) I'd love to join the discussion, but I also need to push out of the gate a first draft because we keep receiving a huge amount of emails of people building something with Baker and so I'd like to have something as simple as possible but with one more piece in place than we currently have: the URI (and you might easily guess why).

I think that if we are going to have a non-merger point, it's due to your focus on book and our focus on page-perfect magazine-like pages. But maybe... it could be just a difference in the same standard.

Even if...

I strive for simplicity. :)

[frank06]

Woah, that's a lot of text to digest.

I am thrilled to notice that both standards are not that far apart, and that everybody is "striving for simplicity". There seems to be, however, some confusion regarding the format's and the reader's concerns.

The (not-really-significant) differences boil down to:

1. One-page vs multi-page
2. Book-oriented vs magazine-oriented
3. Cross-referencing
4. The name

Let me develop:

1

The pages approaches have their pros and cons (previewability for Zhook, collaboration / "every page could potentially have a completeley different stylesheet" for hPub). But seriously, it can be solved either way by running a 3-line shell script as Joseph said.

A format is merely descriptive. Most of what we're discussing here is ultimately up to the Reading System. Zhook has it down right when it comes to using HTML5 markup to describe structure.

2

We are using Zhook to build highly interactive publications that share many features with magazines – and don't feel limited in any way by the format. It's flexible... But remember: it's a book. If you want to build an app, check PhoneGap.

Re: pagination and scrolling: again, up to the reader. Widescript doesn't paginate (we have our reasons) and Zhook works just fine.

3 I haven't dug enough into cross-referencing, but anyway this can be addressed by something on top of Zhook, like a convention on IDs.

4 Hopefully not too difficult to solve :)

[frank06]

And I'm down for discussing about these topics on an "ebook-dev" or similar group. Joseph are you creating one?

[folletto]

1. I agree again on the structure part, but I again say: would you build an entire website using just one HTML5 page? No, and that's for a multitude of reasons. The same reasons apply here: we're moving from "old" text-only books to "new" interactive, animated, web connected books. A paradigm shift. I can take one from the multitude of reasons: the web idea is already at it's core a hyperlinked book, and I think you can hardly say it doesn't have structure. ;) Since books need to move there, keeping the "one file" structure is incredibly limiting to the full potential. And as you said: if you prefer to build a whole website in a single file... good for you! But that should be a choice, not an obligation, and the same applies for books. :) I want that choice in the hands of authors.
2. I thought about this, and I think that it boils down to what's your idea of book. You make the jump from "book" to "app". For me there's a huge middle ground in between, like there's a middle ground between a single HTML page and a multi-continent multi-language flash-enhanced interactive 3-tiers dynamical website. HPub isn't magazine oriented, it's design oriented. It allows books to be designed with pixel precision, being sure on the result. Incidentally, that's something magazine would love too. And that's it. But again, I want to give choice, not impose solutions: you want a book to be fluid, single page, adaptable on multiple screens and devices? Do it. But if you want to simply do a stunning portrait photographic book on the iPad, you should be able too. I want to break free of that constraint about "ebooks" being "just text". I want to give back power to designers to create the books I loved and I love so much, but in digital form. And ready for the web.
3. That's another topic. ;) Check first the issue: https://github.com/Simbul/baker/issues#issue/14
4. I just want something good, I'm not attached to HPub at all. ;)

[joseph]

Do you have a practical example of a book that would be difficult/impossible to do as a single-file Zhook?

By the way, I love it when things pass the nose test! I hadn't discovered that bit of UI in Baker.

When it comes to URIs, well, here's two URIs illustrating why I think it's a non-issue:

• http://wind-in-the-ddcb27.reading.a.booki.sh
• http://www.20thingsilearned.com/html5/1

My answer to the question you asked on Twitter is 'http://'. :)

[folletto]

Oh I'd never say impossible (and I'd never say never... whatever :P). With a good effort and sweat you could always do everything with computers, you know. But I'm trying to be simple, and that's the point. :)

But I think it's a good point to move forward the discussion. As a reference: try re-doing "5x15 in Tokyo", the book we already published on the store, in Zhook, pixel perfect, with each page starting and ending where the author decided and not where the reader software wanted. And that's a quite simple book. :)

URI: please, could you move that part on the the other issue so I can comment? This one is already a complex discussion by itself. ;)

[joseph]

But that completely confuses the matter — unless the file format is tied to the reading system (as might be the case with HPub and Baker), the file format is independent of any one reading system's interaction model.

You could absolutely have Zhook as the underlying file format for Baker, where a statement like "with each page starting and ending where the author decided and not where the reader software wanted" actually makes sense.

In this case, each "page" is an article element. Zhook's recommended componentization algorithm does this for you.

So it would be quite straightforward to make a Zhook that did what you suggest.

But remember that the statement actually only makes sense in the context of an interaction model where you have scrolling pages.

[joseph]

Is the source of 5x15 available somewhere? Maybe creating a Zhook would focus the discussion a bit.

[frank06]

Yes, let's get practical. This is a Zhook: https://github.com/widescript/dynamo-paper/blob/master/index.html (just an example).

@foletto: Observe the <article> tags and imagine how they could be tore apart. It's that simple.

That's what we do at Widescript. Each <article> becomes a "chapter". Just like in 5x15 – only with a different navigational model but that's exclusively the Reading System's concern.

Again. The difference is simply a tradeoff:

• One Big File which you can easily preview, but perhaps too big for collaboration
• Multiple Small Files which you can collaborate better on, but you can't easily preview (unless you copy-paste you <head> tag across all of them).

That's it. Both compatible with a tiny shell script to go back and forth between "Zhook mode" or "hPub mode".

"Pixel-perfect"? Not sure what you mean. <article>s delimit chapters. You can split Zhook's index into multiple pages like this: /<article\b[^>]*>.*?<\/article>/ or xpath or whatever.

If you mean non-reflowable / Definite Content (http://craigmod.com/journal/ipad_and_books/) then that's a whole new discussion.

if you want to simply do a stunning portrait photographic book on the iPad, you should be able too

There's no reason why this can't be done with Zhook. One picture per <article> and the rest is CSS.

5x15 is totally feasible in Zhook. I can help out if needed.

[folletto]

Are you telling me and every author that is now working on Baker that instead of using a simple, clear and no-explanation solution, identical to the way we are already working, with a technical effort of zero, we should impose an arbitrary set of rules that requires paragraphs of text to be explained to split an HTML5 file? :)

I suppose I don't have to argue on that, right? :)

[segundoseis]

I think the html5 is the best option. I personally prefer multiple files with hyperlinks.

[frank06]

I'm not suggesting that at all.

I'm not even arguing against multiple files – that could be a good thing. Even having to copy-paste <head>, etc.

All I'm saying is the differences are smaller than you believe. ("One vs. multiple files" aside, what else is radically different that makes it worth fragmenting?)

How do we move it forward then?

@segundoseis: HTML5? Both formats use HTML5.

[segundoseis]

Sorry, I meant multiple HTML files.

[frank06]

This is my concrete proposal. We could build on from here.

It is a compromise between both formats. For practical purposes, let's call it NEF (New E-book Format, whatever).

Let's say NEF is a superset of Zhook, which makes it compatible with hPub.

• Multiple files are allowed. Ordered lexicographically.
• Multiple <article> tags are allowed in any page, but not required. If there are no <article> tags in a given page it means the whole page behaves like one big <article>.
• Metadata placed in the <head> section (as described in the Zhook spec) of several files is merged by the Reading System, removing duplicates.

Corollaries:

• A Zhook is a NEF
• An hPub is a NEF

Examples:

• If you go more the Zhook way, you get previewability remaining DRY. I would author a photographic album like this: several <article> tags, including one <img> tag each. Can't get simpler.
• If choose the hPub way, you get previewability repeating yourself, but with smaller chunks which are best for organization/collaboration. I would write a long book including metadata (<meta> tags) in the first chapter, and have one file per chapter. Much more flexible.

Authors would now have the power to build something that can be published virtually anywhere. This advantage comes at the expense of complicating matters a bit, but might be worth it.

Thoughts?

[segundoseis]

Sounds good. :) Would be possible that you make an example as you mentioned (photographic album), in both formats? For test would be fantastic.

[folletto]

Er I don't know what to say, because I don't have to change anything in the HPub specification to fit your proposal (HPub's spec today is incredibly simple). So for me it's fine. :D

Just a note: I'm trying to build a simple spec (see the top of this topic), so I won't probably go too much in detail as you did about metadata, reading systems and previewability. But I think that's a marginal point.

[joseph]

I like this idea, but I have two suggestions:

• call the first file index.html, and only get metadata from here
• rather than ordering alphabetically, use <link rel="next"> to chain subsequent files

Putting all the metadata in a single place will be easier for book creators. Calling the first file index.html conforms to standard web-server configuration, and therefore has practical benefits. It's also immediately obvious to any web developer which file is the central one.

Specifically chaining files together via a link is more "HTML5" than alphabetical ordering, and allows two possibilities: HTML files that are not part of the linear flow (such as might be included in an iframe, for instance), and "remote" parsing — that is, parsing where the contents of the directory is unknown, such as when getting a book off the web rather than from a local filesystem.

[folletto]

May I ask the reason behind those ideas? My guess would be:

1. Having a clear place for putting metadata, easier for creators
2. The ability of specifying a subset of files and not all the files

So, if I'm right:

1. Metadata: I completely agree on the principle of having everything in one place. This one was exactly my doubt about frank's suggestion above. Having a single place where I'm sure to find all my metadata is the best solution (both for simpleness and automation). But I'm not sure that modifying the simplicity and cleanliness of the alphabetical sorting part of the spec is worth forcing an index file.
2. Subset: The current HPub spec already handles this: subfolders. But you're right, I should have made it more clear. I'll edit the spec above. Handling the list with subfolders keeps the amazing simplicity of the list of files, but keeps allowing complex HTML inclusions like the one you were referring to (iframe).

~

As a sidenote for future discussion, note that the alphabetical ordering is a design choice that solves a huge amount of problems at once:

• Simple for authors, visually: every filesystem will show you the files always in alphabetical order by default. So to know that a file comes before another you just have to look at the filesystem. Dead simple, you don't even have to think about it.
• Simple for authors, freedom: they can have any kind of internal organization or structure and just use the first part of the filename to reflect it. 001-cover, 002-index, 003-chapter1, 003-chapter2? Do it. Prefer 01, 02, 03? Do it. And so on. Do you want to build a Zhook book with a single file? index, done!
• Simple for parsers: they just have to get all the files in the zip or in the folder and sort them. This is a basic operation that doesn't require any specialization or any parsing. And can be performed also remotely on webservers: just allow directory listing or add an index.html. ;)
• Simple as a spec: saying "the order is alphabetical" is crystal clear to everybody (well, maybe we have to catch some cases like japanese, chinese and korean, but in that case again is just a matter of building on the shoulder of giants when they built sorting algorithms).

[joseph]

It's strange that you're against a naming convention for the primary file in the book, but in favour of arbitrary rules for all the others (non-linear content must be in a subdirectory, alphabetical order must be preserved).

I mean, I like that CBR seems to be your inspiration, it's a nice simple format. I written two different comic book viewers over the years — one in C# and the other in JS. I've seen a lot of broken comics because of the alphabetical ordering requirement. Of course that's trickier to fix in the case of CBR, because augmenting JPGs with sequence data is not for lay-people. We can avoid it here, because HTML is much more informationally rich and editable. All it costs you is one <link> for each linear component.

To me, a link tag is the answer to the question: What Would HTML Do? That's the premise of Zhook — following HTML rules and conventions gives you constantly improving support in all rendering engines (and other HTML consumers, like spiders and web server modules).

I'm not happy with "just allow directory listing or add an index.html" — the latter is a manifest file in all but name, and the former is the sort of server configuration requirement that impedes adoption.

It may seem like an edge case, but this feature gives the file format one massive advantage: streamability. You can ignore the zip packaging altogether and just read your books straight off the server — either directly in a browser or via a reading system. To me, that's the future of ebooks: a format where the consumer no longer even sees the file. (And that's why I said 'http://' is the answer to your URI question.)

[elmimmo]

Contrary to popular belief, what is alphabetical order is neither obvious nor absolute. Think about when the name contains numbers (or, possibly, accented characters).

An example: Two files named "88.html" and "8b.html" would be ordered differently depending on the app:

• In Mac OS X 10.6.4, the Finder and other UI-based apps (such as Simple Comic) consider the appropriate order as 8b.html > 88.html
• The Terminal considers 88.html > 8b.html

Throw non-latin characters into the mix and people do not even know what "alphabetical order" means.

Besides, when creating content, deleting and adding content in beween other content is only natural. Having to change filenames every time that happens would not be a nice thing.

[elmimmo]

Another example of alphabetical order quirks:

1. 01.html
2. 01 addendum.html

Yet, the space comes before the period, so pretty much all software likes the following better:

1. 01 addendum.html
2. 01.html

[folletto]

@Joseph: Index.html, streamability, linked files... you're building a website Joseph, not a book format! :D

@elmimmo: Yes, you're right. However please note that the behaviour is consistent everywhere, with the only exception of OSX UI that treat numbers in a different way, handling sequences more cleanly (1, 2, 3, ..., 10, 11, 12) instead of overlapping due to proper alphabetical order (1, 10, 11, ..., 19, 2, 20, 21, ...). But that's a OSX variation (aid?) from the norm.

The problem you raised about inserting pages is however a good point, I was thinking about it yesterday and I think it's the only big drawback when using alphabetical ordering: "I need to add a file... oh I have to rename all the following until the last chapter".

I still don't feel that the solution lies in hard-linking files internally, because the structure gets hidden away making the work of authors and parser a real pain: "is this file before or after?", "wait, if I add one file here I need to edit the file before and the file after...every time".

The third approach that came into my mind was using an alphabetical default and then if the toc exists in the manifest file, it gets read and takes priority. Handling the sequence in a single, central file makes the content much easily organized and understood, at the price of requiring again another file.

There's not obviously clear solution to me here.

[frank06]

Compelling points against alphabetical ordering.

Linking seems to be the best we've got. If you don't want to mess with "is this file before or after...", just use the article tag approach. You should be able to combine multiple files and several articles in a same file, in any degree.

I would avoid manifest/metadata files. Start adding extra stuff, and you'll end up creating the new EPUB.

The beauty of Joseph's proposal is that everything remains semantic HTML (and ultimately yes, I do think ebooks are websites).

[folletto]

Joseph's solution isn't just beautiful. Is elegant, simple and clean. But it's designed to solve a different need than the one we're solving here. Otherwise we would have just taken Zhook. Right? :)

I hope that everybody here also understands that avoiding at all a manifest file is utopian, so we are surely trying to avoid it as much as possible, but not just for a principle: because avoiding it would make it simpler and cleaner. But at one point it's going to appear, and the file ordering might be a reason for that to appear: writing a filename in the right position in a file is without any doubt faster, cleaner and understandable than chaining files together. Just to make an example: to understand a book I'd just have to check it and I can understand, I can even read comments maybe on its structure, and so on, without the need to open each and every file to follow the chaining.

But I need a little more feedback to clear up my mind on this because:

• Alphabetical: while incredibly cleaner, it needs probably a more specific definition of "alphabetic" (ISO 12199:2000? Unicode Collation Algorithm?) and is a bit slower when inserting files in some cases.
• Linking: is always slow, it hides the structure and it's hard to maintain.
• Manifest: adds a manifest file, while cleaner (even if a little less clean than alphabetical) and understandable (not to forget: commentable).

[totanus]

Hi, I've been invited by Folletto to participate in this discussion because I had another one with him privately and we both think it could be better to go on publicly in this thread.

I apologize if something I'll say has already been discussed. I tried to read all the posts but I jumped here and there hoping to get the whole sense of the discussion without spending all the day in it.

Here's my cent.

Baker is a framework dedicated to simplicity and I love it. Its current biggest limitation - if we may consider it so - is that hPub doesn't support a real multi-document/multi-level navigation. The main reason is that it doesn't provide a set of rules for advanced T.O.C. creation (declarative or implicit).

I'm not talking about the hypertextual capabilities of the documents, of course, but I refer to the capabilities provided by the specs, which rule also the capabilities of the framework (or other future readers based upon the format specs :) ) in parsing and understanding the inner structure of the hPub.

If I use hPub "now" I'm stuck with a linear 1-level navigation.

In my opinion hPub should offer rules and directives, supported by proper tools, with the aim to create an authoring environment based on progressive complexity, instead.

Simple documents deserves simple structures, therefore light configuration is mandatory. At the same time magazines and books seldom have a completely linear navigation system, or however I wouldn't bet that tomorrow they will behave in the same way they behave now.

In my opinion all that can be achieved in few simple steps:

1) Introducing a manifest file with, at least, an URI protocol. The URI gives a proper ID to the publication for reference in-device and, maybe, on the web. 2) Extending the alphabetical parsing to the subfolders (according to unicode mapping) 3) If the manifest file contains a TOC this one overrides the alphabetical parsing (as Folletto wrote some days ago)

So, let the author take the decision between:

• simple alphabetical sorting
• advanced alphabetical sorting (with linking and subfolders)
• custom navigation (with a dedicated TOC section in the manifest)

The real challenge, imho, is to build a specs document that allows to start with a linear navigation/structure but lets the author create complex multi-level and multi-document structures, connecting multiple, even encapsulates, different islands of content.

hPub should be whole-comprehensive while Baker should be flexible in parsing hPub documents, discerning different levels of complexity in the documents as they are explained in the current (or missing) TOC section of the manifest file.

Thank you! Teo

[folletto]

Hi Teo, I agree on most of what you said. I have just a couple of questions:

1. Do you thing that 3 levels are really needed? Wouldn't be better, for simplicity's sake, to just have alphabetic (simple alphabetical sorting) + declarative (toc)?
2. What is the need of including subfolders in the alphabetical ordering?
3. Alphabetical with subfolders or TOC based, the hierarchy wouldn't be visible to anyone, so: how do you expect a complex toc or folder hierarchy to be expressed in the UI?

[segundoseis]

More than one levels, could we getting a multiple hpub document? ...many hpubs within an unic hpub. Examples:

• Several issues of a book.
• Monthly magazines. ...

[folletto]

What's the need for that that can't be solved by simply handling multiple HPubs? :)

[totanus]

Hi Folletto. Please consider a magazine concept and a magazine workflow. A contributor works independently creating his own article structure (not necessarily a simple one), then he sends the directory to the editor. She just checks the content and then collects every directory and aggregates them in a single publication, adding just the needed html documents in the root. Somewhere in the publication there is a link to an article previously published. The app automatically checks (using the URI ID provided) if the article is present on the device. If not it asks for purchase or wish-list.

This scenario needs subfolder and link parsing to create an automated (and extensible) TOC and that could answer to your questions #1 and #2.

About #3, almost every e-reader features a TOC interface (the base UI is a tree-view) and I think it could be useful both for the authors and the users. (edit: almost any serious paper book features a navigation map reflecting the author mental model of the document. ;) )

If you wish further details or scenarios I need some more time... :) However, yes, I think that those 3 levels may cover the 99,9% of cases and create the environment for new kinds of editorial interactions.

Thank you! Teo

[folletto]

Uhm I'm still missing something in your scenario, I'll try to provide one, slightly different, with a proposed solution.

Let's start from the magazine workflow.

The contributor working independently would do everything that you said, with an html for each article and folders for its assets. Something like:

• article-ozone-layer.html
• article-ozone-layer/image1.png
• article-ozone-layer/image2.png
• article-ozone-layer/style.css
• article-ozone-layer/facebook.js

The editor takes all the html + assets and merges those together in the HPub, renaming just the html files if required

• article-01.html
• article-ozone-layer/image1.png
• article-ozone-layer/image2.png
• article-ozone-layer/style.css
• article-ozone-layer/facebook.js
• article-02.html
• article-sequoia/image1.png
• article-sequoia/image2.png
• article-sequoia/image3.png
• article-sequoia/style.css
• article-sequoia/facebook.js
• ...

This works even without any explicit ToC file description, because the editor just needs to put the HTML files one along the other, with an alphabetical renaming. Nothing else needs to be changed. Consider that Baker's interaction model is an extension from the book model, so pages are sequential (linear, not hierarchical), even if unlike physical books each page can scroll vertically. So we are going always to have a flat line somewhere.

Where does this example fail from your point of view?

~

About the ToC UI, we are planning to leave this up again to book authors. There was a suggestion about creating an HTML overlay. The aim here is giving complete control on the book visuals, without any UI element (well, at some point we are probably required to add our own UI, but ToC I think that would be fine if it's managed by the author, both in structure and visual styling).

Do you think that an automated ToC generation is mandatory?

[bru]

can confirm that with collaborative efforts (magazines and non-fiction books), being able to bundle and send your contribution to the editor/curator is a common, desirable practice. That said, I'm not convinced that having several ways of determining the order of the files (e.g. alphabebetic, TOC, folders) would make anybody's life easier.

[bru]

to build on that, I'm still making up my mind on the various options, but so far an index(.html) file seems to be the most straightforward, and yet powerful, option: no hidden magic, everything clearly described, and just one place where to access and edit your metadata, ecc.

[totanus]

@Bru, are you suggesting to use the index.html as manifest file, perhaps using a microformat html-based tagging to configure the publication's metadata? I couldn't agree more... It sounds absolutely fine to me.

@Folletto: your example scenario does work. But I'm still concerned about manageability. However, let's pass over this. I think you got the point and made your decision about it.

About the need for a TOC structure. I just want to make clear that I separate what the hPub definition is and what is the Baker implementation of the definition. Well, that said, I think that the hPub 'definition' should cover also a TOC generation section because there's no such a thing called "document" without a its own structure and when you delegate the structure definition to the file naming, well, its like building a city without planning (imho, of course). You can do that for small docs, but it's not convenient for books and magazines. Furthermore, TOC is not just a navigational tool to the reader, but it is also an outliner and a validator to the author.

Then, Baker is still in charge to manage the TOC generation or parsing/visualization the way you want (ie: a visual drag&drop with html5 native features, for toc generation), or even completely ignore any implementation of the specs about that topic.

It's not for the sake of complexity, but it's just about a philological procedure.

That's it.

Hi, again! Teo