Closed JonathanMagus closed 4 years ago
dougmassay
commented
9 years ago Thanks for the input. It will be taken into account. However, please note that in no way, shape, or form have we indicated that Sigil is now an EPUB3 editor. The 0.9.x series will represent intermediate steps in that direction. The hope is that v1.0.0 will be an "EPUB3 editor."
kevinhendricks
commented
9 years ago In Sigil-0.8.901, If your content.opf package tag actually specifies version 3.0 and if you have the proper html5 style doctype specified, Sigil should not play with the DocType at all. If it does, then it is a bug. My tests of reading in an epub3 show the correct html5 doctype after opening even with Clean on Open set.
As for font files, yes, woff is new to epub3 and this is a trivial fix I to make sure it gets put in the right place in the Sigil hierarchy.
Please a small testcase or sequence of steps with an epub3 that corrupts the doctype and I will try to track down what iscausing the issue and get it fixed.
Kevin
kevinhendricks
commented
9 years ago Hi,
Just committed the one line fix for recognizing .woff files as fonts. That said, I simply can not get the html5 style DOCTYPE to get corrupted at all after reading in many epub3s, even with Clean on Open set, and even after editing the files in BookView.
Of course, these epub3 have proper package tags in the content.opf. If package version is not set to 3, then the DOCTYPEs will inccorrectly be replaced with their epub2 versions. So please double-check you have properly edited the content.opf and that the package version tag is set properly to 3.0.
I am closing this issue as I can't recreate your error. Please feel free to reopen it if you can supply either a sequence of steps to recreate the problem with any epub3 or alternatively post a test case epub3 someplace and send a personal message to "KevinH" on the MobileRead Forum site with the link. If I can recreate the issue, then I should be able to fix it asap before our fast approaching Sigil-0.9.0 release.
Thanks for your bug report about woff fonts not being recognized. This should now be fixed in master and in our upcoming Sigil-0.9.0 release.
JonathanMagus
commented
9 years ago Hi The test file validates without error as ePUB3; however, a more recent version of it is handled by Sigil without changes to the <!DOCTYPE html> tag. So, I will have to investigate why that is.
The problem with the XML files remains though. These are files that are not part of the content and are related to interactive functionality. In one case Sigil seems not to recognise empty tags ('/>') and fails the file (so doesn't re-write it), whereas in the other, which has opening and closing tags, it incorrectly re-writes the file as XHTML and adds a completely inappropriate 'DOCTYPE' header, assuming it to be a content section of ePUB2.
Admittedly this is probably not a typical example of ePUB3 by current standards; however, it is a valid file that falls within the rules of ePUB3; and is the kind of thing that is likely to occur in advanced examples. So, it would seem sensible for Sigil to recognise when an XML file is what it says it is and not attempt to convert it to something else.
I will send you links via MobileRead in the morning.
JonathanMagus
commented
9 years ago Hi again
Having just checked it, the earlier version of the file doesn't have a <!DOCTYPE html> declaration at all, though does state correctly that it is ePUB3 in the .opf file. At the time it was made that was technically correct (the XHTML5 spec being fluid); and as I said it still validates as ePUB3. This presumably means Sigil is not recognising the .opf statement and is writing in an ePUB2 doctype with a DTD, where it should just add a simple <!DOCTYPE html>.
I will send a link to that file too in the morning.
The XML problem is in both of them.
JonathanMagus
commented
9 years ago Links sent now in fact.
kevinhendricks
commented
9 years ago Hi, So this is some sort of epub 3 torture test, is that correct? And the issue is in how Sigil has treated xml files as xhtml files and messed them up by adding xhtml doctype and headers? Is that correct? Given that googles gumbo parser is an html5 parser, I am sure using it on pure xml files will introduce lots of errors as parsing polyglot html5 is quite different from parsing pure xml. If I can identify them on import I should be able to prevent that.
Thanks for the test case. It will be some time before Sigil will be able to handle many of these features.
JonathanMagus
commented
9 years ago Well no, not intentionally. It is a sample file from over a year ago that showcases some of the potential of ePUB3 when media and interactive elements are incorporated.
Yes, an issue is that Sigil is not differentiating between an XML and an XHTML file and yes, it would be good if that could be fixed.
The other point is that the earlier version of the sample file doesn’t use DOCTYPE, however, does declare itself as ePUB3. The fact that this validates, so arguably doesn’t need to be corrected, isn’t the issue here. It is that Sigil introduces an ePUB2 DOCTYPE statement, when the .opf file clearly states it is ePUB3. If it is going to auto-correct in that circumstance it should do so correctly.
Not accepting (or allowing the creation of) logical folder structures and component positioning as determined by the user remains a stumbling block, as is evident with the sample supplied, although presumably an ePUB3 template based on Sigil’s locked-in structure would currently work in it?
I am not sure that Sigil needs to be able to create complex content to be a great piece of software. That may be done with toggled editing tools. Interestingly, apart from the XML issue, it accepts the later version of the sample file and the only other error it thinks is present is the absence of a toc.ncx, which isn’t strictly necessary in an ePUB3, though is always good to have I suppose.
So, Sigil doesn’t seem far away from being a viable ePUB3 editor, which is great!
Thanks.
kevinhendricks
commented
9 years ago Changing Sigil to allow fully user-selectable destinations would require a complete and I mean complete rewrite. That may happen some time in the future when everything else is taken care of but not now as it would set us way back.
My plan now is to add a Scripts folder to hold all javascript pieces; (similar to what you had) and then use Misc for pdfs and other non-traditional ebook pieces. The .smil and .pls would go into Audio or Video or Misc, not sure yet.
But skins and layout pure xml like the Vallon360 is an issue. Perhaps an XML folder should become the standard location? These are all things we have to think about as we make the move to epub 3.
Thanks,
Kevin
JonathanMagus
commented
9 years ago It would certainly make sense to me to have a Scripts folder as part of a logical generic structure and in theory that would offer eBook creators more flexibility when beginning a work directly in Sigil. However, always organising files of the same type in the same place, whilst in principle best practice, has limited potential in the longer term and doesn’t fit the model for many instances of interactivity.
In the sample ePUB3 some of the folders house self-contained interactive elements, which have different file types in them in a mini-relationship that fulfils a purpose (including scripts and XML, like the 360 folder). Interactive HTML5 authoring tools typically output like this (sometimes with proprietary content) and moving their components around tends to mess things up. This is also the way that the IDPF’s Scriptable Components Specification is currently heading, towards the packaged widget with its own independent structure.
I have only used a Scripts folder in the sample ePUB3 for items that were relatively easy to separate off and to avoid repetition of scripts with generic functionality. The more specific instances needed to stay where they were (applying the quit whilst you are ahead principle in my case). That made sense to me at the time, although it could be argued that repeating a multi-purpose script locally might sometimes be the better option if it kept a widget self-contained.
So I wonder, as well as a Scripts folder, possibly as an interim solution, is it possible to create a piece of neutral territory in Sigil, a Widgets folder perhaps, where whatever is imported is left as it is? Or, where the user can create sub-structures and decide where things go?
Apologies if I am talking nonsense here. I am not a programmer and this is just speculative thinking, which may be impractical from your perspective.
Jonathan
kevinhendricks
commented
9 years ago Hi, A Widgets folder might make sense. The issue is when Sigil opens an epub3, for the first time, how can it tell that this specific .xml file, specific .xhtml file and specific .js should not be touched and that really Sigil should only store them. I guess if someone would create an epub with a Widgets directory, Sigil could see this and store things properly, but how would Sigil know to do that with regular epub3's anyway that were not designed to be opened in Sigil?
It is too bad that the epub3 manifest properties do not somehow specify, manifest entries that should be treated in this special way. Having a manifest property that tells me if scripted, switch, mathml, svg is generally pretty worthless for any reader and makes it horrible to keep the content.opf up to date when xhtml editing is being done.
It would be much better for the content.opf manifest properties to call this a "widget". As far as I can tell, there is no way to detect that something like the Vallon360 folder should be treated differently than any other set of .js, .xml, .xhtml files upon initial load.
Is there any IDPF proposal on the table for identifying self-contained widgets and their pieces somehow in the opf?
Thanks,
Kevin
JonathanMagus
commented
9 years ago Hi Kevin
As I understand it what the IDPF propose is here:
http://www.idpf.org/epub/sc/pkg/
and here:
http://www.idpf.org/epub/do/#h.br96k0msbdvd
I have only skimmed through it and haven’t made complete sense of it yet, however, think it means a widget would appear as a collection in the .opf file (I could have that wrong though).
Since the ePUB3 sample was made over a year ago and the Draft Specifications have yet to be adopted it doesn’t attempt this. My impression is that the IDPF is thinking about reusable widget components though, so I am not sure this would always be applied.
Presumably similar issues would arise for Sigil where other separately structured content is referenced, as for example where the ePUB3 file contains two ePUB documents, or two ePUBs with shared components, which are possibilities.
And yes, an ePUB not designed in Sigil may have a completely different, or alternatively named, folder structure and different file names, whilst being completely valid. So, as long as Sigil imposes its own organisational preferences it is not going to be able to edit that effectively regardless of whether it has a Widgets folder and a Scripts (or js) folder, or not.
That doesn’t prevent it evolving as an authoring tool and editor of its own output, as long as the folder architecture it imposes is sufficiently flexible for different scenarios, although until it allows users to override and extend some of what it does there are bound to be limitations.
Actually the repositioning of components only really matters if the links between them are broken. An incentive for leaving the pieces of a widget together is the complexity of working out which bits reference which others when an authoring tool has been used to create it.
Sigil doesn’t seem unduly upset when it opens examples from the EPUBTest suite I see, although whether it saves them intact I haven’t checked:
http://epubtest.org/testsuite/
Regards
Jonathan
kevinhendricks
commented
9 years ago Hi,
Thanks for those links. As for reformatting the epub upon load to standard folders, that really is not what creates the issue. The issue really happens when in Sigil you merge xhtml files, split html files, rename xhtml files, etc. If there is some xml someplace that references those files (outside of the opf and ncx) then those need to have a dtd and be parseable by an xml parser and Sigil somehow needs to be able to update links in them no matter where those files are.
So standard folder names and relocating things to them or not, Sigil still needs to support auto link correction and allow merging, splitting, and renaming of resources. In order to do that, Sigil needs to be able to grok/parse anything that references those xhtml files in any way.
So even leaving an island or container, to properly support things, Sigil needs to understand what references to xhtml files exist outside of the content.opf, and toc.ncx in these container xml files.
Anything else just reduces Sigil to be just a text editor and not smart enough to make the kind of changes in xhtml that epub authors may want to have happen.
Hope this explains my predicament with these islands/containers that reference material in the book.
Perhaps Sigil should disallow all of these containers and require the author to install these containers, after Sigil is done with it.
Thanks again for your ideas and feedback.
Kevin
pettarin
commented
8 years ago @kevinhendricks Hi, first of all, thank you for your work on Sigil. While skeptical at first, now that Sigil has plugins and a roadmap for EPUB 3, I am very willing to contribute some time to the project, since I see it as the only FLOSS project that has a chance to become a great EPUB3 authoring tool.
Back to this issue: I would like to briefly mention that for EPUB 3 Media Overlays (SMIL), the current "put them in Misc/" treatment breaks them, since each SMIL file references one XHTML file (or more) and one audio file (or more), with the usual relative-path convention.
If you need a sample, download this CC-licensed file (I am the guy behind it): http://www.readbeyond.it/samples/bierce01.epub
Let's consider this simple example: 1 SMIL file (OEBPS/Text/p001.xhtml.smil), referencing 1 XHTML file (OEBPS/Text/p001.xhtml) and 1 audio file (OEBPS/Audio/01.mp3).
Interestingly, in this case the OPF gives you some clue about the relative positioning of the SMIL file and the XHTML file (notice the media-overlay attribute, referring to the id of the associated SMIL file):
<item id="p001" href="Text/p001.xhtml" media-type="application/xhtml+xml" media-overlay="s001" />
<item id="s001" href="Text/p001.xhtml.smil" media-type="application/smil+xml"/>
<item id="m001" href="Audio/01.mp3" media-type="audio/mpeg"/>Unfortunately, the OPF does not give you any clue about the fact that the SMIL file references the audio file m001. To get that, you need to parse the SMIL file:
<smil xmlns="http://www.w3.org/ns/SMIL" xmlns:epub="http://www.idpf.org/2007/ops" version="3.0">
<body>
<seq id="seq1" epub:textref="p001.xhtml" epub:type="bodymatter chapter">
<par id="p000001"><text src="p001.xhtml#f000001"/><audio clipBegin="0:00:00.000" clipEnd="0:00:02.600" src="../Audio/01.mp3"/></par>
<par id="p000002"><text src="p001.xhtml#f000002"/><audio clipBegin="0:00:02.600" clipEnd="0:00:05.120" src="../Audio/01.mp3"/></par>
<par id="p000003"><text src="p001.xhtml#f000003"/><audio clipBegin="0:00:05.120" clipEnd="0:00:10.720" src="../Audio/01.mp3"/></par>
...Luckily, the SMIL file is XML with a know, fixed schema, so this point should be workable somehow.
=== === ===
I have just forked the repo, and I plan to study the code, as I am willing to give as much help as I can for supporting creating/editing Media Overlays in Sigil.
I currently see three possibilities:
Misc/ , Text/ , Audio/ are all plausible choices, albeit somewhat arbitrary);What do you think?
kevinhendricks
commented
8 years ago @pettarin Please let us keep all epub3 related and (issue if moved without updating the opf properly) things here. So I have closed the issue 162. There are a number of things that will have to be dealt with on our way to epub3, smil and parsing raw xml (with and without a dtd/schema) are all related.
It may come down to simply "stop moving things" but I am hoping there is some way to deal with many of these issues. For example if someone were to delete the xhtml file that had a specific overlay pointed at it, how should that be treated? Does that mean delate the overlay as well, and the related audio file?
If someone renames an xhtml file that has an overlay, then we should properly need to handle the metadata refinements that point the overlay at that xhtml file. So a lot of new state information is going to be needed to be kept in the opf in some sane way that will let Sigil automate these types of tasks.
At least, the new container related tags in the proposed epub3/OPF at least give us some hope of identifying the containers you first talked about. I am still studying the proposed epub3 changes. I just wish they would stop playing with it all of the time. It is simply too hard to hit a moving target.
My plan is to evolve Sigil slowly. The first step is to get Sigil able to handle epub3's that are basically just updates to current epub2's with improved metadata gui editing and nav generation. Once we have that, we can start on overlays next. Then move on to containers, displaying mathml (which is still an issue as Qt's webkit and Chrome's webkit variant do not support mathml yet.) and other things.
Thanks,
Kevin
pettarin
commented
8 years ago Sure, no problem. If you need help with the transition to EPUB3 (e.g. questions on the spec), I will be glad to help, especially with Media Overlays (MO), as I have a fairly deep knowledge of their intricacies.
While MO are nowadays mostly used for EPUB 3 in Fixed Layout rendition, I know they could be beneficial for readers with special needs (dyslexic people, SEN kids, etc.), when coupled with reflowable rendition. In fact, DAISY voted for EPUB 3 + MO as the format to replace its previous DAISY Talking Book format. That's why I am so interested in MO in reflowable, and in having some sort of support for them in Sigil.
Given the status of the project, probably the best route at the moment is me to develop a (output) plugin, and gather feedback.
(And yes, I totally understand the pain IDPF is inflicting to independent developers, I am in that number.)
kevinhendricks
commented
8 years ago @pettarin @JonathanMagus
FYI
The upcoming release of Sigil 0.9.3 will include some rather large changes to implement experimental epub3 support.
These include a General Settings to set how you want new epubs to be created (epub2 or epub3). All Mend, Prettify, Split, Index generation, and other Sigil Tools are now epub type aware to prevent introducing any breakage. The plugin interface has been extended with improved epub3 interface calls in bookcontainer. There is now an epub3 Tools menu under Tools that includes an Update Manifest Properties, Generate Nav from NCX/Guide, and Generate NCX from Nav.
SMIL files are now recognized on input and automatically updated when repositioned to Misc/. I tested this with the pub3_samples moby_dick_mo.epub and nothing was messed up in the src links after loading it into Sigil. Of course Sigil will not play overlays like a reader would, it can still be used to help edit/create them.
Using the SMIL code as a model, I was also able to generate similar recognition code for page-map.xml files. In the future I will add support for other common XML files that have links that point into the main contents of the book.
I have started using the epub3 Testsuite to see exactly what is supported.
Of course, when editing in CodeView you can use any features you want but in BookView and in Preview the following will not show up as expected:
The only other test failures were as follows:
Everything else passed. We have even added support for LocalStorage to BV/Preview.
So things are looking better for basic epub3 support.
Future work will be directed at:
Then once all of those are done and any bugs ironed out, I will begin work on removing Sigil's need to put everything in a standard location. But that is still only a goal for after getting something useful for epub3 out there.
All of these changes are now in Sigil master if you build your own. Anyway, if either of you has Mac OS X, I would also be happy to create a preview build of Sigil 0.9.3 and post it for you if you are willing to test it out and help track down epub3 bugs prior to release.
Just let me know.
KevinH
pettarin
commented
8 years ago Hi, thanks for letting me know, and great job!
Building from the current master right now. I will test the MO-related stuff later today and let you know.
A few other comments on the other things you wrote:
I will write more feedback after I test the current code.
kevinhendricks
commented
8 years ago @pettarin
Fyi - at the last minute, I added in a single file mathjax.js polyfill for Preview only. It works just fine on MacOSX but will probably not work for Linux/Windows yet as we have not yet had a chance to add the code to install it properly on Windows and Linux. DiapDealer will take a shot at it today. I'll let you know when its buildable/working in Linux/Windows.
Kevin
pettarin
commented
8 years ago @kevinhendricks
I mainly checked MO-related stuff, so no problem about the MathJax polyfill.
Everything seems to be working well, except one critical issue: the paths in the patched SMIL files are missing a "../" prefix. For example, you have a
src="Audio/01.mp3"where it should be:
src="../Audio/01.mp3"Since the SMIL file lives inside the OEBPS/Misc directory, while audio and text files are in OEBPS/Audio and OEBPS/Text, the relative path should be e.g. ../Audio/01.mp3.
Steps to reproduce:
I guess it is an easy fix in performSMILUpdates() inside src/Resource_Files/python3lib/xmlprocessor.py where the rel path is computed.
I am not sure whether the same problem affects performPageMapUpdates() as well.
=== === ===
As far as I can tell, the MO-related OPF elements (<meta property="media:duration">, <meta property="media:active-class">, <meta property="media:narrator">, <item media-overlay=...>) are all handled correctly.
=== === ===
I also tried my plugin icarus, and it needs some fixes, due to the fact that the new code is patching the SMIL/OPF files. I will take care of this issue when you release the next version of Sigil (0.9.3?).
I was wondering whether there is a way for a plugin (i.e., the plugin Python code) to know the Sigil version it is running on. (That way, I can code my plugin to work on Sigil v0.9.3 (new code) and v0.9.2 (old code), instead of having two versions of my plugin, one for the old Sigil, and one for the new Sigil.)
In case such a function is not there yet, I suggest adding it to the Plugin APIs.
=== === ===
Thank you for your hard work on Sigil!
AP
kevinhendricks
commented
8 years ago @pettarin Thanks for catching that. I will correct it today. I will also check with the page-map parts as well. Yes, the bookcontainer call to launcher version will return a yyymmdd date as an int different for each release. There is a mapping to Sigil version numbers kept up to date in the Plugin thread at Mobileread. There is also new interface code in bookcontainer to determine easily the epub version your plugin has been handed.
The epub3-itizer next version will only allow epub2 epubs as input otherwise it will bail.
I have found a set of polyfills for epub:switch, epub:trigger, and several mo player and related javascript polyfills from the Readium project here:
Any idea if the mo related ones are any good.
Also I am hoping to find some code to map ttml into vtt since Qt's webkit does not support ttml. I found info on mapping from 1 to the other here:
http://w3c.github.io/ttml-webvtt-mapping/
Any idea if a js or python based conversion code from ttml to webvtt exists anyplace under any type of usable license? Or any other way to support ttml in Qt's webkit?
Thanks, Kevin
pettarin
commented
8 years ago aeneas:read TTML: https://github.com/readbeyond/aeneas/blob/master/aeneas/syncmap.py#L1048 write WebVTT: https://github.com/readbeyond/aeneas/blob/master/aeneas/syncmap.py#L1071
but it handles only a subset of features and it might require you to extract a few other functions (like the time value formatters). You might find easier to just write it yourself, taking advantage of the "parsing strategy" suggested by the above code.
kevinhendricks
commented
8 years ago Hi Alberto,
On Jan 17, 2016, at 10:36 AM, Alberto Pettarin notifications@github.com wrote:
• Fix: great, thank you.
The above fix has now been pushed.
• MO polyfill: Readium JS works well, but, not being a Web dev, I have always had problems in isolating the MO code from all the rest of Readium JS code, to the extent that I found easier to code my own plain JS (admittedly, for a subset of SMIL trees, namely those that are "linear chains without gaps").
Is this the .js. stuff you pointed me to earlier? Is your code under a license Sigil can use?
• TTML to WebVTT: https://github.com/pbs/pycaption is under Apache, but Python 2.7 only. I have some code in aeneas:
read TTML: https://github.com/readbeyond/aeneas/blob/master/aeneas/syncmap.py#L1048 write WebVTT: https://github.com/readbeyond/aeneas/blob/master/aeneas/syncmap.py#L1071
Thanks, I will take a look at that once things are more stable.
but it handles only a subset of features and it might require you to extract a few other functions (like the time value formatters). You might find easier to just write it yourself, taking advantage of the "parsing strategy" suggested by the above code.
Understood.
Thanks again for all of your help!
Kevin
pettarin
commented
8 years ago Is this the .js. stuff you pointed me to earlier? Is your code under a license Sigil can use?
Yes, I was referring to rb_smil_emulator and finetuneas, linked above. Both are under the MIT License. Again, not sure they will save more time than they require to extract the useful bits. If you can work it out, Readium JS is probably the best option.
kevinhendricks
commented
8 years ago Based on these discussions - the following plan is for future releases after the upcoming Sigil-0.9.3:
High Priority:
Medium Priority:
Low Priority:
Long Term Only
Does this sound reasonable? Do you think given your toolsets for mo, that employing them as edit plugins for epub3 is the best approach here in the short and medium term and maybe even long term?
Thanks,
Kevin
pettarin
commented
8 years ago It looks fine. I have only three observations:
A. "gui for editing mo metadata" => The MO metadata required by the EPUB 3 MO spec are the media:duration (one refined meta for each SMIL asset, plus a meta with the sum) and ideally they should be auto-added by inspecting the SMIL files [1] --- the user should not want to change them. The other MO metadata, which a user might be interested in editing, are the optional: media:narrator (0 or more), media:active-class (0 or 1) and media:active-playback-class (0 or 1). Instead of creating a specialized MO meta GUI, you might want to let the user take advantage of the generic "epub3 metdata gui editor".
B. python code to convert TTML to VTT => if you give me the interface (function signature, with I/O types) you like, I can code that.
C. if by "fixed format" you mean "fixed layout" rendition (FXL), I would suggest thinking 7 times 7 about supporting FXL in Sigil, for two reasons: a. the avg FXL creator wants InDesign-like UI and tools (and I think that would cost you a lot of work); b. Sigil has been great because it has focussed on reflowable books. I had a similar experience when adding FXL support to my reading app Menestrello, which started as reflow-only: FXL is half-backed and I still regret wasting time on it. Of course, just my 2 cents.
[1] the general case is more complex (surprise surprise!), since a SMIL file might omit clipEnd time(s), and one should look at the duration of the physical audio file(s)... but, as a first approximation, one can compute the duration of a SMIL overlay as the (sum over all the different audio files referenced in the SMIL file of) clipEnd value of the last <audio> element minus the clipBegin value of the first <audio> element.
JonathanMagus
commented
8 years ago Hi Kevin, Alberto
All of this sounds good.
Except that ePUB3 involves potentially much more complex mark-up and structural semantics than ePUB2 and although it would be possible to create this in a toggled editor to take back into Sigil, if it did not rewrite it of course, I cannot see Sigil becoming useful enough on its own unless it is either set up with suitable toggled features permanently incorporated, or has extended ‘Insert’ or ‘Tools’ options that cover this.
Maybe the cascading, categorised, hierarchy of ‘snippets’ that I have for my own use could be used for this, or perhaps made into an enabling plugin? I am happy to provide a copy of the XML that generates the input boxes if you want to consider this.
Taking up Alberto’s points about fixed layout—yes in a sense I agree that fixed layout ePUB3 is an unfortunate afterthought forced on the IDPF by an entrenched publishing industry and embraced by print production skillsets and mentalities, which should not be encouraged.
However, the facts are that fixed layout ePUB3 exists, does in fact suit some genres; and may continue to play a part in hybrid versions of ePUB3 output that are already possible in theory (where there are alternative flowing and fixed versions in the same wrapper, or some pages are fixed and others are not depending on content). So, it makes sense for Sigil to recognise it.
That does not necessarily mean facilitating the same sort of WYSIWYG processes employed by InDesign for the print based and the ‘mark-up shy’ though. Fixed layout may also be arrived at simply by fixing the styles and dimensions of page-sized chunks of otherwise flowing content. In other words it can be created in more direct ways and should still display in Sigil’s viewing panel, albeit with some scrolling necessary. It is essentially just content with alternative CSS styling for a different viewing option.
Including some optional templates in Sigil for different types of ePUB3 might help make these options available to all its users—which I would also be happy to input to.
My understanding of the Readium licencing is that it would allow its incorporation into Sigil as Open Source within Open Source, subject to required acknowledgements. I can check on that if necessary.
Also, responding to an earlier question. I do have Mac OS X, though only have experience of installing Sigil as a DMG and typically use Sigil on Windows, installing from an _x64.exe. However, I am willing to test the preview if you can provide one or both of these.
Regards
Jonathan
kevinhendricks
commented
8 years ago Hi Jonathan, Thanks for your insights. As I tried to explain before, moving things on import to Sigil does not actually impose a burden, that wouldn't exist otherwise. Even if everything was untouched on import, and the user decided to simply rename an xhtml file, all of the files that reference that xhtml file must have their link to it automatically updated. Otherwise, Sigil is doing nothing for you. As it stands now, if a user renames an xhtml file referenced in other xhtml files, content.opf, ncx entries, nav entries, and now page-map entries and smil entries are all autoupdated.
To do that in xml like the page-map.xml and .smil files, I have added a base xml resource that uses lxml to parse and update "links" of any sort. If you want these external containers apps to be worth anything, you either have to know what "links" out and in they have and how to update them. If not, a simple rename would break them. That is just silly. So as long as these container apps are xml based and have tags or attributes that tie them to underlying xhtml files, then the ability to update those links by Sigil is needed. Without it, you might as well use any standalone text/xml editor to make all of your own changes.
This ability is the exact same ability used to relocate files to standard folders. So nothing is really gained by not relocating things, if as a result a simple rename breaks things.
So my focus is on understanding those "links" in and out and being able to manage them in some sane manner.
Take care,
Kevin
Sent from my iPad
On Jan 18, 2016, at 2:09 PM, JonathanMagus notifications@github.com wrote:
Hi Kevin, Alberto
All of this sounds good.
Except that ePUB3 involves potentially much more complex mark-up and structural semantics than ePUB2 and although it would be possible to create this in a toggled editor to take back into Sigil, if it did not rewrite it of course, I cannot see Sigil becoming useful enough on its own unless it is either set up with suitable toggled features permanently incorporated, or has extended ‘Insert’ or ‘Tools’ options that cover this.
Maybe the cascading, categorised, hierarchy of ‘snippets’ that I have for my own use could be used for this, or perhaps made into an enabling plugin? I am happy to provide a copy of the XML that generates the input boxes if you want to consider this.
Taking up Alberto’s points about fixed layout—yes in a sense I agree that fixed layout ePUB3 is an unfortunate afterthought forced on the IDPF by an entrenched publishing industry and embraced by print production skillsets and mentalities, which should not be encouraged.
However, the facts are that fixed layout ePUB3 exists, does in fact suit some genres; and may continue to play a part in hybrid versions of ePUB3 output that are already possible in theory (where there are alternative flowing and fixed versions in the same wrapper, or some pages are fixed and others are not depending on content). So, it makes sense for Sigil to recognise it.
That does not necessarily mean facilitating the same sort of WYSIWYG processes employed by InDesign for the print based and the ‘mark-up shy’ though. Fixed layout may also be arrived at simply by fixing the styles and dimensions of page-sized chunks of otherwise flowing content. In other words it can be created in more direct ways and should still display in Sigil’s viewing panel, albeit with some scrolling necessary. It is essentially just content with alternative CSS styling for a different viewing option.
Including some optional templates in Sigil for different types of ePUB3 might help make these options available to all its users—which I would also be happy to input to.
My understanding of the Readium licencing is that it would allow its incorporation into Sigil as Open Source within Open Source, subject to required acknowledgements. I can check on that if necessary.
Also, responding to an earlier question. I do have Mac OS X, though only have experience of installing Sigil as a DMG and typically use Sigil on Windows, installing from an _x64.exe. However, I am willing to test the preview if you can provide one or both of these.
Regards
Jonathan
— Reply to this email directly or view it on GitHub.
JonathanMagus
commented
8 years ago Hi Kevin
There seems to be a misunderstanding. That's not what I was saying on this occasion.
My main point was this:
ePUB3 differs substantially from ePUB2 in its use of semantic mark-up. There is a lot of it in ePUB3, which defines the structure and purpose of the content in detail. The full potential of ePUB3 for eTextbooks and graphic novels for example depends on this (see the specifications covering EDUPUB and Comics). It is also relevant for complex works and potentially important for accessibility.
For Sigil to be a viable authoring and editing tool for ePUB3 there has to be a convenient way of adding this contextual mark-up (a lot of it being specific epub:type=‟” attributes added to particular tags) as valid drop-in components. Without that it isn't really advancing its capabilities much beyond an enhanced version of ePUB2; and may in practice have less functionality than the ‘standalone text/xml editor’, which you use as an analogy, for ePUB3 creation, despite its ability to auto correct and effectively manage other features. Sigil is not a very good text/xml editor on its own.
For the advanced user it is time saving to have this feature. For the beginner it is enabling. (Compare the drop-down menus in Blue Griffon ePUB Edition for example.)
Unless I am misunderstanding it, the proposed development roadmap doesn't include anything that adds this capability directly.
Yes Sigil has to recognise it as valid, which the implication of your original post about version 0.9.3 suggests is now going to be the case, however, first the user has to be able to put it there.
The XML I was referring to adds this as an expandable menu to another piece of Open Source software—a toggled mark-up editor, which responds with drop-in content or input boxes that facilitate the appropriate ePUB3 semantic mark-up. I was thinking that might be a starting point, or useful reference, for adding something similar to Sigil.
Others could do the same as me of course independently, or add content of their choice to the 'Clip editor', however, that would not be beefing up Sigil itself for the benefit of all its users.
This has nothing to do, for the most part, with links or relocating things. It is just about making ePUB3 mark-up easier to create (semantic XHTML5) in Sigil.
I do understand that Sigil's ability to correct things in the background has advantages that are worth preserving, however, its inability to do certain other things at all are also obstacles in the way of its development that ought to be considered equally important.
I am also guessing (just guessing) that it might be an easy win for the roadmap too, which (again I may be misunderstanding) seems more focused on aspects of the ePUB container's elements than the requirements of the content itself.
Or, if version 0.9.3 recognises epub:type and leaves it alone, then Sigil could be moved towards an ePUB focused IDE by incorporating the better editor too.
Hopefully this clarifies my previous comments.
Regards
Jonathan
kevinhendricks
commented
8 years ago Hi Jonathon,
The entire internals of Sigil have been changed since the Sigil 0.9.X series started. No Tidy. So no removing unknown epub3 things like epub:type attributes and the like. In fact, a true html5 compliant repair parser called gumbo is now used. So keeping epub semantics like epub:type attributes is now not a problem.
As for adding them, the user can do this in CodeView now and if it seems like there is enough interest, we can expand Sigil's gui to make a selected vocabulary available to insert attributes into a tag easily. This is not a serious problem, and really has nothing to do with your bug report of "repositioning things causes errors".
My earlier reply was directed toward your bug report. I have been playing with your Travel epub3. It will now load without getting hacked to pieces by the gumbo parser. That said, your javascripts expect hard coded paths for resources. For example, your Galleria javascript could easily be made to find its own stylesheet. The paths in it are hard-coded now (ie. the stylesheet needs to be next to the galleria.classicmin.js file). With just two quick edits, the Galleria stuff will now work.
If there are more tools like Galleria that you would like to use with your epub, we either have to teach Sigil how to update links inside javascripts (quite doable) or we create a Sigil plugin tool that inserts the container app into Sigil and it understands how to update itself. Either is very doable.
So we seems to be speaking across each other. I was trying to respond to your original bug report.
Thanks,
Kevin
On Jan 19, 2016, at 09:29 AM, JonathanMagus notifications@github.com wrote:
Hi Kevin There seems to be a misunderstanding. That's not what I was saying on this occasion. My main point was this: ePUB3 differs substantially from ePUB2 in its use of semantic mark-up. There is a lot of it in ePUB3, which defines the structure and purpose of the content in detail. The full potential of ePUB3 for eTextbooks and graphic novels for example depends on this (see the specifications covering EDUPUB and Comics). It is also relevant for complex works and potentially important for accessibility. For Sigil to be a viable authoring and editing tool for ePUB3 there has to be a convenient way of adding this contextual mark-up (a lot of it being specific epub:type=‟” attributes added to particular tags) as valid drop-in components. Without that it isn't really advancing its capabilities much beyond an enhanced version of ePUB2; and may in practice have less functionality than the ‘standalone text/xml editor’, which you use as an analogy, for ePUB3 creation, despite its ability to auto correct and effectively manage other features. Sigil is not a very good text/xml editor on its own. For the advanced user it is time saving to have this feature. For the beginner it is enabling. (Compare the drop-down menus in Blue Griffon ePUB Edition for example.) Unless I am misunderstanding it, the proposed development roadmap doesn't include anything that adds this capability directly. Yes Sigil has to recognise it as valid, which the implication of your original post about version 0.9.3 suggests is now going to be the case, however, first the user has to be able to put it there. The XML I was referring to adds this as an expandable menu to another piece of Open Source software—a toggled mark-up editor, which responds with drop-in content or input boxes that facilitate the appropriate ePUB3 semantic mark-up. I was thinking that might be a starting point, or useful reference, for adding something similar to Sigil. Others could do the same as me of course independently, or add content of their choice to the 'Clip editor', however, that would not be beefing up Sigil itself for the benefit of all its users. This has nothing to do, for the most part, with links or relocating things. It is just about making ePUB3 mark-up easier to create (semantic XHTML5) in Sigil. I do understand that Sigil's ability to correct things in the background has advantages that are worth preserving, however, its inability to do certain other things at all are also obstacles in the way of its development that ought to be considered equally important. I am also guessing (just guessing) that it might be an easy win for the roadmap too, which (again I may be misunderstanding) seems more focused on aspects of the ePUB container's elements than the requirements of the content itself. Or, if version 0.9.3 recognises epub:type and leaves it alone, then Sigil could be moved towards an ePUB focused IDE by incorporating the better editor too. Hopefully this clarifies my previous comments. Regards Jonathan — Reply to this email directly or view it on GitHub.
JonathanMagus
commented
8 years ago Hi Kevin
Yes, I was talking about the development roadmap that was being discussed in the thread, not the original topic—although they interrelate of course.
I am sending you a separate follow up email on this to avoid any further confusion.
Regards
Jonathan
JonathanMagus
commented
8 years ago Sent with attachment and links.
kevinhendricks
commented
5 years ago I realize this is an old issue, but FYI Sigil now has a "nomove" branch that is almost ready for an alpha release. This branch stops updating and moving things inside the epub. Instead it has a Restructure Epub tool that will standardize the epub only if the epub dev wants. The changes have been quite invasive so a lot of testing will be needed. That said, I hope within 1 to 3 months to finally be able to close this issue.
kevinhendricks
commented
4 years ago Closing this issue as it is now fixed in master.
This latest version's ability to open a fairly complex ePUB3 file and allow further work on it, is certainly a big improvement; however, Sigil still appears to correct and re-write mark-up that it shouldn't.
For example it insists on re-writing the correct XHTML5 <!DOCTYPE html> with the wrong ePUB2 version of the tag thus introducing a mass of errors. It also queries and attempts to correct XML files that are present as part of the interactivity, not as content pages.
Presumably some of this might be put back as it was originally if output back to ePUB3 using the ePUB3-itizer plugin, although it would be much better if Sigil did not mess with content that is correctly marked-up for ePUB3 in the first place.
Errors are also introduced when Sigil forces file types and folder patterns to match its own preconceptions. So for example a .woff font gets moved from the 'Fonts' folder and put in a 'Misc' file, thereby breaking the link to it from the style sheet.
Perhaps the 'Misc' folder is a temporary display; however, it really isn't helpful to have a logical folder structure that is ePUB3 compliant, which might for example contain a 'Scripts' folder, replaced with a 'one size fits all' representation that shows components in the wrong place.
Sigil will not become an ePUB3 editor until it recognises valid ePUB3 and stops trying to turn it into hybridised ePUB2.
All that said, there are big advances and improvements here.
Thanks.