Closed javagl closed 2 years ago
javagl
commented
2 years ago The current state is that it contains a specification/Specification.adoc that was largely auto-generated from the specification/README.md, and that can be used to create first, basic (not-so-pretty) PDF and HTML outputs. It also contains a BUILDING.md with some instructions for this process.
javagl
commented
2 years ago An update and some sort of checklist
The conversion of the "core" of the specification to AsciiDoc is largely done. The JSON schema property references are still missing.
It is possible to generate the single-page HTML version of the specification. And it is possible to generate the single-document PDF version of the specification.
What is still to be done:
Formatting tweaks This refers to things like table column widths or image sizes, but nothing big.
Fixing some links Most of the links should be fixed via the last ~40 commits or so. The cross-linking between documents required some workaround, described in this state of the instructions. Suggestions for better solutions are welcome. Systematically going through all documents and checking all links could be tedious, but doable. For Markdown, I once created a small tool that actually checks all the cross-links, and reports them as in
Error in file:/C:/3d-tiles/specification/README.md
Not valid : file:/C:/3d-tiles/specification/README.md#property-reference
Anchors are [3d-tiles-format-specification, contents, introduction, file-extensions-and-media-types, json-encoding, uris, units, coordinate-reference-system-crs, concepts, tiles, tile-content, geometric-error, refinement, replacement, additive, bounding-volumes, region, box, sphere, content-bounding-volume, extensions, viewer-request-volume, transforms, tile-transforms, gltf-transforms, gltf-node-hierarchy, y-up-to-z-up, example, implementation-example, tile-json, tileset-json, external-tilesets, bounding-volume-spatial-coherence, spatial-data-structures, quadtrees, k-d-trees, octrees, grids, implicit-tiling, metadata, metadata-schema, assigning-metadata, metadata-statistics, specifying-extensions-and-application-specific-extras, extensions-1, extras, tile-format-specifications, declarative-styling-specification, license] for file:/C:/3d-tiles/specification/README.md
LinkInfo:
text : Property reference
destination : #property-reference
destinationUri : #property-reference
scheme : null
fullUriWithFragment : file:/C:/3d-tiles/specification/README.md#property-reference
fullUriWithoutFragment: file:/C:/3d-tiles/specification/README.md
fragment : property-reference
fileName : and I'll consider generalizing this for AscciDoc, but it's hard to make promises about the timeline here, so the manual approach may be more predictable.
Reducing the PDF size The single PDF has a whopping 45MB right now, so it should be compressed. Unfortunately, the common tools that seem to be used for this task cause all internal links of the document to break. For details, refer to https://bugs.ghostscript.com/show_bug.cgi?id=699830 . In doubt: Batch-processing all images and converting them to JPG with a limited size and maybe 70% quality should already reduce the size significantly
As usual, the "core" was the 80% of the work that took 20% of the time. The JSON schema property reference is supposed to be generated with wetzel, and quite some work will have to go into that.
A short summary of what has to be done:
class.schema.json properties, it is important to include this information (namely, that they have to be class.property.schema.json). This is addressed in https://github.com/CesiumGS/wetzel/pull/72 , but this is only a draft, and has to be compared to or aligned with https://github.com/CesiumGS/wetzel/pull/68 tile refers to tile (via its children). This is addressed in https://github.com/CesiumGS/wetzel/pull/74 , but this will require a deeper restructuring of wetzel (details in the issue)#anchors-for-cross-linking, but these have to be disambiguated for the case that multiple property references should end up in a single HTML document. (This is partially addressed in https://github.com/CesiumGS/wetzel/pull/75 , but far from done). Also, there usually are "table of contents" inserted in the references, and these will have to be flattened and/or cross-linked for 3D Tiles. The path forward here is not yet clear.If When the property references can be created as individual AsciiDoc files, then it should be easy to integrate them here. A first experiment was done in https://github.com/javagl/3d-tiles/commit/226a5bf00c618b6dcdfc66f2f1ea6fcc8892ef3a (not yet part of this PR), and there was no obvious blocker for this part until now.
javagl
commented
2 years ago To elaborate on one aspect of the Linking and structure point mentioned above:
wetzel inserts links from the "Property Reference" to the JSON schema. For example, see glTF:
(An aside: Note that the Property Reference is normative, while the JSON schema is not normative!)
I assume that a similar solution should also be found for 3D Tiles. On a very basic level, this already works:
This cross-linking takes place has some caveats. In the current state, wetzel generates this by writing out include::schema/example.schema.json statements into the AsciiDoc document. Actually resolving that (when generating the HTML) is left to AsciiDoc itself. The directory - schema in the example - is configured via a command-line parameter. So does not work out-of-the-box when the schema directory could actually be schema/TileFormats. For now, I took the approach of actually inlining the schema files into the AsciiDoc document. But this has to be re-evaluated at some point.
On top of that, all this raises a bunch of questions regarding links:
rootProperty? (Or to put it that way: Where exactly should extras and extensions appear in the HTML output?).adoc files for the HTML generation and for hosting on GitHub. In one case, the link anchors are local ones (inside the same document). In the other case, they are actual links to external files (in complicated, relative directories...)But I'll try to sort all that out...
javagl
commented
2 years ago An update with the latest state:
Most of the recent changes have been link fixes (details in previous comments). One goal was to make sure that links work (in general... and) on GitHub as well as the PDF/HTML output. For example, one can now browse at https://github.com/javagl/3d-tiles/blob/asciidoc-conversion/specification/README.adoc and click the first link to glTF Tile Format, and actually end up at the right page, and this link still works in the single-page HTML file or the PDF document.
(It will take a few iterations until all broken links are weeded out, but most of them should already work...)
A large refactoring of wetzel is in progress. In the current form, it has so many issues with the 3D Tiles spec that the tl;dr can only be: "It does not work (period)". Some early experiments with that less-then-ideal wetzel output are in https://github.com/javagl/3d-tiles/tree/asciidoc-conversion-experiments , but more work will have to go into that.
The current state of the PDF is attached here. Note that this is a compressed version, with low-resolution images. (The original one has 45MB, because of the large PNG files. Compressing that PDF is a pain in the back, and requires a carefully selected version of GhostScript - details are given in the build instructions)
javagl
commented
2 years ago An updated state of the specification is attached here. This time with a Property Reference and a JSON Schema Refrence section.
Some notes about the property reference and wetzel:
The property reference is based on a local version of wetzel that does not have much in common with what is currently in the wetzel repository (c.f. https://en.wikipedia.org/wiki/Ship_of_Theseus ). If this is supposed to ever be made public, I'll have to do lots of cleanups, but even then, it's not unlikely that the changes are not considered to be "improvements", and therefore, might never be merged.
At some points, the 3D Tiles spec is already hitting a limit of which sort of documentation can be auto-generated from the schema. While most the browsing/linking should currently work - for example, linking from b3dm.featureTable.BATCH_LENGTH to the type featureTable-definitions-globalPropertyInteger - there are types that simply have no names, and thus, can hardly be squeezed into a form that does not boil down to a plain ADOC/MD version of the JSON schema itself. We might even consider changing the schema if we wanted to improve browsability here.
javagl
commented
2 years ago I'll mark this as "ready for review" now. There certainly are still some issues (probably some links not working), or possible improvements (as in ~"That's not how you do this in AsciiDoc!"), but it should currently be a reasonably consistent state.
A summary of some possibly relevant points:
README.adoc file for each former README.md
.adoc file by default, For example, when looking at the directory https://github.com/javagl/3d-tiles/tree/asciidoc-conversion/specification/TileFormats/PointCloud , then it shows the AsciiDoc file. Specification.adoc. It includes all README.adoc files, as well as the APPENDIX.adoc and the "properties reference" that is generated with wetzel, and contained in PropertiesReference_3dtiles.adoc and PropertiesReference_3dtiles_schema.adocPreviews of the resulting specifications are attached here.
The PDF file was compressed (as described in the build instructions). For the HTML version, I downscaled/compressed the images, but the HTML file can just be dropped into the local 3dtiles/specification subdirectory to see it with the original images. (I didn't want to upload 90MB here, due to a slow connection).
javagl
commented
2 years ago I wrote some comments to the inlined ones, and summarize them as a short TODO list here:
[X] Consider adding a title image/text with ifdef::env-github[] into the main README.adoc (right now, these are only in Specification.adoc)
[x] Look at the CSS to reduce the spacing between bullet point lists and table rows
[x] Check whether 'Acknowledgements' should be a section or appendix. Investigate AsciiDoc acknowledgments section type
[ ] Check if the properties reference can be sensibly "grouped" to be more similar to the folder structure
[X] MAYBE move "Metadata Semantic Reference" to its own (non-normative?) section. Also see https://github.com/CesiumGS/3d-tiles/issues/643
[X] Investigate why the 3D Tiles logo was small in some versions of the PDF. (Maybe just an artifact from of a missing pdfwidth and some DPI changes due to the compression or so...)
[ ] Figure out what causes the warning WARNING: Could not locate the character ... (\u2063) ... when creating the PDF
[X] Experiment with the PDF compression modes. screen is too aggressive and makes images unreadable. ebook (the one that the preview PDF was created from) seems OK. Check the lower-level options for limiting image sizes and quality.
ebook, which seems to be OK for most uses, and added a link to https://www.ghostscript.com/doc/9.54.0/VectorDevices.htm#distillerparams if further tweaks are required.[x] OT: Replace "MIME type" with "Media Type"
javagl
commented
2 years ago An aside: I added the logo and an introductory sentence to the (GitHub-rendered version of the) README.adoc that is shown at https://github.com/javagl/3d-tiles/tree/asciidoc-conversion/specification . But we might consider to create a dedicated README.md for this directory. The current README.adoc that contains the actual specification text could then be something like MAIN_SPECIFICATION.adoc or so.
This new "top-level" README.md could then be something like
Hi there, look, all this is ADOC now, here's the entry point for 3D Tiles 1.1: MAIN_SPECIFICATION.md
Here are some PDF files: Specification-1.1.0.pdf ...
...
so that this directory/README.md rather serves as "the entry point" for all spec-related documents (similar to https://github.com/KhronosGroup/glTF/tree/main/specification )
(Not a strong opinion, just a thought...)
lilleyse
commented
2 years ago This new "top-level" README.md could then be something like
Eventually this might make sense, but it is another level of indirection that I don't think is worth it right now.
javagl
commented
2 years ago The latest commits remove the now obsolete .md files, and update the links that had been pointing directly to these Markdown files from other parts of the repository. (This was mainly from the former 'next' extensions).
One thing that I remembered while doing this: Some of the Markdown documents contained a local Table Of Contents (e.g. the Metadata/README.md). These TOCs can not directly be part of the AsciiDoc files when they are supposed to be combined into a single PDF/HTML. So the Metadata/README.adoc does not contain a TOC.
Using the ifdef::env-github[] construct, we could insert a TOC that only would show up on GitHub. This may impose some maintenance effort (because these TOCs cannot be generated automatically, as it was done with the MarkdownAllInOne VS plugin). But if this is desired, I could add these TOCs. (They had all been removed in a single commit: https://github.com/javagl/3d-tiles/commit/3d4971495e4219572d50b02b799ef942cd7fc37a )
lilleyse
commented
2 years ago Let's skip the TOC for now.
lilleyse
commented
2 years ago I noticed a lot of changes when I generated the property reference with wetzel locally. Does the property reference need to be re-generated now that https://github.com/CesiumGS/3d-tiles/pull/688 is merged?
javagl
commented
2 years ago Indeed, the state here was generated based on a state without BinaryBodyOffset. Updated it with the latest state.
lilleyse
commented
2 years ago Thanks @javagl! A huge amount of work went into this and I'm glad to see it merged.
A DRAFT of the first experiments for converting the specification to AsciiDoc (see https://github.com/CesiumGS/3d-tiles/issues/590 )