search

metanorma / metanorma-plateau

Metanorma for Project PLATEAU by MLIT
Other
2 stars 0 forks source link

(URGENT) PDF output on GitHub Actions are broken due to old version of mn2pdf #121

Open ronaldtse opened 6 days ago

ronaldtse commented 6 days ago

This is reported by @ReesePlews and confirmed by myself.

The GHA generated PDFs are currently malformed, they cannot be open in Preview or Adobe Acrobat. However, when locally generated, the PDFs are fine.

I suspect this has to do with the mn2pdf changes lately (there was a ticket about locking mn2pdf to version 1.99), and the Docker container may have used the old version of mn2pdf that does not support the latest XML changes (@opoudjis said it might be to do with the new TOC encoding).

In the metanorma-plateau action, I can see mn2pdf being updated to 2.06 from a Docker version of 1.99:

In the mn-samples-plateau action, the metanorma version command does not show mn2pdf version (which it should!), there is no upgrade in the mn2pdf gem mentioned, so it means the Docker default of mn2pdf 1.99 is still used:

This means we need to re-build the Docker container.

@opoudjis @Intelligent2013 can you tell me what version of mn2pdf is required (and are the components released to indicate this)? Will a re-release of a new metanorma-cli version fix this?

opoudjis commented 6 days ago

metanorma gem v2.0.7 was released with spec.add_runtime_dependency "mn2pdf", "~> 2 on Monday. This is nothing to do with metanorma-cli, which grabs the latest version of metanorma. A rerelease of metanorma-cli will not make any difference, that's been done on Monday. My local environment has mn2pdf 2.06:

19:40:08 * ~/Documents/Arbeit/upwork/ribose/mn-samples-jis[main] $ bundle update
Fetching gem metadata from https://rubygems.org/..............
Resolving dependencies...
Fetching date 3.4.0 (was 3.3.4)
Using json 2.7.5 (was 2.7.4)
Fetching webrick 1.9.0 (was 1.8.2)
Using mn2pdf 2.06 (was 1.99)
Fetching shale 1.2.0 (was 1.1.0)
Using pubid-core 1.12.10 (was 1.12.9)
Using lutaml-model 0.3.19 (was 0.3.15)
Fetching graphql 2.4.0 (was 2.3.19)
Using activesupport 7.2.2 (was 7.2.1.2)
Using relaton-iso-bib 1.19.2 (was 1.19.1)
Using relaton-render 0.7.10 (was 0.7.9)
Using isodoc 2.12.0 (was 2.11.4)
Installing date 3.4.0 (was 3.3.4) with native extensions
Installing webrick 1.9.0 (was 1.8.2)
Installing shale 1.2.0 (was 1.1.0)
Using metanorma 2.0.7 (was 2.0.6)
Using lutaml 0.9.20 (was 0.9.19)
Installing graphql 2.4.0 (was 2.3.19)
Using metanorma-standoc 2.9.10 (was 2.9.9)
Using metanorma-iso 2.8.9 (was 2.8.8)
Using metanorma-ieee 1.3.3 (was 1.3.2)
Using metanorma-itu 2.5.6 (was 2.5.5)
Using metanorma-ogc 2.6.5 (was 2.6.4)
Using metanorma-cc 2.5.5 (was 2.5.4)
Using metanorma-csa 2.5.5 (was 2.5.4)
Using metanorma-iho 1.0.6 (was 1.0.5)
Using metanorma-bipm 2.5.5 (was 2.5.4)
Using metanorma-iec 2.5.6 (was 2.5.5)
Using metanorma-jis 0.3.7 (was 0.3.6)
Using metanorma-plateau 0.1.9 (was 0.1.8)
Using metanorma-cli 1.10.11 (was 1.10.10)

There is nothing further for me to do here, this is a Docker concern now.

opoudjis commented 6 days ago

In the mn-samples-plateau action, the metanorma version command does not show mn2pdf version (which it should!),

You've just seen that I've updated my mn-samples-plateau locally, and it updated just fine.

Metanorma has 250 gem dependencies, and cat Gemfile.lock exists. If metanorma is going to need to list mn2pdf and relaton and lutaml-model and plurimath and Liquid and asciidoctor in its version info, then the problem isn't metanorma (what other software dumps out 50 gem versions when you ask it for a version), the problem is dependency control.

And this time around, that is not a problem internal to metanorma-cli.

Intelligent2013 commented 5 days ago

I've downloaded the artifacts from https://github.com/metanorma/mn-samples-plateau/actions/runs/11611146598. Both PDFs generated by mn2pdf v2.06 (version is ok):

/Producer (Ribose Metanorma mn2pdf version 2.06)
/CreationDate (D:20241031114446Z)

The PDF doesn't end with %%EOF. It means the generation interrupted by some reason. I'll investigate it.

Intelligent2013 commented 5 days ago

Locally generated PDF broken also. In .pdf.err log there are messages like this:

SVG error: <unknown>:
The attribute "style" represents an invalid CSS declaration ("fill:#737A7A;stroke:#737A7A;stroke-width:6.250000e-02;stroke-miterlimit:10;").
Original message:
The "stroke-width" property does not support dimension values.
org.w3c.dom.DOMException: <unknown>:
The attribute "style" represents an invalid CSS declaration ("fill:#737A7A;stroke:#737A7A;stroke-width:6.250000e-02;stroke-miterlimit:10;").
Original message:
The "stroke-width" property does not support dimension values.
    at org.apache.batik.css.engine.CSSEngine.getCascadedStyleMap(CSSEngine.java:824)
    at org.apache.batik.css.engine.CSSEngine.getComputedStyle(CSSEngine.java:867)
    at org.apache.batik.bridge.CSSUtilities.getComputedStyle(CSSUtilities.java:81)
    at org.apache.batik.bridge.CSSUtilities.convertDisplay(CSSUtilities.java:563)
    at org.apache.batik.bridge.GVTBuilder.buildGraphicsNode(GVTBuilder.java:206)
    at org.apache.batik.bridge.GVTBuilder.buildComposite(GVTBuilder.java:171)
    at org.apache.batik.bridge.GVTBuilder.buildGraphicsNode(GVTBuilder.java:219)
    at org.apache.batik.bridge.GVTBuilder.buildComposite(GVTBuilder.java:171)
    at org.apache.batik.bridge.GVTBuilder.build(GVTBuilder.java:82)
    at org.apache.fop.render.pdf.PDFImageHandlerSVG.handleImage(PDFImageHandlerSVG.java:104)
    at org.apache.fop.render.intermediate.AbstractIFPainter.drawImage(AbstractIFPainter.java:255)
    at org.apache.fop.render.intermediate.AbstractIFPainter.drawImage(AbstractIFPainter.java:211)
    at org.apache.fop.render.intermediate.AbstractIFPainter.drawImageUsingImageHandler(AbstractIFPainter.java:174)
    at org.apache.fop.render.intermediate.AbstractIFPainter.drawImageUsingDocument(AbstractIFPainter.java:325)
    at org.apache.fop.render.pdf.PDFPainter.drawImage(PDFPainter.java:291)
    at org.apache.fop.render.intermediate.IFParser$Handler$ImageHandler.endElement(IFParser.java:799)
    at org.apache.fop.render.intermediate.IFParser$Handler.endElement(IFParser.java:413)
    at org.apache.xalan.transformer.TransformerIdentityImpl.endElement(TransformerIdentityImpl.java:1149)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:610)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1784)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2969)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:113)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:507)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:867)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:796)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:142)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1216)
    at org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:519)
    at org.metanorma.fop.PDFGenerator.runFOP(PDFGenerator.java:656)
    at org.metanorma.fop.PDFGenerator.convertmn2pdf(PDFGenerator.java:518)
    at org.metanorma.fop.PDFGenerator.process(PDFGenerator.java:326)
    at org.metanorma.fop.mn2pdf.main(mn2pdf.java:350)

May be something wrong in the SVG graphic. I'll check.

Intelligent2013 commented 5 days ago

Something wrong in the image 030.svg (may be there are another SVGs).

Just for experiment I've removed all content except one line:

<text transform="matrix(0.85 0 0 1 450.376 1281.3411)" class="st3 st4">«FeatureType»</text>

and PDF is broken. I've changed the text «FeatureType» to 区域モデル:

<text transform="matrix(0.85 0 0 1 450.376 1281.3411)" class="st3 st4">区域モデル</text>

and PDF is OK.

One letter - PDF is broken:

<text transform="matrix(0.85 0 0 1 450.376 1281.3411)" class="st3 st4">A</text>

No error, no warnings... I'll think...

Intelligent2013 commented 5 days ago

Complicated issue... The issue relates to the missing fonts, specified in SVG:

.st3{font-family:'YuGothicUI-Regular';}
...
.st5{font-family:'YuGothicUI-Bold';}

mn2pdf replaces them to Noto Sans fonts:

WARNING: Font 'YuGothicUI-Regular' (font name 'YuGothicUI-Regular', font style 'normal', font weight 'normal') doesn't exist. Replaced by 'C:\Users\TestUser\.fontist\fonts\NotoSans-Regular.ttf'.
WARNING: Font 'YuGothicUI-Bold' (font name 'YuGothicUI-Bold', font style 'normal', font weight 'bold') doesn't exist. Replaced by 'C:\Users\TestUser\.fontist\fonts\NotoSans-Bold.ttf'.

but Batik API doesn't see them for some reason (the issue relates to https://github.com/metanorma/mn2pdf/issues/248)

In case of missing fonts, Apache FOP can't generated PDF in the PDF/UA mode, therefore mn2pdf turns off the PDF/UA in this case. But PDF/A-3a mode is still active. Therefore PDF is broken. I'll add the disabling PDF/A-3a mode too.

The latest mn2pdf version without PDF/A-3a mode is 1.99, therefore the PDF generated in those version is correct.

Intelligent2013 commented 4 days ago

The issue fixed in mn2pdf https://github.com/metanorma/mn2pdf/releases/tag/v2.08. But the docker image contains v2.06.

@CAMOBAP in PR https://github.com/metanorma/mn-samples-plateau/pull/224 I've added uses: metanorma/ci/docker-gem-install@main to retrieve the latest gem from Gemfile( mn2pdf gem v2.08). The process fails (https://github.com/metanorma/mn-samples-plateau/actions/runs/11644366019/job/32426198109):

> bundle add mn2pdf
Fetching gem metadata from [https://rubygems.pkg.github.com/metanorma/..](https://rubygems.pkg.github.com/)
Fetching gem metadata from https://rubygems.org/........
Resolving dependencies...
Fetching gem metadata from [https://rubygems.pkg.github.com/metanorma/..](https://rubygems.pkg.github.com/)
Fetching gem metadata from https://rubygems.org/........
Resolving dependencies...
Fetching gem metadata from [https://rubygems.pkg.github.com/metanorma/..](https://rubygems.pkg.github.com/)
Fetching gem metadata from https://rubygems.org/..............
Could not find gem 'metanorma-plateau (~> 0.1.9)' in rubygems repository
[https://rubygems.pkg.github.com/metanorma/.](https://rubygems.pkg.github.com/metanorma/)

The source contains the following gems matching 'metanorma-plateau':
  * metanorma-plateau-0.0.2
  * metanorma-plateau-0.0.3
  * metanorma-plateau-0.0.4
  * metanorma-plateau-0.0.5
  * metanorma-plateau-0.0.6
  * metanorma-plateau-0.0.7
  * metanorma-plateau-0.0.8
  * metanorma-plateau-0.1.0
  * metanorma-plateau-0.1.1

I don't understand why it tries to install metanorma-plateau.

I've tried to add metanorma-plateau additionally (with updated lib\metanorma\plateau\version.rb )

gem "metanorma-plateau", git: "https://github.com/metanorma/metanorma-plateau", branch: "use_in_docker"

The error (https://github.com/metanorma/mn-samples-plateau/actions/runs/11644447927/job/32426368266?pr=224);

> bundle add metanorma-plateau --git https://github.com/metanorma/metanorma-plateau --branch use_in_docker
[!] There was an error parsing `injected gems`: You cannot specify the same gem twice with different version requirements.
You specified: metanorma-plateau (~> 0.1.9) and metanorma-plateau (>= 0). Gem already added. Bundler cannot continue.
 #  from /__w/mn-samples-plateau/mn-samples-plateau/injected gems:1
 #  -------------------------------------------
 >  gem "metanorma-plateau", ">= 0", :git => "https://github.com/metanorma/metanorma-plateau", :branch => "use_in_docker"
 #  -------------------------------------------

Could you take a look?

ronaldtse commented 3 days ago

The metanorma version command is to allow users to know the versions of the Metanorma stack.

and cat Gemfile.lock exists

Not for typical users of Metanorma, who mostly use the single binary version, who have no access to Gemfile.lock.

Anyway, this is a degree issue. There is an argument for the command to "show no version", "show some versions", or "show all versions".

Right now we show the standoc version, and it's not a flavor. mn2pdf is part of the core Metanorma stack, just like a flavor is.

ReesePlews commented 3 days ago

just a comment about printing versions in the github actions when a document is generated:

in a couple of projects the github action prints the version of metanorma and related flavors. this has been helpful when we submit comments related to metanorma and a specific template.

not sure if you are considering to remove or change this, but i think it is very important to retain this version information when the github action is run. no reply is needed. thank you.

opoudjis commented 3 days ago

Right now we show the standoc version, and it's not a flavor. mn2pdf is part of the core Metanorma stack, just like a flavor is.

Which other gems do you regard as part of the core Metanorma stack, @ronaldtse.

Reply in https://github.com/metanorma/metanorma-cli/issues/345

ronaldtse commented 1 day ago

Replied at https://github.com/metanorma/metanorma-cli/issues/345