search

metanorma / coradoc

Coradoc is the Core AsciiDoc Parser used by Metanorma
MIT License
1 stars 4 forks source link

(URGENT) reverse_adoc: broken table formatting causing document compilation failure #77

Closed ronaldtse closed 5 months ago

ronaldtse commented 5 months ago

Steps:

Obtain HTML file

This is a document from:

Set prerequisites

Specify Gemfile:

source "https://rubygems.org"
gem "metanorma-cli"
gem "coradoc"

Convert via reverse_adoc

bundle exec reverse_adoc -rcoradoc/reverse_adoc/plugins/plateau \
  --split-sections 2 --external-images \
  -o plateau-document-01-v4/document.adoc \
  plateau-document-01-v4.html

Add document header information

Add this header information to the plateau-document-01-v4/document.adoc.

= Handbook of 3D City Models
:docnumber: 1
:edition: 4
:revdate: 2024-03-22
:published-date: 2024-03-22
:copyright-year: 2024
:language: ja
:title-intro-ja: 3D都市モデル導入のためのガイドブック
:title-main-ja: 3D都市モデル標準製品仕様書
:publisher: 国土交通省 都市局
:sponsor: （協力）内閣府 地方創生推進事務局
:title-intro-en: Handbook of 3D City Models
:title-main-en: Standard Data Product Specification for 3D City Model
:doctype: international-standard
:imagedir: images
:mn-document-class: iso
:mn-output-extensions: xml,html,pdf,rxl
:local-cache-only:
:data-uri-image:

include::sections/abstract-01.adoc[]

include::sections/abstract-02.adoc[]
...

Compile with Metanorma

Run this command:

bundle exec metanorma plateau-document-01-v4/document.adoc

See this error:

...
Table: (ID _2b02ef02-66e8-069d-d7a8-bea27f7d67da): Table rows in table cannot go outside thead: check rowspan
Table: (ID _2b02ef02-66e8-069d-d7a8-bea27f7d67da): Table rows in table cannot go outside tbody: check rowspan
Table: (ID _ba972cee-5f75-b194-e8c9-7dcd7d2a6b48): Table rows in table cannot go outside thead: check rowspan
Table: (ID _ba972cee-5f75-b194-e8c9-7dcd7d2a6b48): Table rows in table cannot go outside tbody: check rowspan

These are the tables:

sections/section-04/section-05.adoc

[[toc4_05_03_01]]
===== (1)　Transportation（CityGML）

===== 1)　tran:Track

[cols="1,1,2"]
|===
.6+| 型の定義 2+| 徒歩道。徒歩道とは、人や車両等の通行の用に供される道路のうち、道路法第3条に示された道路の種類及び建築基準法第42条による「道路」を除く道路を指す。徒歩道には、作業規程の準則　付録７公共測量標準図式における徒歩道及び庭園路（ただし、庭園路のうち、自動車ターミナル内の道路は、広場として取得するため、徒歩道には含まない）を含む。 ここで、作業規程の準則付録７公共測量標準図式における徒歩道とは、「道路縁及び軽車道に接続するもの、登山、観光等に利用されるもの、神社等主要な地点へ到達するもの、耕地の区画等の景観を表現するために必要なもの」であり、庭園路とは、「公園、住宅地等で自動車の通行を規制している道路及び工場等特定の敷地内の道路」である。 徒歩道の延長方向は、以下の場所で区切る。 • 交差部（四差路、多差路及び三差路） • 道路構造の変化点（トンネル、橋梁） • 位置正確度（地図情報レベル）や取得方法 同一のLODにおいて、連続する徒歩道の境界は一致しなければならない。 tran:Trackは、LOD0ではネットワーク（中心線）又は徒歩道縁により取得する。 LOD1以上では、面として取得する。 LOD2以上では、tran:Trackの面を、tran:TrafficAreaとtran:AuxiliaryTrafficAreaに細分する。 さらに、LOD3では、各地物の面に高さを付与する。 以下に、取得例を示す。

sections/section-04/section-18.adoc

===== 2)　地形モデル（LOD1）の定義

地形モデル（LOD1）の定義として、含むべき地物型、各地物型の空間属性の型、取得基準、取得方法及び補足を示す。

[cols="43,43,43,48,43,100,80"]
|===
| | | 地物型 | 空間属性の型 | 取得基準 | 取得方法 | 補足

| LOD1 | ● | ReliefFeature | ― | • | • MassPointRelief又はTINReliefの集まりとして取得する。 | ReliefFeatureは地物の集まりとして表現し、空間属性はもたない。

sections/section-04/section-18.adoc

[[toc4_18_01_03]]
===== (3)　地形モデル（LOD2）

===== 1)　地形モデル（LOD2）の概要

地形モデル（LOD2）では、地形を、標高をもつ点の集まり又は標高をもつ点から構成する三角形の集合により表現する。

地形モデル（LOD2）は、格子間隔2m以内（地図情報レベル2500）としている。

===== 2)　地形モデル（LOD2）の定義

地形モデル（LOD2）の定義として、含むべき地物型、各地物型の空間属性の型、取得基準、取得方法及び補足を示す。

[cols="43,43,43,48,43,100,80"]
|===
| | | 地物型 | 空間属性の型 | 取得基準 | 取得方法 | 補足

| LOD2 | ● | ReliefFeature | ― | | • MassPointRelief又はTINReliefの集まりとして取得する。 | ReliefFeatureは地物の集まりとして表現し、空間属性はもたない。
| LOD2

sections/section-04/section-18.adoc

[[toc4_18_01_04]]
===== (4)　地形モデル（LOD3）

===== 1)　地形モデル（LOD3）の概要

地形モデル（LOD3）では、地形を、標高をもつ点の集まり又は標高をもつ点から構成する三角形の集合により表現する。

地形モデル（LOD3）は、格子間隔1m以内（地図情報レベル1000）としている。

===== 2)　地形モデル（LOD3）の定義

地形モデル（LOD3）の定義として、含むべき地物型、各地物型の空間属性の型、取得基準、取得方法及び補足を示す。

[cols="43,43,43,48,43,100,80"]
|===
| | | 地物型 | 空間属性の型 | 取得基準 | 取得方法 | 補足

| LOD3 | ● | ReliefFeature | ― | 　 | • MassPointRelief又はTINReliefの集まりとして取得する。 | ReliefFeatureは地物の集まりとして表現し、空間属性はもたない。
| LOD3

Conclusion

These tables should not cause errors.

ronaldtse commented 5 months ago

@hmdne the second document isn't generating properly:

$ bundle exec reverse_adoc -rcoradoc/reverse_adoc/plugins/plateau \
  --split-sections 2 --external-images \
  -o plateau-document-02-v4/document.adoc \
  reference-docs/plateau-document-02-v4.html
coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters/table.rb:218:in `ensure_row_column_integrity_and_get_column_sizes': undefined method `-' for nil:NilClass (NoMethodError)

        column_sizes += [nil] * (cpr.first - column_sizes.length)
                                           ^
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters/table.rb:56:in `style'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters/table.rb:7:in `to_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters.rb:29:in `block in process_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/plugin.rb:110:in `html_tree_run_hooks'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters.rb:32:in `block (2 levels) in process_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters.rb:34:in `process_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters/base.rb:29:in `treat_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters/base.rb:24:in `block in treat_children_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/nokogiri-1.15.6-arm64-darwin/lib/nokogiri/xml/node_set.rb:235:in `block in each'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/nokogiri-1.15.6-arm64-darwin/lib/nokogiri/xml/node_set.rb:234:in `upto'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/nokogiri-1.15.6-arm64-darwin/lib/nokogiri/xml/node_set.rb:234:in `each'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters/base.rb:23:in `inject'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters/base.rb:23:in `treat_children_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters/div.rb:6:in `to_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters.rb:29:in `block in process_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/plugin.rb:110:in `html_tree_run_hooks'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters.rb:32:in `block (2 levels) in process_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters.rb:34:in `process_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters/base.rb:29:in `treat_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters/base.rb:24:in `block in treat_children_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/nokogiri-1.15.6-arm64-darwin/lib/nokogiri/xml/node_set.rb:235:in `block in each'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/nokogiri-1.15.6-arm64-darwin/lib/nokogiri/xml/node_set.rb:234:in `upto'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/nokogiri-1.15.6-arm64-darwin/lib/nokogiri/xml/node_set.rb:234:in `each'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters/base.rb:23:in `inject'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters/base.rb:23:in `treat_children_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters/div.rb:6:in `to_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters.rb:29:in `block in process_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/plugin.rb:110:in `html_tree_run_hooks'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters.rb:32:in `block (2 levels) in process_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters.rb:34:in `process_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters/base.rb:29:in `treat_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/coradoc-96075a61208d/lib/coradoc/reverse_adoc/converters/base.rb:24:in `block in treat_children_coradoc'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/nokogiri-1.15.6-arm64-darwin/lib/nokogiri/xml/node_set.rb:235:in `block in each'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/nokogiri-1.15.6-arm64-darwin/lib/nokogiri/xml/node_set.rb:234:in `upto'
    from ~/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/nokogiri-1.15.6-arm64-darwin/lib/nokogiri/xml/node_set.rb:234:in `each'
ronaldtse commented 5 months ago

@reeseplews has discovered the workaround and will be posted here.

ReesePlews commented 5 months ago

hi @ronaldtse and @webdev778

i was working with these files (locally on my machine and have not pushed anything back to github; running mn locally)

https://www.mlit.go.jp/plateaudocument/ => plateau-document-01-v4.html.zip

using these commands

bundle install

bundle exec reverse_adoc -rcoradoc/reverse_adoc/plugins/plateau --split-sections 2 --external-images -o sources/001-v4_0603a/index.adoc reference-docs/plateau-document-01-v4.html

it worked very quickly and finished without any error to the terminal. i know the document.xml.abort file was created but at first i did not check it.

next i updated the document.adoc file by removing the header and replacing it with data from the v3 processing a couple months back.

using irfanview i converted all of the webp files to png. and i updated those file names in all the code files using vs.

then i ran this command to generate a generic html output from the adoc files.

metanorma --type generic -x html sources/001-v4/document.adoc

this command identified two tables which could not be processed and the command stopped.

Table: (ID _4ae96f30-ab86-5425-7781-2f3bee8660b8): Table rows in table cannot go outside thead: check rowspan
Table: (ID _4ae96f30-ab86-5425-7781-2f3bee8660b8): Table rows in table cannot go outside tbody: check rowspan

Table: (ID _e559ea15-61bb-06de-ebbf-d582da970e94): Table rows in table cannot go outside thead: check rowspan
Table: (ID _e559ea15-61bb-06de-ebbf-d582da970e94): Table rows in table cannot go outside tbody: check rowspan

i dont know why the errors were duplicated.

using the table id, i searched in the document.xml.abort to identify the tables (based on pieces of unique text) and found them in the doc01v4 PDF file.

error for table id: 4ae96f30-ab86-5425-7781-2f3bee8660b8

pg 248 in the PDF image

was encoded in the asciidoc file: sources\001-v4\sections\section-04\section-05.adoc

this shows what i changed. first i removed the large bullets but that did not make a difference. next i checked the asciidoc code inside of https://asciidoclive.com, and noticed that something was possibly wrong with the header, so i put in a CRLF before that large block of text at the top. that solved this issue.

image

error for table id: e559ea15-61bb-06de-ebbf-d582da970e94

pg 823 in the PDF image

was encoded in the asciidoc file: sources\001-v4\sections\section-04\section-18.adoc

this table rendered correctly in https://asciidoclive.com but i could not figure out how to change it. i think the issue is the upper left empty cell then merged cells for the header. the header should probably be across the entire top row but i could not figure out how to change it. the encoding of the tables in the asciidoc files were a bit different than i was used to. my poor solution is shown below. i just commented out the original table and then remade very simplified, but it seemed to work.

image

summary

out of such a long document it was only these two tables that were giving the fatal errors. i dont understand everything well enough to suggest why they are causing mn to have problems.

in the final solution, the first table (large table) will be generated via lutaml reading in a UML model and making the output, so those type of tables in the document will be removed and lutaml code will be added. the second table is a static table. we can modify that style, so i am not so worried for my project, but the original output from coradoc was rendered correctly in asciidocliv.com but then for some reason caused an error in mn processing.

thank you for checking these issues. if you have questions, please let me know.

hmdne commented 5 months ago

@ReesePlews

The problem with our implementation, is that there are tables like this:

<table>
<thead>
  <tr>
    <td>
    <td rolspan='2'>
  </tr>
</thead>
<tbody>
  <tr>
    <td>
  </tr>
</tbody>
</table>

This produces a table of this structure, where the first row is THEAD:

+---+---+
|   |   |
+---+   |
|   |   |
+---+---+

Web browsers do accept it, Asciidoctor also, but Metanorma doesn't, because it checks for both THEAD and TBODY to be independently valid tables (see: https://github.com/metanorma/metanorma-standoc/blob/3333ce41ce001b78ee42dcbd79a91262066b1029/lib/metanorma/standoc/validate_table.rb#L4 ).

But a fix that I have introduced with #78 (which has been merged a couple days ago, so you can recheck) ensures that if the first row has some rows cells than 1 row, AsciiDoc is generated with options="noheader" to ensure that THEAD is not produced and everything goes to the single TBODY.

The logic on header generation in reverse_adoc is not robust. It generates a single header row, basically always, unless there are issues with AsciiDoc syntax. So... this is why I call this fix an interim one.

For document 1, you can rerun the generation. You will need to git pull the repository. From my check, it now works correctly.

@ronaldtse I haven't worked at all with document 2 yet.

ReesePlews commented 5 months ago

@hmdne thank you for the update and the code fix, i will test as you have suggested.

in my project, if there are really unique table structures that are too difficult to support in metanorma, there is some flexibility to consider a different source layout, at this point in time.

i am also not checking the conversion of doc2 at this time. i will begin that work later.

ReesePlews commented 5 months ago

hello @hmdne i have tried to check using the following process:

bundle install
...
Using coradoc 0.3.0 from https://github.com/metanorma/coradoc (at main@70275d6)
...

then

bundle exec reverse_adoc -rcoradoc/reverse_adoc/plugins/plateau --split-sections 2 --external-images -o sources/001-v4_0604a/index.adoc reference-docs/plateau-document-01-v4.html
The `reverse_adoc` executable in the `coradoc` gem is being loaded, but it's also present in other gems (reverse_adoc).
If you meant to run the executable for another gem, make sure you use a project specific binstub (`bundle binstub <gem_name>`).
If you plan to use multiple conflicting executables, generate binstubs for them and disambiguate their names.
C:/tools/ruby31/lib/ruby/gems/3.1.0/bundler/gems/coradoc-70275d6332eb/exe/reverse_adoc:55:in `read': No such file or directory @ rb_sysopen - reference-docs/plateau-document-01-v4.html (Errno::ENOENT)
        from C:/tools/ruby31/lib/ruby/gems/3.1.0/bundler/gems/coradoc-70275d6332eb/exe/reverse_adoc:55:in `<top (required)>'
        from C:/tools/ruby31/lib/ruby/gems/3.1.0/bin/reverse_adoc:25:in `load'
        from C:/tools/ruby31/lib/ruby/gems/3.1.0/bin/reverse_adoc:25:in `<main>'

this is the first time this error occurred. the file is present in the reference-docs directory

ls reference-docs/plateau-document-01-v4.html
    Directory: E:\github\mn-samples-plateau\reference-docs
Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        2024-05-31  14:16 PM       55207264 plateau-document-01-v4.html

am i doing something wrong?

hmdne commented 5 months ago 
bundle exec reverse_adoc -rcoradoc/reverse_adoc/plugins/plateau --split-sections 2 --external-images -o sources/001-v4_0604a/index.adoc reference-docs/plateau-document-01-v4.html
The `reverse_adoc` executable in the `coradoc` gem is being loaded, but it's also present in other gems (reverse_adoc).
If you meant to run the executable for another gem, make sure you use a project specific binstub (`bundle binstub <gem_name>`).
If you plan to use multiple conflicting executables, generate binstubs for them and disambiguate their names.
This is not an error, it's a warning, but things work correctly in this case. This will need more investigation, but for now, ensure you don't have `reverse_adoc` in your Gemfile/gemspec to surpress this warning.

C:/tools/ruby31/lib/ruby/gems/3.1.0/bundler/gems/coradoc-70275d6332eb/exe/reverse_adoc:55:in read': No such file or directory @ rb_sysopen - reference-docs/plateau-document-01-v4.html (Errno::ENOENT) from C:/tools/ruby31/lib/ruby/gems/3.1.0/bundler/gems/coradoc-70275d6332eb/exe/reverse_adoc:55:in<top (required)>' from C:/tools/ruby31/lib/ruby/gems/3.1.0/bin/reverse_adoc:25:in load' from C:/tools/ruby31/lib/ruby/gems/3.1.0/bin/reverse_adoc:25:in

'


This is the error.

this is the first time this error occurred. the file is present in the reference-docs directory

We haven't touched this fragment recently.

What I may suggest in general, is to run Ruby under WSL. While we do test for Windows and Windows port is robust in general, still Linux is the primary platform on which Ruby is developed, so there may be some regressions.

ls reference-docs/plateau-document-01-v4.html
    Directory: E:\github\mn-samples-plateau\reference-docs
Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        2024-05-31  14:16 PM       55207264 plateau-document-01-v4.html

am i doing something wrong?

Try entering a full path to this file. I will try to debug this issue on my Windows VM in meantime.

hmdne commented 5 months ago

@ReesePlews I have unsuccessfully tried to reproduce your issue:

image

Those are all clean installs from RubyInstaller.

All I can recommend is for you to ensure you have correct paths.

ReesePlews commented 5 months ago

@hdmne, thanks for checking this. my ruby version is:

ruby --version
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x64-mingw-ucrt]

do you think that would be an issue? i will try and update ruby.

i tried to install WSL on this machine (where i am using MS-PowerShell) but was not successful. i have WSL on another machine and do have a linux machine but use it for other work.

for now i will stick with powershell, it was working well a few days ago. i will report back after the ruby update.

ReesePlews commented 5 months ago

@hmdne i have updated ruby

ruby --version
ruby 3.3.2 (2024-05-30 revision e5a195edf6) [x64-mingw-ucrt]

after that

bundle install
...PS E:\github\mn-samples-plateau> bundle install
Bundler 2.5.9 is running, but your lockfile was generated with 2.3.7. Installing Bundler 2.3.7 and restarting using that version.
Fetching gem metadata from https://rubygems.org/.
Fetching bundler 2.3.7
Installing bundler 2.3.7
Fetching gem metadata from https://rubygems.pkg.github.com/metanorma/..
Fetching https://github.com/metanorma/isodoc
Fetching gem metadata from https://rubygems.org/........
Fetching https://github.com/metanorma/coradoc
Your bundle is locked to nokogiri (1.15.6-x64-mingw-ucrt) from rubygems
repository https://rubygems.org/ or installed locally, but that version can no
longer be found in that source. That means the author of nokogiri
(1.15.6-x64-mingw-ucrt) has removed it. You'll need to update your bundle to a
version other than nokogiri (1.15.6-x64-mingw-ucrt) that hasn't been removed in
order to install.

retried the conversion tool using relative path syntax

bundle exec reverse_adoc -rcoradoc/reverse_adoc/plugins/plateau --split-sections 2 --external-images -o ./sources/001-v4_0604a/index.adoc ./reference-docs/plateau-document-01-v4.html
The `reverse_adoc` executable in the `reverse_adoc` gem is being loaded, but it's also present in other gems (coradoc).
If you meant to run the executable for another gem, make sure you use a project specific binstub (`bundle binstub <gem_name>`).
If you plan to use multiple conflicting executables, generate binstubs for them and disambiguate their names.
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/reverse_adoc-0.3.7/exe/reverse_adoc:40:in `<top (required)>': invalid option: -rcoradoc/reverse_adoc/plugins/plateau (OptionParser::InvalidOption)
        from C:/tools/ruby31/bin/reverse_adoc:25:in `load'
        from C:/tools/ruby31/bin/reverse_adoc:25:in `<main>'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/reverse_adoc-0.3.7/exe/reverse_adoc:40:in `<top (required)>': invalid option: r (OptionParser::InvalidOption)
        from C:/tools/ruby31/bin/reverse_adoc:25:in `load'
        from C:/tools/ruby31/bin/reverse_adoc:25:in `<main>'

i believe the error has changed, it is no longer saying file not found... does this provide any additional information on why it is not working?

ReesePlews commented 5 months ago

this is the Gemfile in my repo (mn-samples-plateau) i have never modified

cat Gemfile

source "https://rubygems.org"

gem "metanorma-plateau", source: "https://rubygems.pkg.github.com/metanorma"
#gem "mn2pdf", git: "https://github.com/metanorma/mn2pdf-ruby", branch: "main"
gem "isodoc", git: "https://github.com/metanorma/isodoc", branch: "main"
gem "metanorma-cli"
gem "coradoc", git: "https://github.com/metanorma/coradoc", branch: "main"

# WARNING: only for developers
#gem "metanorma-plateau", git: "git@github.com:metanorma/metanorma-plateau.git"
#gem "metanorma-plateau", path: "~/src/mn/metanorma-plateau"
#gem "sassc"
hmdne commented 5 months ago

@ReesePlews run bundle update. After updating Ruby, old version of Nokogiri persists in your Gemfile.

Also this:

The `reverse_adoc` executable in the `reverse_adoc` gem is being loaded, but it's also present in other gems (coradoc).
If you meant to run the executable for another gem, make sure you use a project specific binstub (`bundle binstub <gem_name>`).
If you plan to use multiple conflicting executables, generate binstubs for them and disambiguate their names.

Now means that it tries to load reverse_adoc from reverse_adoc (old) gem, not coradoc (new).

Therefore, ensure to run bundle update or remove Gemfile.lock.

ReesePlews commented 5 months ago

hi @hmdne thank you for the instructions. everything was working well until i tried to convert the document using metanorma, then this error appeared.

PS E:\github\mn-samples-plateau> metanorma --type generic -x html sources/001-v4_0605a/document.adoc
C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1480:in `rescue in block in activate_dependencies': Could not find 'isodoc' (~> 2.10.2) among 319 total gem(s) (Gem::MissingSpecError)
Checked in 'GEM_PATH=C:/Users/admin/.gem/ruby/3.3.0;C:/tools/ruby33/lib/ruby/gems/3.3.0' at: C:/tools/ruby33/lib/ruby/gems/3.3.0/specifications/metanorma-standoc-2.8.10.gemspec, execute `gem env` for more information
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1477:in `block in activate_dependencies'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1466:in `each'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1466:in `activate_dependencies'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1448:in `activate'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1484:in `block in activate_dependencies'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1466:in `each'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1466:in `activate_dependencies'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1448:in `activate'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1484:in `block in activate_dependencies'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1466:in `each'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1466:in `activate_dependencies'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1448:in `activate'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems.rb:280:in `block in activate_bin_path'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems.rb:279:in `synchronize'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems.rb:279:in `activate_bin_path'
        from C:/tools/ruby33/bin/metanorma:25:in `<main>'
C:/tools/ruby33/lib/ruby/3.3.0/rubygems/dependency.rb:313:in `to_specs': Could not find 'isodoc' (~> 2.10.2) among 319 total gem(s) (Gem::MissingSpecError)
Checked in 'GEM_PATH=C:/Users/admin/.gem/ruby/3.3.0;C:/tools/ruby33/lib/ruby/gems/3.3.0' , execute `gem env` for more information
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1478:in `block in activate_dependencies'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1466:in `each'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1466:in `activate_dependencies'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1448:in `activate'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1484:in `block in activate_dependencies'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1466:in `each'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1466:in `activate_dependencies'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1448:in `activate'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1484:in `block in activate_dependencies'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1466:in `each'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1466:in `activate_dependencies'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems/specification.rb:1448:in `activate'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems.rb:280:in `block in activate_bin_path'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems.rb:279:in `synchronize'
        from C:/tools/ruby33/lib/ruby/3.3.0/rubygems.rb:279:in `activate_bin_path'
        from C:/tools/ruby33/bin/metanorma:25:in `<main>'

i also tried 

bundle update isodoc
....
Bundler attempted to update isodoc but its version stayed the same

i can send the gem env output if you think that would be helpful.

after doing the initial instructions is is necessary to reboot the machine? i did restart the shell, but there was no change. any advice would be most helpful.

i tried again to install WSL. this time i was successful with the install but no distro has been selected. i would pick ubuntu but what do you recommend that works smoothly? thank you.

hmdne commented 5 months ago

@ReesePlews

I would recommend Ubuntu for WSL.

metanorma --type generic -x html sources/001-v4_0605a/document.adoc

Do: bundle exec metanorma so that it uses the environment specified in Gemfile.

hmdne commented 5 months ago

For most cases it's enough to just restart the shell, no need to reboot the machine.

ReesePlews commented 5 months ago

hello @hmdne, thank you for the information. i take it "bundle exec" is a ruby construct, not a windows shell specific construct. if it is advisable to always use "bundle exec" before any metanorma related tool please suggest an update for the documentation team. i found only one example (googling "bundle exec" site:metanorma.org) i think many non-developers have no idea what ruby is or how to use it.

i did as you suggested and the tool was running very well but unfortunately then stopped with an error... thinking it could be an earlier issue i tried bundle update which gave this output...

PS E:\github\mn-samples-plateau> bundle update
Fetching https://github.com/metanorma/coradoc
Fetching https://github.com/metanorma/isodoc
Fetching gem metadata from https://rubygems.pkg.github.com/metanorma/..
Fetching gem metadata from https://rubygems.org/...........
Resolving dependencies...........................................................................................................
Using rake 13.2.1
...

and then tried again:

bundle exec metanorma --type generic -x html sources/001-v4_0605a/document.adoc
...
...
Style: (ID _0079b7fe-8980-e37c-815f-657da3603446): Table should have title
#<Thread:0x000002f570ec11a8 C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/metanorma-2.0.0/lib/metanorma/util/worker_pool.rb:17 run> terminated with exception (report_on_exception is true):
C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/css.rb:78:in `require': cannot load such file -- sassc (LoadError)
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/css.rb:78:in `convert_scss'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/css.rb:101:in `generate_css'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/css.rb:28:in `populate_css'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/html_function/html.rb:11:in `block (2 levels) in convert1'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/nokogiri-1.15.6/lib/nokogiri/xml/builder.rb:433:in `insert'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/nokogiri-1.15.6/lib/nokogiri/xml/builder.rb:415:in `method_missing'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/html_function/html.rb:9:in `block in convert1'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/nokogiri-1.15.6/lib/nokogiri/xml/builder.rb:331:in `initialize'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/nokogiri-1.15.6/lib/nokogiri/xml/builder.rb:295:in `new'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/nokogiri-1.15.6/lib/nokogiri/xml/builder.rb:295:in `with'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/metanorma-utils-1.7.7/lib/utils/xml.rb:68:in `noko_html'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/function/utils.rb:31:in `noko'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/html_function/html.rb:8:in `convert1'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/convert.rb:168:in `convert'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/html_convert.rb:26:in `convert'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/metanorma-generic-2.6.4/lib/metanorma/generic/processor.rb:46:in `output'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/metanorma-2.0.0/lib/metanorma/compile/compile.rb:174:in `process_output_threaded'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/metanorma-2.0.0/lib/metanorma/compile/compile.rb:166:in `block in process_exts1'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/metanorma-2.0.0/lib/metanorma/util/worker_pool.rb:21:in `block (3 levels) in init_thread'
        from <internal:kernel>:187:in `loop'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/metanorma-2.0.0/lib/metanorma/util/worker_pool.rb:19:in `block (2 levels) in init_thread'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/metanorma-2.0.0/lib/metanorma/util/worker_pool.rb:18:in `catch'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/metanorma-2.0.0/lib/metanorma/util/worker_pool.rb:18:in `block in init_thread'
C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/css.rb:78:in `require': cannot load such file -- sassc (LoadError)
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/css.rb:78:in `convert_scss'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/css.rb:101:in `generate_css'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/css.rb:28:in `populate_css'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/html_function/html.rb:11:in `block (2 levels) in convert1'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/nokogiri-1.15.6/lib/nokogiri/xml/builder.rb:433:in `insert'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/nokogiri-1.15.6/lib/nokogiri/xml/builder.rb:415:in `method_missing'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/html_function/html.rb:9:in `block in convert1'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/nokogiri-1.15.6/lib/nokogiri/xml/builder.rb:331:in `initialize'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/nokogiri-1.15.6/lib/nokogiri/xml/builder.rb:295:in `new'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/nokogiri-1.15.6/lib/nokogiri/xml/builder.rb:295:in `with'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/metanorma-utils-1.7.7/lib/utils/xml.rb:68:in `noko_html'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/function/utils.rb:31:in `noko'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/html_function/html.rb:8:in `convert1'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/convert.rb:168:in `convert'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/html_convert.rb:26:in `convert'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/metanorma-generic-2.6.4/lib/metanorma/generic/processor.rb:46:in `output'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/metanorma-2.0.0/lib/metanorma/compile/compile.rb:174:in `process_output_threaded'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/metanorma-2.0.0/lib/metanorma/compile/compile.rb:166:in `block in process_exts1'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/metanorma-2.0.0/lib/metanorma/util/worker_pool.rb:21:in `block (3 levels) in init_thread'
        from <internal:kernel>:187:in `loop'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/metanorma-2.0.0/lib/metanorma/util/worker_pool.rb:19:in `block (2 levels) in init_thread'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/metanorma-2.0.0/lib/metanorma/util/worker_pool.rb:18:in `catch'
        from C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/metanorma-2.0.0/lib/metanorma/util/worker_pool.rb:18:in `block in init_thread'
PS E:\github\mn-samples-plateau>

but still resulted in this error. i still see "noko" (nokogiri) in the message... my current Gemfile is

source "https://rubygems.org"

gem "metanorma-plateau", source: "https://rubygems.pkg.github.com/metanorma"
#gem "mn2pdf", git: "https://github.com/metanorma/mn2pdf-ruby", branch: "main"
gem "isodoc", git: "https://github.com/metanorma/isodoc", branch: "main"
gem "metanorma-cli"
gem "coradoc", git: "https://github.com/metanorma/coradoc", branch: "main"

# WARNING: only for developers
#gem "metanorma-plateau", git: "git@github.com:metanorma/metanorma-plateau.git"
#gem "metanorma-plateau", path: "~/src/mn/metanorma-plateau"
#gem "sassc"

and at the last line i see mention of "sassc"

if the Gemfile and Gemfile.lock were removed, would they be recreated using bundle install + bundle update?

any advice is most appreciated.

i will install ubuntu on WSL, but it will be a few days. do you think these errors are related to using the tools on windows powershell or another matter?

thank you

ronaldtse commented 5 months ago

<Thread:0x000002f570ec11a8 C:/tools/ruby33/lib/ruby/gems/3.3.0/gems/metanorma-2.0.0/lib/metanorma/util/worker_pool.rb:17 run> terminated with exception (report_on_exception is true):

C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/css.rb:78:in require': cannot load such file -- sassc (LoadError) from C:/tools/ruby33/lib/ruby/gems/3.3.0/bundler/gems/isodoc-c0d740f2093a/lib/isodoc/css.rb:78:inconvert_scss'

Can you uncomment the # gem "sassc" line in Gemfile? Then it might work?

hmdne commented 5 months ago

i take it "bundle exec" is a ruby construct, not a windows shell specific construct

By default, Ruby uses gems from the global gems storage (ie. ones installed by gem install). If you have a Gemfile (which is a Bundler config), by using bundle exec you make it only use gems specified in the Gemfile, at the exact resolved versions needed, no more, no less. So in your case, it doesn't include sassc and no other Gem depends on that. So you need to add (uncomment) it manually.

If there's no Gemfile available, bundle exec won't work or make sense.

In particular, your Gemfile references git versions of the repositories, which wouldn't really be possible with just gem install.

if the Gemfile and Gemfile.lock were removed, would they be recreated using bundle install + bundle update?

Gemfile.lock is created by executing bundle install. It locks exact versions. Therefore, to upgrade the versions, you need to execute bundle update or remove the file. Gemfile shouldn't be removed.

i will install ubuntu on WSL, but it will be a few days. do you think these errors are related to using the tools on windows powershell or another matter?

They are not. But while developing Ruby applications, I have encountered many issues people had on Windows due to its... difficult semantics, so it was my first suspicion. If you have succeeded running reverse_adoc already, then I'm not sure it will help.

ReesePlews commented 5 months ago

thanks @ronaldtse that seemed to do the trick, but after uncommenting # gem "sassc" i also needed to do bundle install as it seemed sassc was not installed. but it finished correctly.

@hmdne i will keep this open a bit longer and may make some more checks in a day or two.

thank you.

ronaldtse commented 5 months ago

@ReesePlews how about we close this “table” issue and have the “command line on windows matter” in a new issue?

ronaldtse commented 5 months ago

This is confirmed fixed. Thanks!