metanorma / metanorma-ietf

Metanorma processor for IETF documents
BSD 2-Clause "Simplified" License
6 stars 5 forks source link

noMethodError when processing document #205

Closed douglm closed 7 months ago

douglm commented 8 months ago

Get the following when processing the tasks rfc document:

https://github.com/CalConnect/PUBLIC_DRAFTS/tree/master/ical-tasks

$ metanorma --type ietf sources/draft-ietf-calext-ical-tasks.adoc
[relaton] Info: detecting backends:
[relaton] CalConnect Task Architecture V1.0 does not have a recognised prefix: CN, IEC, IETF, ISO, ITU, NIST, OGC, CC, OMG, UN, W3C, IEEE, IHO, BIPM, ECMA, CIE, BSI, CEN.
See https://github.com/relaton/relaton/ for instructions on prefixing and wrapping document identifiers to disambiguate them.
[relaton] CalConnect Task Architecture V1.0 does not have a recognised prefix: CN, IEC, IETF, ISO, ITU, NIST, OGC, CC, OMG, UN, W3C, IEEE, IHO, BIPM, ECMA, CIE, BSI, CEN.
See https://github.com/relaton/relaton/ for instructions on prefixing and wrapping document identifiers to disambiguate them.
[relaton-un] ("UN/EDIFACT") fetching...
Fatal Error: undefined method `[]' for nil:NilClass
Traceback (most recent call last):
    81: from /__ruby_packer_memfs__/local/metanorma:211:in `<main>'
    80: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-cli-1.4.17/lib/metanorma/cli.rb:34:in `start'
    79: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
    78: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
    77: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
    76: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/thor-hollaback-0.2.0/lib/thor/hollaback.rb:62:in `run'
    75: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
    74: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-cli-1.4.17/lib/metanorma/cli/command.rb:45:in `compile'
    73: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-cli-1.4.17/lib/metanorma/cli/command.rb:45:in `each'
    72: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-cli-1.4.17/lib/metanorma/cli/command.rb:45:in `block in compile'
    71: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-cli-1.4.17/lib/metanorma/cli/command.rb:210:in `compile_document'
    70: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-cli-1.4.17/lib/metanorma/cli/compiler.rb:35:in `compile'
    69: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-cli-1.4.17/lib/metanorma/cli/compiler.rb:30:in `compile'
    68: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-cli-1.4.17/lib/metanorma/cli/compiler.rb:45:in `compile_file'
    67: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-1.3.12/lib/metanorma/compile.rb:30:in `compile'
    66: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-1.3.12/lib/metanorma/compile.rb:90:in `process_input'
    65: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-1.3.12/lib/metanorma/compile.rb:107:in `process_input_adoc'
    64: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-1.3.12/lib/metanorma/processor.rb:21:in `input_to_isodoc'
    63: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-1.3.12/lib/metanorma/input/asciidoc.rb:18:in `process'
    62: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/asciidoctor-2.0.16/lib/asciidoctor/convert.rb:120:in `convert'
    61: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/asciidoctor-2.0.16/lib/asciidoctor/document.rb:948:in `convert'
    60: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/utils.rb:13:in `convert'
    59: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/base.rb:163:in `document'
    58: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-ietf-2.4.3/lib/asciidoctor/ietf/converter.rb:32:in `makexml'
    57: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/base.rb:201:in `makexml'
    56: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/base.rb:195:in `makexml1'
    55: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/utils.rb:34:in `noko'
    54: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.12.5-x86_64-darwin/lib/nokogiri/xml/builder.rb:292:in `with'
    53: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.12.5-x86_64-darwin/lib/nokogiri/xml/builder.rb:292:in `new'
    52: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.12.5-x86_64-darwin/lib/nokogiri/xml/builder.rb:334:in `initialize'
    51: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/base.rb:195:in `block in makexml1'
    50: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/base.rb:223:in `middle'
    49: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.12.5-x86_64-darwin/lib/nokogiri/xml/builder.rb:416:in `method_missing'
    48: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.12.5-x86_64-darwin/lib/nokogiri/xml/builder.rb:433:in `insert'
    47: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/base.rb:224:in `block in middle'
    46: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/asciidoctor-2.0.16/lib/asciidoctor/document.rb:998:in `content'
    45: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/asciidoctor-2.0.16/lib/asciidoctor/abstract_block.rb:84:in `content'
    44: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/asciidoctor-2.0.16/lib/asciidoctor/abstract_block.rb:84:in `map'
    43: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/asciidoctor-2.0.16/lib/asciidoctor/abstract_block.rb:84:in `block in content'
    42: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/asciidoctor-2.0.16/lib/asciidoctor/abstract_block.rb:75:in `convert'
    41: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/utils.rb:13:in `convert'
    40: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/section.rb:68:in `section'
    39: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/utils.rb:34:in `noko'
    38: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.12.5-x86_64-darwin/lib/nokogiri/xml/builder.rb:292:in `with'
    37: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.12.5-x86_64-darwin/lib/nokogiri/xml/builder.rb:292:in `new'
    36: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.12.5-x86_64-darwin/lib/nokogiri/xml/builder.rb:334:in `initialize'
    35: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/section.rb:94:in `block in section'
    34: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/ref_sect.rb:23:in `bibliography_parse'
    33: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.12.5-x86_64-darwin/lib/nokogiri/xml/builder.rb:416:in `method_missing'
    32: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.12.5-x86_64-darwin/lib/nokogiri/xml/builder.rb:433:in `insert'
    31: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/ref_sect.rb:26:in `block in bibliography_parse'
    30: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/asciidoctor-2.0.16/lib/asciidoctor/abstract_block.rb:84:in `content'
    29: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/asciidoctor-2.0.16/lib/asciidoctor/abstract_block.rb:84:in `map'
    28: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/asciidoctor-2.0.16/lib/asciidoctor/abstract_block.rb:84:in `block in content'
    27: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/asciidoctor-2.0.16/lib/asciidoctor/list.rb:30:in `convert'
    26: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/asciidoctor-2.0.16/lib/asciidoctor/abstract_block.rb:75:in `convert'
    25: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/utils.rb:13:in `convert'
    24: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/lists.rb:39:in `ulist'
    23: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/ref_sect.rb:13:in `reference'
    22: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/utils.rb:34:in `noko'
    21: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.12.5-x86_64-darwin/lib/nokogiri/xml/builder.rb:292:in `with'
    20: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.12.5-x86_64-darwin/lib/nokogiri/xml/builder.rb:292:in `new'
    19: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.12.5-x86_64-darwin/lib/nokogiri/xml/builder.rb:334:in `initialize'
    18: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/ref_sect.rb:14:in `block in reference'
    17: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/ref_sect.rb:14:in `each'
    16: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/ref_sect.rb:14:in `block (2 levels) in reference'
    15: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/ref.rb:217:in `reference1'
    14: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/ref.rb:157:in `refitem'
    13: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/ref.rb:165:in `refitem1'
    12: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/metanorma-standoc-1.10.7/lib/asciidoctor/standoc/ref_sect.rb:70:in `fetch_ref'
    11: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/relaton-1.9.1/lib/relaton/db.rb:72:in `fetch'
    10: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/relaton-1.9.1/lib/relaton/db.rb:260:in `combine_doc'
     9: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/relaton-1.9.1/lib/relaton/db.rb:385:in `check_bibliocache'
     8: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/relaton-1.9.1/lib/relaton/db.rb:405:in `new_bib_entry'
     7: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/relaton-1.9.1/lib/relaton/db.rb:420:in `net_retry'
     6: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/relaton-un-1.9.0/lib/relaton_un/processor.rb:19:in `get'
     5: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/relaton-un-1.9.0/lib/relaton_un/un_bibliography.rb:26:in `get'
     4: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/relaton-un-1.9.0/lib/relaton_un/un_bibliography.rb:41:in `isobib_search_filter'
     3: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/relaton-un-1.9.0/lib/relaton_un/un_bibliography.rb:10:in `search'
     2: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/relaton-un-1.9.0/lib/relaton_un/un_bibliography.rb:10:in `new'
     1: from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/relaton-un-1.9.0/lib/relaton_un/hit_collection.rb:28:in `initialize'
/__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/relaton-un-1.9.0/lib/relaton_un/hit_collection.rb:85:in `page_resp': undefined method `[]' for nil:NilClass (NoMethodError)
opoudjis commented 8 months ago

I've recently done an update (to be released Monday) which will cope better with failures to retrieve records. There also has been a recent update to relaton-un which copes better with such failure to retrieve (https://github.com/relaton/relaton-un/issues/14); I replicated a crash before updating the bibliography.

I'm reporting this failure to fetch to our bibliographic software on the same ticket, but it is possible that the identifier is not what relaton expects; stay tuned. I recommend that if the document is not available for download, the reference should be formatted so as to sidestep online retrieval, formatting the information semantically, as:

https://www.metanorma.org/author/topics/document-format/bibliography/#disable-auto-fetch https://www.metanorma.org/author/topics/document-format/bibliography/#annotated-spans

* [[[EDISTS,nofetch(UN/EDIFACT, D14.A)]]], span:organization[UN Economic Commission for Europe], 
span:docid.UN[UN/EDIFACT, D14.A], span:series[STS STATUS], span:date[2014-04-30],
span:uri[http://www.unece.org/fileadmin/DAM/trade/untdid/d14a/trsd/trsdsts.htm]

After all, we are trying to generate RFC XML, which decomposes bibliographic entries into semantic elements.

Given the Asciidoctor:


[bibliography]
== Bibliography
* [[[EDISTS,nofetch(UN/EDIFACT, D14.A)]]], span:organization[UN Economic Commission for Europe], span:docid.UN[UN/EDIFACT, D14.A], span:title[STS STATUS], span:date[2014-04-30], span:uri.src[http://www.unece.org/fileadmin/DAM/trade/untdid/d14a/trsd/trsdsts.htm]
* [[[EDISTS1,UN/EDIFACT, D14.A]]], UN Economic Commission for Europe, UN/EDIFACT, D14.A, STS STATUS, April 30, 2014,http://www.unece.org/fileadmin/DAM/trade/untdid/d14a/trsd/trsdsts.htm

Metanorma will generate the RFC XML output:

   <back>
    <references anchor="_bibliography">
      <name>Bibliography</name>
      <reference target="http://www.unece.org/fileadmin/DAM/trade/untdid/d14a/trsd/trsdsts.htm" anchor="EDISTS"><front> <title>STS STATUS</title> <author><organization ascii="UN Economic Commission for Europe">UN Economic Commission for Europe</organization></author> <date day="30" year="2014" month="April"></date> </front> <format target="http://www.unece.org/fileadmin/DAM/trade/untdid/d14a/trsd/trsdsts.htm" type="src"></format> <refcontent>UN/EDIFACT, D14.A</refcontent></reference>
      <reference anchor="EDISTS1"><front> <title>UN Economic Commission for Europe, UN/EDIFACT, D14.A, STS STATUS, April 30, 2014,http://www.unece.org/fileadmin/DAM/trade/untdid/d14a/trsd/trsdsts.htm</title><author surname="Unknown"></author> </front> <refcontent>UN/EDIFACT, D14.A</refcontent></reference>
    </references>
  </back>

Which renders to text as:

   [EDISTS]   UN Economic Commission for Europe, "STS STATUS", UN/
              EDIFACT, D14.A, 30 April 2014,
              <http://www.unece.org/fileadmin/DAM/trade/untdid/d14a/
              trsd/trsdsts.htm>.

   [EDISTS1]  "UN Economic Commission for Europe, UN/EDIFACT, D14.A, STS
              STATUS, April 30,
              2014,http://www.unece.org/fileadmin/DAM/trade/untdid/d14a/
              trsd/trsdsts.htm", UN/EDIFACT, D14.A.

As you can see,

opoudjis commented 8 months ago

@douglm Release should now be out, could you please verify behaviour now?

opoudjis commented 7 months ago

@douglm Please confirm document is now compiling as expected.

douglm commented 7 months ago

I just tried rebuilding the document (with some changes) and it worked.

I didn't update metanorma however. Reinstated EDISTS and it failed.

Tried brew upgrade metanorma and tried again still failed - then saw metanorma/metanorma/metanorma 1.4.17.20211007_1 already installed

For the moment I'll just remove EDISTS

ronaldtse commented 7 months ago

@douglm that's very strange -- the version is rather old.

Can you try this:

brew install metanorma/metanorma/metanorma

(yes "metanorma" 3 times)

Latest versions from installing from homebrew:

/opt/homebrew/bin/metanorma -v
Metanorma 1.7.3
Metanorma::Cli 1.9.5
Metanorma::Standoc 2.8.6/IsoDoc 2.9.4
Metanorma::ISO 2.7.5
Metanorma::Iec 2.4.5
Metanorma::IEEE 1.2.6
Metanorma::Ietf 3.3.5
Metanorma::Generic 2.6.3
Metanorma::BIPM 2.4.5
Metanorma::CC 2.4.5
Metanorma::Csa 2.4.5
Metanorma::IHO 0.9.5
Metanorma::JIS 0.2.5
Metanorma::UN 0.12.6
Metanorma::Ogc 2.5.5
Metanorma::ITU 2.4.5
ronaldtse commented 7 months ago

The problem here is the identifier of UN/EDIFACT triggers the load of relaton-un but the UN Docs system now no longer allows machine access. We are in negotiation with the UN Docs team to see how we can fetch from it again.

relaton-un has been temporarily disabled in https://github.com/relaton/relaton-un/commit/30f0ed58ec85e5ea388c3d3c269391387d5dba06

ronaldtse commented 7 months ago

@douglm @opoudjis I can confirm that the issue is fixed at the latest Metanorma version.

I've made a PR here as well: