search

relaton / relaton-nist

NistBib: retrieve NIST Standards for bibliographic use using the BibliographicItem model
https://www.metanorma.com
MIT License
2 stars 1 forks source link

NistBib crashes on search of "NIST Framework for Improving Critical Infrastructure Cybersecurity Version 1.1" #17

Closed ronaldtse closed 5 years ago

ronaldtse commented 5 years ago 
fetching NIST Framework for Improving Critical Infrastructure Cybersecurity Version 1.1...
bundler: failed to load command: metanorma (/Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/bin/metanorma)
ArgumentError: expected :on or :form argument
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-bib-0.1.5/lib/relaton_bib/bibliographic_date.rb:27:in `initialize'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-bib-0.1.5/lib/relaton_bib/xml_parser.rb:159:in `new'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-bib-0.1.5/lib/relaton_bib/xml_parser.rb:159:in `block in fetch_dates'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/node_set.rb:204:in `block in each'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/node_set.rb:203:in `upto'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/node_set.rb:203:in `each'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-bib-0.1.5/lib/relaton_bib/xml_parser.rb:158:in `map'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-bib-0.1.5/lib/relaton_bib/xml_parser.rb:158:in `fetch_dates'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-bib-0.1.5/lib/relaton_bib/xml_parser.rb:24:in `item_data'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nistbib-0.1.6/lib/nistbib/xml_parser.rb:13:in `item_data'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nistbib-0.1.6/lib/nistbib/xml_parser.rb:7:in `from_xml'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nistbib-0.1.6/lib/relaton/processor.rb:19:in `from_xml'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-0.3.4/lib/relaton/db.rb:141:in `bib_retval'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-0.3.4/lib/relaton/db.rb:162:in `check_bibliocache'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-0.3.4/lib/relaton/db.rb:42:in `fetch'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/ref.rb:104:in `fetch_ref'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/ref.rb:128:in `refitem'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/ref.rb:181:in `reference1'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/ref.rb:192:in `block (2 levels) in reference'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/ref.rb:191:in `each'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/ref.rb:191:in `block in reference'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/builder.rb:293:in `initialize'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/builder.rb:254:in `new'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/builder.rb:254:in `with'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/utils.rb:116:in `noko'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/ref.rb:190:in `reference'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/lists.rb:18:in `ulist'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/utils.rb:96:in `convert'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/asciidoctor-1.5.8/lib/asciidoctor/abstract_block.rb:75:in `convert'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/asciidoctor-1.5.8/lib/asciidoctor/list.rb:30:in `convert'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/asciidoctor-1.5.8/lib/asciidoctor/abstract_block.rb:84:in `block in content'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/asciidoctor-1.5.8/lib/asciidoctor/abstract_block.rb:84:in `map'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/asciidoctor-1.5.8/lib/asciidoctor/abstract_block.rb:84:in `content'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/section.rb:186:in `block in norm_ref_parse'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/builder.rb:391:in `insert'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/builder.rb:375:in `method_missing'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/section.rb:184:in `norm_ref_parse'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/section.rb:39:in `block in section'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/builder.rb:293:in `initialize'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/builder.rb:254:in `new'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/builder.rb:254:in `with'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/utils.rb:116:in `noko'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/section.rb:36:in `section'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/utils.rb:96:in `convert'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/asciidoctor-1.5.8/lib/asciidoctor/abstract_block.rb:75:in `convert'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/asciidoctor-1.5.8/lib/asciidoctor/abstract_block.rb:84:in `block in content'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/asciidoctor-1.5.8/lib/asciidoctor/abstract_block.rb:84:in `map'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/asciidoctor-1.5.8/lib/asciidoctor/abstract_block.rb:84:in `content'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/asciidoctor-1.5.8/lib/asciidoctor/document.rb:1261:in `content'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/base.rb:187:in `block in middle'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/builder.rb:391:in `insert'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/builder.rb:375:in `method_missing'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/base.rb:186:in `middle'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-rsd-1.1.4/lib/asciidoctor/rsd/converter.rb:101:in `block in makexml'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/builder.rb:293:in `initialize'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/builder.rb:254:in `new'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/builder.rb:254:in `with'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/utils.rb:116:in `noko'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-rsd-1.1.4/lib/asciidoctor/rsd/converter.rb:101:in `makexml'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-rsd-1.1.4/lib/asciidoctor/rsd/converter.rb:122:in `document'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/utils.rb:96:in `convert'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/asciidoctor-1.5.8/lib/asciidoctor/document.rb:1190:in `convert'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/asciidoctor-1.5.8/lib/asciidoctor.rb:1521:in `convert'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-0.3.11/lib/metanorma/input/asciidoc.rb:8:in `process'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-rsd-1.1.4/lib/metanorma/rsd/processor.rb:26:in `input_to_isodoc'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-0.3.11/lib/metanorma/compile.rb:102:in `process_input'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-0.3.11/lib/metanorma/compile.rb:17:in `compile'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-cli-1.1.8/lib/metanorma/cli/compiler.rb:24:in `compile_file'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-cli-1.1.8/lib/metanorma/cli/compiler.rb:12:in `compile'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-cli-1.1.8/lib/metanorma/cli/compiler.rb:16:in `compile'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-cli-1.1.8/lib/metanorma/cli/command.rb:33:in `compile'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-cli-1.1.8/lib/metanorma/cli.rb:56:in `start'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-cli-1.1.8/exe/metanorma:14:in `<top (required)>'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/bin/metanorma:23:in `load'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/bin/metanorma:23:in `<top (required)>'
ronaldtse commented 5 years ago

@andrew2net this is a critical error, can you help fix ASAP? Thanks!

andrew2net commented 5 years ago

@ronaldtse how can I reproduce the error?

ronaldtse commented 5 years ago

@andrew2net add this reference to a Metanorma document:

* [[[NISTCSF11,NIST Framework for Improving Critical Infrastructure Cybersecurity Version 1.1]]], _NIST Framework for Improving Critical Infrastructure Cybersecurity Version 1.1_
andrew2net commented 5 years ago

@ronaldtse seems the XML element /bibitem/date doesn't contain required on or from ement.

bdate = element date {
  attribute type { ( BibliographicDateType | text ) },
  (
(
element from { ISO8601Date },
element to { ISO8601Date }?
) |
element on { ISO8601Date }
)

}

I don't know where is the XML from and why it doesn't contain the required element. If you want I can remove this validation.

ronaldtse commented 5 years ago

Still failing but on a new error:

bundler: failed to load command: metanorma (/Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/bin/metanorma)
ArgumentError: invalid argument: stage (20)
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nistbib-0.1.7/lib/nistbib/document_status.rb:14:in `initialize'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nistbib-0.1.7/lib/nistbib/xml_parser.rb:26:in `new'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nistbib-0.1.7/lib/nistbib/xml_parser.rb:26:in `fetch_status'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-bib-0.1.5/lib/relaton_bib/xml_parser.rb:32:in `item_data'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nistbib-0.1.7/lib/nistbib/xml_parser.rb:13:in `item_data'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nistbib-0.1.7/lib/nistbib/xml_parser.rb:7:in `from_xml'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nistbib-0.1.7/lib/relaton/processor.rb:19:in `from_xml'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-0.3.4/lib/relaton/db.rb:141:in `bib_retval'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-0.3.4/lib/relaton/db.rb:162:in `check_bibliocache'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-0.3.4/lib/relaton/db.rb:42:in `fetch'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/ref.rb:104:in `fetch_ref'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/ref.rb:128:in `refitem'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/metanorma-standoc-1.1.9/lib/asciidoctor/standoc/ref.rb:181:in `reference1'
ronaldtse commented 5 years ago

This search string leads to here: https://www.nist.gov/publications/search?combine_1=NIST+Framework+for+Improving+Critical+Infrastructure+Cybersecurity+Version+1.1&title=&field_publication_authors_value=&field_nist_pub_series_tid=All&field_nist_org_tid=All&field_date_value%5Bmin%5D%5Bdate%5D=&field_date_value%5Bmax%5D%5Bdate%5D=&field_report_number_value=&sort_by=field_date_value&sort_order=DESC&items_per_page=25

Which leads to this document: https://www.nist.gov/publications/framework-improving-critical-infrastructure-cybersecurity-version-11

ronaldtse commented 5 years ago

I have a feeling that you could infer the document type from the DOI link:

https://doi.org/10.6028/NIST.CSWP.04162018

This indicates it is a "CSWP", and since it is not a draft, it is a "final".

andrew2net commented 5 years ago

@ronaldtse nistbib uses https://csrc.nist.gov/publications for search. See https://github.com/metanorma/relaton/issues/21 When I search the reference on the csrc.nist.gov I got this document https://csrc.nist.gov/publications/detail/white-paper/2017/12/05/cybersecurity-framework-v11/draft without any error. The error is caused by invalid stage, but I see we don't have validations for stage and substage anymore. So I'll remove the validations.

ronaldtse commented 5 years ago

@andrew2net now the error is this:

/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/bin/metanorma)
ArgumentError: Type "" is invalid.
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-bib-0.1.5/lib/relaton_bib/bibliographic_item.rb:182:in `initialize'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-bib-0.1.5/lib/relaton_bib/xml_parser.rb:284:in `new'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-bib-0.1.5/lib/relaton_bib/xml_parser.rb:284:in `block in fetch_relations'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/node_set.rb:204:in `block in each'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/node_set.rb:203:in `upto'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/node_set.rb:203:in `each'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-bib-0.1.5/lib/relaton_bib/xml_parser.rb:273:in `map'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-bib-0.1.5/lib/relaton_bib/xml_parser.rb:273:in `fetch_relations'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-bib-0.1.5/lib/relaton_bib/xml_parser.rb:34:in `item_data'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nistbib-0.1.8/lib/nistbib/xml_parser.rb:13:in `item_data'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nistbib-0.1.8/lib/nistbib/xml_parser.rb:7:in `from_xml'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nistbib-0.1.8/lib/relaton/processor.rb:19:in `from_xml'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-0.3.4/lib/relaton/db.rb:141:in `bib_retval'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-0.3.4/lib/relaton/db.rb:162:in `check_bibliocache'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/relaton-0.3.4/lib/relaton/db.rb:42:in `fetch'

If I remove that line, the compilation works.

ronaldtse commented 5 years ago

@andrew2net and we should obtain the "final" document, not the draft?

https://csrc.nist.gov/publications/detail/white-paper/2018/04/16/cybersecurity-framework-v11/final

The first two matches are 100% and are both "Final".

Screen Shot 2019-05-27 at 10 07 53 PM
ronaldtse commented 5 years ago

And we can disable "draft" in search:

Screen Shot 2019-05-27 at 10 08 46 PM
andrew2net commented 5 years ago

@ronaldtse the empty type cause error. I fixed parsing XML in relaton-bib. Pull relato-bib and check again, please.

So if there isn't draft in reference then we should search for final documents only, do we?

ronaldtse commented 5 years ago

So if there isn't draft in reference then we should search for final documents only, do we?

I think so. Thoughts @opoudjis ?

ronaldtse commented 5 years ago

Failure log of git relaton-bib:

ArgumentError: Type "" is invalid.
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/bundler/gems/relaton-bib-e108a0df4881/lib/relaton_bib/bibliographic_item.rb:183:in `initialize'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/bundler/gems/relaton-bib-e108a0df4881/lib/relaton_bib/xml_parser.rb:282:in `new'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/bundler/gems/relaton-bib-e108a0df4881/lib/relaton_bib/xml_parser.rb:282:in `block in fetch_relations'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/node_set.rb:204:in `block in each'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/node_set.rb:203:in `upto'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.5/lib/nokogiri/xml/node_set.rb:203:in `each'
  /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/bundler/gems/relaton-bib-e108a0df4881/lib/relaton_bib/xml_parser.rb:271:in `map'
ronaldtse commented 5 years ago

I cleared relaton cache and it's now working.

However:

  1. The document type is rendering in lowercase "nist" instead of "NIST"
  2. It's getting the 2nd Draft rather than the final.
Screen Shot 2019-05-27 at 10 37 48 PM
opoudjis commented 5 years ago

So if there isn't draft in reference then we should search for final documents only, do we? I think so. Thoughts @opoudjis ?

I don't have enough context, but yes, one should assume by default that the request is for a published document and not for a draft document, unless a draft is explicitly requested.

ronaldtse commented 5 years ago

Funny thing is I don't think we have a syntax for "find draft"... Do we @opoudjis ?

opoudjis commented 5 years ago

I assumed the draft was going to be part of the document ID that is fed into the gem; so you'd pass it the argument "NIST 800-53 (IPD)". And that that's what the draft in the website would actually be listed under. The drafts are a restricted vocabulary, since only public drafts are on the website: IPD, 2PD, 3PD, ... FPD.

Passing the draft as a separate parameter is, I suggest, asking for trouble.

ronaldtse commented 5 years ago

I agree with this. Do we have documentation for this? Can we add this? I feel that this belongs in Relaton?

opoudjis commented 5 years ago

I'm not aware of documentation; I've stayed at arm's length from relaton-nist. It certainly would need to be a feature of how relaton-nist works, and documented there.

I was just hoping that the documents were already labelled as "NIST 800-53 (IPD)". How are drafts identified currently?

andrew2net commented 5 years ago

@ronaldtse

The document type is rendering in lowercase "nist" instead of "NIST"

Do you mean docidentifier?

 <docidentifier type="nist">SP 800-189 (DRAFT)</docidentifier>

Will fix it. What types are possible?

andrew2net commented 5 years ago

@ronaldtse

And we can disable "draft" in search:

There is a problem with these searches options:

Do we need to get in search results final and final-withdrawn documents when we search for finals? If so then we need workaround it somehow.

ronaldtse commented 5 years ago

We need Final and Withdrawn, but prioritize Final over Withdrawn. In the search interface it is possible to search for these two types together.

andrew2net commented 5 years ago

Then the issue is, the option Withdrawn doesn't distinguish between final-withdrawn and draft-withdrawn documents. So we need to get a result from the site and filter documents in our gem.

ronaldtse commented 5 years ago

Yes, we will need to filter documents with our own gem. Their search isn't perfect.

opoudjis commented 5 years ago

@ronaldtse

The document type is rendering in lowercase "nist" instead of "NIST"

Do you mean docidentifier?

 <docidentifier type="nist">SP 800-189 (DRAFT)</docidentifier>

Will fix it. What types are possible?

I've fixed the lowercasing today.