Refactor IEV reference rewrite code into separate relaton-iev gem

[ronaldtse]

Can this be passed to @CAMOBAP795 or @andrew2net ?

[opoudjis]

Yes

[opoudjis]

The code in question is the following routines in metanorma-standoc/lib/asciidoctor/standoc/cleanup_ref.rb:

      # converts generic IEV citation to citation of IEC 60050-n
      # assumes IEV citations are of form
      # <eref type="inline" bibitemid="a" citeas="IEC 60050">
      # <locality type="clause"><referenceFrom>101-01-01</referenceFrom></locality></eref>
      def linksIev2iec60050part(xmldoc)
        parts = Set.new()
        xmldoc.xpath("//eref[@citeas = 'IEC 60050:2011'] | "\
                     "//origin[@citeas = 'IEC 60050:2011']").each do |x|
          cl = x&.at("./locality[@type = 'clause']/referenceFrom")&.text || next
          m = /^(\d+)/.match cl || next
          parts << m[0]
          x["citeas"] = x["citeas"].sub(/60050/, "60050-#{m[0]}")
          x["bibitemid"] = "IEC60050-#{m[0]}"
        end
        parts
      end

      # replace generic IEV reference with references to all extracted
      # IEV parts
      def refsIev2iec60050part(xmldoc, parts, iev)
        new_iev = ""
        parts.sort.each do |p|
          hit = @bibdb&.fetch("IEC 60050-#{p}", nil, keep_year: true) || next
          new_iev += hit.to_xml.sub(/ id="[^"]+"/, %{ id="IEC60050-#{p}"})
          date = hit.dates[0].on.year
          xmldoc.xpath("//*[@citeas = 'IEC 60050-#{p}:2011']").each do |x|
            x["citeas"] = x["citeas"].sub(/:2011$/, ":#{date}")
          end
        end
        iev.replace(new_iev)
      end

      # call after xref_cleanup and origin_cleanup
      def iev_cleanup(xmldoc)
        iev = xmldoc.at("//bibitem[docidentifier = 'IEC 60050:2011']") || return
        parts = linksIev2iec60050part(xmldoc)
        refsIev2iec60050part(xmldoc, parts, iev)
      end

The proposal is to export the functionality of at least some of linksIev2iec60050part and refsIev2iec60050part to a separate gem.

Metanorma references IEV as a single source, IEC 60050. In reality, IEC 60050 is dozens of separate documents, each with their own part number (corresponding to the starting number of the clause reference; so IEV clause 101-2-3 is actually IEC 60050-101 clause 101-2-3.) This functionality helps expand the single IEV reference out into all the references it actually corresponds to in the given document.

The first routine is rewriting IEC 60050 links to include a part number. The second routine is updating the generic year of all IEC 60050 references, to be the year of the part of IEV specific to that reference.

This is not a bibliographical fetch like the other relaton-iev fetches; this is a metanorma-specific rewrite function, that depends on an initial fetch from relaton of the individual IEC 600050 documents. (For that reason, I'm not convinced it should be called relaton-iev at all.)

@ronaldtse, this was raised 6 months ago. Could you confirm that you still want this functionality exported to a separate gem—and why? (Given that it is not doing the same thing that other relaton gems are doing.)

[ronaldtse]

Obviously these two methods should go into relaton-iev.

Didn't we agree that the relaton gem is supposed to handle citation issues from now on?

relaton-iso fetches ISO items. relaton-iev fetches IEV items.

There is another question that we haven't discussed, it is the import of terms from term bases, of which IEV is one.

[opoudjis]

There is another question that we haven't discussed, it is the import of terms from term bases, of which IEV is one.

And we're not going to discuss it either. STOP EXPANDING SCOPE.

The two functions as described are not fetching IEV terms, which can be injected into Metanorma (that is what the existing iev gem does). They are detecting IEV references, fetching them, manipulating them to be IEC 60050 references, and manipulating any cross references to IEV to be IEC 60050 cross references. Which means this is very, very different to what the other relaton- gems do. The other gems do not manipulate entire metanorma documents. (And in fetching terms as opposed to references, the iev gem is not doing what the other relaton- gems do either.)

[ronaldtse]

I concur that “iev” and relaton-iev are two different things. “iev” should fetch terms. “relaton-iev” fetches references for “iev” terms.

[opoudjis]

“relaton-iev” fetches references for “iev” terms.

... and manipulates the document to convert them into IEC 60050 references. But OK.

[andrew2net]

@ronaldtse should we create a new gem relaton-iev?

[ronaldtse]

Yes please!

[andrew2net]

@opoudjis I don't see where @bibdb is defined. Seems it always undefined. Am I wrong?

[opoudjis]

@bibdb is defined in metanorma-standoc, I don't know what you're referring to:

metanorma-standoc/lib/asciidoctor/standoc/base.rb

        @bibdb = Relaton::DbCache.init_bib_caches(
          local_cache: local,
          flush_caches: node.attr("flush-caches"),
          global_cache: global)

This is the bedrock gem of the entire stack, and I am now VERY concerned about what you're about to do. Please discuss with me what you're intending to do: any relaton-iev gem should be doing very, very little. (Which is why I think this gem is pointless.)

[andrew2net]

@opoudjis @bibdb used in the function:

  # replace generic IEV reference with references to all extracted
  # IEV parts
  def refsIev2iec60050part(xmldoc, parts, iev)
    new_iev = ""
    parts.sort.each do |p|
      hit = @bibdb&.fetch("IEC 60050-#{p}", nil, keep_year: true) || next
      new_iev += hit.to_xml.sub(/ id="[^"]+"/, %{ id="IEC60050-#{p}"})
      date = hit.dates[0].on.year
      xmldoc.xpath("//*[@citeas = 'IEC 60050-#{p}:2011']").each do |x|
        x["citeas"] = x["citeas"].sub(/:2011$/, ":#{date}")
      end
    end
    iev.replace(new_iev)
  end

but it isn't defined in Asciidoctor::Standoc::Cleanup. I'm going to make a new gem relaton-iev and put these 3 functions into RelatonIev module. I'm doing to add bibdb argument to the function.

[andrew2net]

here is the new gem https://github.com/metanorma/relaton-iev @ronaldtse @opoudjis please check it. If all ok I'll release it.

[opoudjis]

Will review under https://github.com/metanorma/relaton-iev/issues/2