workflows: review reference extraction in article workflow

[jmartinm]

Current article workflow extracts references for arXiv papers:

ENHANCE_RECORD = [
    # Article Processing
    # ==================
    IF(
        is_arxiv_paper,
        [
            arxiv_fulltext_download,
            arxiv_plot_extract,
            arxiv_refextract,
            arxiv_author_list("authorlist2marcxml.xsl"),
        ]
    ),

TODO:

• [x] arxiv_refextract task needs to be reviewed to see if references are being added into the correct reference fields using latest hep schema.
• [x] Review that dojson is correctly outputting references and check final MarcXML (field 999)
• [x] Make sure we want to be extracting references on labs and what happens on legacy when the automatic task to extract references finds the new paper (it should ignore it if it already has references)

BONUS POINTS:

• [x] If all is good, use refextract to extract references when the user inputs them in free form (in the references textarea)

[rikirenz]

@jmartinm Do we have tests for this kind of extraction?

[jmartinm]

I don't think so, the only thing I see is this mock in the integration test for workflows.

Unit tests are needed for https://github.com/inspirehep/inspire-next/blob/d0cb6ba6d761279a76b849990dadd8160156eecb/inspirehep/modules/refextract/tasks.py#L86

[kaplun]

But, aren't we anyway discarding references before sending to legacy?

[jmartinm]

But, aren't we anyway discarding references before sending to legacy?

It seems so https://github.com/inspirehep/inspire-next/blob/master/inspirehep/modules/workflows/workflows/article.py#L278

I guess they were just extracted to show something in the Holding Pen that helps decide if the article is core/non-core.

Question on this PR was to know if we want to keep doing that or we can extract the references on Labs (and possibly as well the ones given as free text by the user) and send them to legacy.

[kaplun]

Nope we can't yet. We need to port refextract to use journal from Labs, but for the time being this has not yet been done: https://github.com/inspirehep/refextract/issues/3

[david-caro]

@jmartinm if nobody is doing this, move to ready and unassign so anyone can pick it up :+1:

[jacquerie]

This has happened in https://github.com/inspirehep/inspire-next/pull/2558.