Save Corpus ids from PublicApi references matched to PDF Bib Entries

[stupendousC]

• From 4th checkbox in parent task [pipe corpus ids through scholarphi to s2airs] dev/local end to end #35326
  • Parent task has 3 PRs in these 3 repos: scholarphi --> scholarphi-pipeline --> s2airs
  • This PR is for that first repo.
  • Where scholarphi takes the bib entries from a PDF, and the list of references from Public API.
  • Title-matching done on these 2 groups so we know which bib entry corresp to which paper in our corpus.
  • The new thing here is that we also want to save the corpus IDs.

FILES I TOUCHED

• The files I touch in this PR came from studying the flow in Scholarphi.
  • The most important thing is the updated PublicAPI endpoint that provides me with the corpus ids to begin with.
  • Example of what this looks like:
  • 
• Back when I erroneously submited this PR to the main branch, I had a bunch of comments showing in the 1st commit "initial thoughts" for what goes where, but now that I've switched base to chi-2021-demo, they're gone. Oh well, no big deal.
• Recommended viewing order (following the flow mentioned above):
  • fetch_s2_data.py & its test
  • Public Api now returns references' corpus IDs, which is saved in references.csv on L152
  • common/types.py
  • Data classes now store corpus IDs.
  • Also added new string types to make CitationData more understandable.
  • resolve_bibitems.py & its test
  • Now a bib item is matched to an S2 corpus paper, I want to save that corpus ID.
  • entities/citations/types.py
  • upload_citations.py
  • Didn't realize the final "citations.json" is the goal, instead of the intermediate various CSVs. Anyway, CP ids should be in there now.
  • process_pdf.py
  • Not sure why this is still there, it's not being used anymore, esp since the import on L14 refers to something that was deleted in July 2020.
  • Added TODO to consider deleting it later.

TESTING 👍 && 👍

• BOTH approaches below need to be green/thumbsUp before merging!!
• IDEA 1 = E2E testing of scholarphi --> scholarphi-pipeline --> s2airs (as seen in reference PR) 👍
  • DEFINITELY will do this, but need to change stuff in the other 2 repos to prove this.
  • Make sure the entire assembly line works before I merge any PRs for any of the 3 repos.
• IDEA 2 = Test just this scholarphi portion 👍
  • Will involve the scholarphi/data-processing container on my machine
  • docker pull <latest official image> (image)
  • from data-processing/... docker run --rm -it -v $(pwd):/data-processing 60bb4741 /bin/bash
    • mount my file changes to override behavior
  • IDEA 2A = pytest --all as seen in (readme) <-- 👎 but it's out of my hands and nothing to do with me
  • Failed 7 tests (even without my changes), I saw code 404/NotFound.
    • I think the S3 bucket it mentioned doesn't even exist anymore.
    • Seems like these 3 files are what the tests want.
    • So I went digging in S3 and see what buckets we have, I think it's either scholarphi-public or scholarphi4nlp
    • So I went with scholarphi-public S3, and I subbed it in the code, cleaned rm -rf ./cache/AI2020_model, and then pytest --all
    • Now I see 7x code 403/Forbidden... but its' a public bucket 😖
    • My PR doesn't touch the Definitions stuff anyway, so it's unlikely I broke it. Still... it'd be nice to see it pass.
    • TRY: maybe do the scholarphi4nlp bucket url instead. Will re-attempt if I feel like it later.
  • IDEA 2B = pytest is just a quick test, that doesn't deal with the Definitions stuff seen above. It totally passed. 👍
  • IDEA 2C = Run pipeline for just 1 paper 👍
  • Also seen in readme, but don't follow those directions.
  • Instead I used the following, with arxiv_id=1512.02595...

    # from inside the docker container:
    python scripts/run_pipeline.py \
     --one-paper-at-a-time \
     --arxiv-ids 1512.02595 \
     --output-forms file \
     --output-dir data \
     --entities citations

  • Full log printout here for fun:

    root@6cf004dd1cc2:/data-processing# python scripts/run_pipeline.py    --one-paper-at-a-time    --arxiv-ids 1512.02595    --output-forms file    --output-dir data --entities citations
    wandb: WARNING W&B installed but not logged in.  Run `wandb login` or set the WANDB_API_KEY env variable.
    2023-02-07 03:28:21,145 [INFO]: The following commands will be run, in this order: ['fetch-arxiv-sources', 'fetch-s2-metadata', 'unpack-sources', 'compile-tex', 'normalize-tex', 'compile-normalized-tex', 'raster-pages', 'extract-bibitems', 'resolve-bibitems', 'locate-bounding-boxes-for-citation-fragments', 'locate-citations', 'upload-citations']
    2023-02-07 03:28:21,146 [INFO]: Running pipeline for paper 1512.02595
    2023-02-07 03:28:21,148 [INFO]: Launching command fetch-arxiv-sources
    2023-02-07 03:28:32,695 [INFO]: Finished running command fetch-arxiv-sources
    2023-02-07 03:28:32,698 [INFO]: Launching command fetch-s2-metadata
    2023-02-07 03:28:32,699 [INFO]: Issuing request to S2 @ api.semanticscholar.org
    2023-02-07 03:28:36,295 [INFO]: Finished running command fetch-s2-metadata
    2023-02-07 03:28:36,298 [INFO]: Launching command unpack-sources
    2023-02-07 03:28:36,423 [INFO]: Finished running command unpack-sources
    2023-02-07 03:28:36,424 [INFO]: Launching command compile-tex
    2023-02-07 03:28:39,286 [INFO]: Finished running command compile-tex
    2023-02-07 03:28:39,288 [INFO]: Launching command normalize-tex
    2023-02-07 03:28:39,410 [INFO]: Finished running command normalize-tex
    2023-02-07 03:28:39,412 [INFO]: Launching command compile-normalized-tex
    2023-02-07 03:28:43,432 [INFO]: Finished running command compile-normalized-tex
    2023-02-07 03:28:43,433 [INFO]: Launching command raster-pages
    2023-02-07 03:28:44,960 [INFO]: Finished running command raster-pages
    2023-02-07 03:28:44,962 [INFO]: Launching command extract-bibitems
    2023-02-07 03:28:45,930 [INFO]: Finished running command extract-bibitems
    2023-02-07 03:28:45,932 [INFO]: Launching command resolve-bibitems
    /usr/local/lib/python3.7/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.
    warnings.warn(msg, category=FutureWarning)
    2023-02-07 03:28:46,219 [WARNING]: Could not find a sufficiently similar reference for bibitem ModernGPU of paper 1512.02595
    2023-02-07 03:28:47,044 [WARNING]: Could not find a sufficiently similar reference for bibitem Elsen2015 of paper 1512.02595
    2023-02-07 03:28:48,347 [WARNING]: Could not find a sufficiently similar reference for bibitem NervanaGPU of paper 1512.02595
    2023-02-07 03:28:49,428 [WARNING]: Could not find a sufficiently similar reference for bibitem szegedy2014googlenet of paper 1512.02595
    2023-02-07 03:28:49,754 [INFO]: Paper has 70 references, able to match 66 reference(s) to S2 data.
    2023-02-07 03:28:49,755 [INFO]: Finished running command resolve-bibitems
    2023-02-07 03:28:49,757 [INFO]: Launching command locate-bounding-boxes-for-citation-fragments
    2023-02-07 03:29:39,475 [INFO]: Finished running command locate-bounding-boxes-for-citation-fragments
    2023-02-07 03:29:39,476 [INFO]: Launching command locate-citations
    2023-02-07 03:29:39,706 [INFO]: Finished running command locate-citations
    2023-02-07 03:29:39,708 [INFO]: We will be writing output to files.
    2023-02-07 03:29:39,708 [INFO]: Launching command upload-citations
    2023-02-07 03:29:39,728 [WARNING]: Missing S2 match for bibitem with key ModernGPU for paper 1512.02595
    2023-02-07 03:29:39,729 [WARNING]: Missing S2 match for bibitem with key Elsen2015 for paper 1512.02595
    2023-02-07 03:29:39,730 [WARNING]: Missing S2 match for bibitem with key NervanaGPU for paper 1512.02595
    2023-02-07 03:29:39,731 [WARNING]: Missing S2 match for bibitem with key szegedy2014googlenet for paper 1512.02595
    2023-02-07 03:29:39,731 [WARNING]: Missing S2 match for bibitem with key Elsen2015 for paper 1512.02595
    2023-02-07 03:29:39,732 [WARNING]: Missing S2 match for bibitem with key szegedy2014googlenet for paper 1512.02595
    2023-02-07 03:29:39,732 [WARNING]: Missing S2 match for bibitem with key NervanaGPU for paper 1512.02595
    2023-02-07 03:29:39,733 [WARNING]: Missing S2 match for bibitem with key ModernGPU for paper 1512.02595
    2023-02-07 03:29:39,733 [INFO]: Saving to file...
    2023-02-07 03:29:39,734 [INFO]: About to write 184 entity infos to data/citations.json (version: v0).
    2023-02-07 03:29:39,755 [INFO]: Finished running command upload-citations

    • Log printouts make sense with each other, 4 of those bib entries did not get matched to a reference seen in Public API.
  • OMG YES!!!!!!!
    • 

[stupendousC]

PARKING this as a DRAFT

Just for now. So I don't accidentally try to merge this.

Will flip back to review status once the E2E testing across scholarphi --> scholarphi-pipelines --> s2airs is done.

RN I'm 95% confident I'm done with this repo. Moving onto the scholarphi-pipelines & s2airs now. Will come back and change things here as necessary.

BTW: TC CI for this branch.

[ca16]

A more general question - just want to double check that for Reference, SerializableReference and BibItemMatch, to which this PR adds new fields, all their usages have been accounted for?