add routeviews data

[bhuffaker]

Create a script that will add the papers from https://www.routeviews.org/routeviews/index.php/papers/ and link them to a Routeviews datasets.

• [x] make a modified version of scripts/catalog-dataset-summary-download.py
  • it will data/download_urls.txt for a list of URL to download and filenames to store the files in as CSV with '#' comments

    # comment
    URL, filename
    https://users.caida.org/~dataadm/catalog/catalog-dataset-summary.jsonl, data/catalog-dataset-summary.jsonl
    https://www.routeviews.org/routeviews/index.php/papers/, data/routeviews-data.txt

  • it will check the age of the file on disk and download a fresh copy it is too old
• [x] write a script that will parse the data/routeviews-data.txt and create placeholder papers for each paper that link to the new dataset:routeviews object in sources/paper
  • this will take the form of data

[trdavidt]

https://www.routeviews.org/routeviews/index.php/papers/

How do I scrape the data in the table on the next page? There doesn't seem to be a url for the "next" button. (edit: resolved, all the table data was already in the html for this page but it was hidden)

[bhuffaker]

All the data is already on the page, the Javascript is hiding it when it presents the table.

On Mar 14, 2023, at 10:31 AM, David Tran @.***> wrote:

https://www.routeviews.org/routeviews/index.php/papers/

How do I scrape the data in the table on the next page? There doesn't seem to be a url for the "next" button.

— Reply to this email directly, view it on GitHub https://github.com/CAIDA/catalog-data/issues/553#issuecomment-1468531702, or unsubscribe https://github.com/notifications/unsubscribe-auth/AECPT7PE7IYQKZOYQFURXXLW4CTPNANCNFSM6AAAAAAU5ECFXM. You are receiving this because you authored the thread.

[trdavidt]

Awaiting review, I added one of the placeholder paper objects in the commit out of 1009 total papers. The remaining paper placeholders should look the same. Do the fields look okay, or should there be more?

[trdavidt]

Also, there was a small issue with the data/routeviews-data.txt generated by my script: some papers have the authors' names separated by comma instead of semicolon (which most papers use). I corrected this by hand.

[bhuffaker]

I think we should be adding to the YAML external Paper file.

• https://github.com/CAIDA/catalog-data/blob/master/data/data-papers.yaml

The data-papers.yaml file is manually maintained, so you should put these automated files into a separate file. Put the YAML output into data/data-papers-routeviews.yaml. Then update the Makefile and scripts/externallinks_placeholder.py so it can read input multiple files. Make sure that it checks for duplicates. It should read the manual file first, and produce a warning if the duplicate is in the automated file.

[trdavidt]

What is the externallinks_placeholder scripts used for?

[bhuffaker]

Makefile:

• add the URL to the stop of the file

  EXTERNAL_ROUTEVIEWS_URL=https://www.routeviews.org/routeviews/index.php/papers/ 
  EXTERNAL_ROUTEVIEWS_FILE=data/data/data-papers.yaml

• add scripts/externallinks_routeviews.py to the Makefile. This script = downloads and converts https://www.routeviews.org/routeviews/index.php/papers/ into the same format of data/data-papers.yaml. It should only download if https://www.routeviews.org/routeviews/index.php/papers is older then 5 days.

• then expanded externallinks_placeholder so it can take multiple files as input. It will parse both data/data-papers.yaml and data/data-papers-routeviews.yaml, since they are the same format, to create sources/paper/(paper).json files for each paper in those files.

add here: https://github.com/CAIDA/catalog-data/blob/master/Makefile#L69

[trdavidt]

This script = downloads and converts https://www.routeviews.org/routeviews/index.php/papers/ into the same format of data/data-papers.yaml.

Here is my data/data-papers-routeviews.yaml generated by this script. What should the format be for the MARKER field in the yaml? It seems like the markers in in /data/data-papers.yaml are formatted like YYYY_last_f_<some substring of URL>.

[bhuffaker]

I have merged your 554 branch into this your 553. Please work in this branch. I have updated the Makeflie to have the routeviews target.

external:
    python3 scripts/download_url.py -O ${EXTERNAL_ROUTEVIEWS_HTML} ${EXTERNAL_ROUTEVIEWS_URL}
ifneq ("$(wildcard ${EXTERNAL_ROUTEVIEWS_HTML})", :"")
    python3 scripts/routeviews-parse.py -O ${EXTERNAL_ROUTEVIEWS_FILE} ${EXTERNAL_ROUTEVIEWS_HTML}
    python3 scripts/externallinks_placeholder.py ${EXTERNAL_MANUAL_FILE} ${EXTERNAL_ROUTEVIEWS_FILE}
else
    python3 scripts/externallinks_placeholder.py ${EXTERNAL_MANUAL_FILE}
endif

This depends on three scripts:

• scripts/download_url.py (already added exists)
  • downloads the file from routeviews_url and stores it into routeviews_yaml
• scripts/external_routeviews_parse.py (needs to be created from your existing scripts)
  • parse the routeviews_html file and produces routeviews_yam
  • scripts/externallinks_placeholder.py
  • needs to be updated so it parses two files
  • needs to check for duplicate , if the paper exists in both files, keep only the metadata from the first file, but merge the TOPKEYS

notes:

• You shouldn't be changing anything in data/paper-data.yaml. At this point, overwrite the data/data-paper.yaml with the copy from master.
• don't commit ${EXTERNAL_ROUTEVIEWS_FILE} or ${EXTERNAL_ROUTEVIEWS_HTML}
• once you have the code working, remove the existing files down to just adding
  • scripts/download_url.py
  • scripts/external_routeviews_parse.py

Let me know if you have any questions.