related: #58 #59 #60

Any one who is interested in this PR is welcome to review even when it is still a draft. Thank you!

Background

OpenWebText is mostly for a comparable baseline to GPT-2 for HTML. In other words, if we don't need such a benchmark, any part of C4 with HTML would do. Either way, we need HTML from WARC of Common Crawl (CC).

Status

No actual changes yet, just a plan and some caveats.

[ ] Get every CC record as fast as possible
- [ ] Parse OpenWebText URL filenames for their year and month
- [ ] Use year-month as from_ts and to for narrowing the CC indexes down (see the 1st snippet below)
  - Otherwise CC Indexer API will look for the most recent records. Not only a waste of time but also confusing when there are same-URL records with updated or even disappeared (page moved) contents.
- [ ] Run queries in parallel (but not too much to violate CC's policy)
[ ] Extract HTML from each CC record's WARC on Jean Zay
- [ ] Upload CC records to Jean Zay
- [ ] Locate each record by WARC path and offset
- [ ] Use warcio to extract HTML

Tentative code snippets

For a test case of https://www.defense.gov/News/News-Releases/News-Release-View/Article/621692/dod-identifies-air-force-casualties/ Based on the corresponding file of the OpenWebText URL list, we already know that the URL is from 2018-08.

The library is cdx_toolkit because other famous choices are either inconvenient (not library but command line) or insufficient (e.g., michaelharms/comcrawl is usually faster but no filters for 301).

import cdx_toolkit

cdx = cdx_toolkit.CDXFetcher(source="cc")

def yield_cc_indexed_info(url, limit=10):
    for obj in cdx.iter(
        url,
        filter="!=status:301",
        from_ts="20180801000000",  # TODO: extract parameter
        to="20180831235959",  # TODO: extract parameter
        limit=limit
    ):
        yield obj

from timeit import timeit

url = "https://www.defense.gov/News/News-Releases/News-Release-View/Article/621692/dod-identifies-air-force-casualties/"
cc_record_gen = yield_cc_indexed_info(url)
my_globals = globals()
cc_record_data = []
my_globals.update({'cc_record_data': cc_record_data})
one_record_time = timeit("cc_record_data.extend(x.data for x in cc_record_gen)", number=1, globals=my_globals)
assert one_record_time <= 10
print(cc_record_data)

[{'digest': 'IT2PGYG6YQN2DGZPHHMDR2DNPLYIHGJK',
  'encoding': 'UTF-8',
  'filename': 'crawl-data/CC-MAIN-2018-34/segments/1534221215487.79/warc/CC-MAIN-20180820003554-20180820023554-00336.warc.gz',
  'languages': 'eng',
  'length': '26391',
  'mime': 'text/html',
  'mime-detected': 'text/html',
  'offset': '847331857',
  'status': '200',
  'timestamp': '20180820005801',
  'url': 'https://www.defense.gov/News/News-Releases/News-Release-View/Article/621692/dod-identifies-air-force-casualties/',
  'urlkey': 'gov,defense)/news/news-releases/news-release-view/article/621692/dod-identifies-air-force-casualties'}]

The expected outcome in this case

HTML of https://www.defense.gov/News/News-Releases/News-Release-View/Article/621692/dod-identifies-air-force-casualties/

```html DOD Identifies Air Force Casualties > U.S. DEPARTMENT OF DEFENSE > News Release View

This information is provided for historical purposes only. It may contain outdated information and links may no longer function. Please contact the DoD Webmaster if you have any questions about this archive.

IMMEDIATE RELEASE

DOD Identifies Air Force Casualties

Oct. 3, 2015
News Release
Release No: NR-384-15

The Department of Defense announced today the deaths of six airmen who were supporting Operation Freedom's Sentinel. They died Oct. 2 in the crash of a C-130J Super Hercules aircraft at Jalalabad Airfield, Afghanistan. The cause of the accident is currently under investigation.

Killed were:

Capt. Jonathan J. Golden, 33, of Camarillo, California.

Capt. Jordan B. Pierson, 28, of Abilene, Texas.

Staff Sgt. Ryan D. Hammond, 26, of Moundsville, West Virginia.

Senior Airman Quinn L. Johnson-Harris, 21, of Milwaukee, Wisconsin.

All four were assigned to the 39th Airlift Squadron, Dyess Air Force Base, Texas. For more information, media may contact the 7th Bomb Wing Public Affairs Office at 325-696-2863.

Also killed were:

Senior Airman Nathan C. Sartain, 29, of Pensacola, Florida.

Airman 1st Class Kcey E. Ruiz, 21, of McDonough, Georgia.

Both were assigned to the 66th Security Forces Squadron, Hanscom Air Force Base, Massachusetts. For more information, media may contact the 66th Air Base Group Public Affairs Office at 781-225-1686.