use WET files from CommonCrawl

[dportabella]

CommonCrawl has the WET files, which are WARC files where HTML response has been converted to plain text (and non html pages has been removed).

Is it possible to use WET files with warcbase?

I tried as follows:

val archives = RecordLoader.loadArchives(in, sc)

val htmlPages =
  if (isWET)
    archives
      .map(r => r.getContentString)
  else
    archives
      .keepValidPages()
      .map(r => RemoveHTML(r.getContentString))

If in = "/data/sample.wet.gz", it complains with invalid exception. As I see that the format is quite similar, I tried renaming the file to /data/sample.warc.gz. However, htmlPages.count is zero when isWET is true.

Any clue?

[jrwiebe]

RecordLoader.loadArchives() only supports ARC and WARC files. We use the webarchive-commons library for processing these, which does not have WET support.

On Tue, Sep 27, 2016 at 12:06 PM, David Portabella <notifications@github.com

wrote:

CommonCrawl has the WET files http://commoncrawl.org/the-data/get-started/, which are WARC files where HTML response has been converted to plain text (and non html pages has been removed).

Is it possible to use WET files with warcbase?

I tried as follows:

val archives = RecordLoader.loadArchives(in, sc)

val htmlPages = if (isWET) archives .map(r => r.getContentString) else archives .keepValidPages() .map(r => RemoveHTML(r.getContentString))

If in = "/data/sample.wet.gz", it complains with invalid exception. As I see that the format is quite similar, I tried renaming the file to /data/sample.warc.gz. However, htmlPages.count is zero when isWET is true.

Any clue?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/lintool/warcbase/issues/250, or mute the thread https://github.com/notifications/unsubscribe-auth/AEaUjdlX_oiDTzg1Kt8xIWkqR4aDYmVDks5quT7-gaJpZM4KH2rv .

[dportabella]

shall I implement this feature (to handle WET archives)?

is this a feature that you would include in Warcbase library?

do you see any shortcoming/problem/comment on this?

[jrwiebe]

I'll leave it to @lintool and @ianmilligan1 to comment on this feature's desirability.

I would suggest that if you do decide to implement a WET reader, which seems pretty straightforward, do it as a fork of webarchive-commons and see if they accept it.

On Tue, Sep 27, 2016 at 12:24 PM, David Portabella <notifications@github.com

wrote:

shall I implement this feature (to handle WET archives)?

is this a feature that you would include in Warcbase library?

do you see any shortcoming/problem/comment on this?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/lintool/warcbase/issues/250#issuecomment-249918238, or mute the thread https://github.com/notifications/unsubscribe-auth/AEaUjRDd48w4B2gv-b5odlhbJ_SarMC9ks5quUNXgaJpZM4KH2rv .

[ianmilligan1]

Thanks for this. My sense is that only CommonCrawl uses WET files, right?

I don't think we would frame it as a priority for our own development time, but if you were able to get WETs incorporated into webarchive-commons or warcbase, we would love to see it. With the combo of reading from S3 directly (as your other issue suggested) and WET functionality, I think that'd make CommonCrawl analysis very useful.

[dportabella]

So, it seems that we can use the webarchive-commons library as it is: https://github.com/iipc/webarchive-commons/issues/66#issuecomment-250450623

We need only two changes from warcbase: 1- accept the wet.gz extension 2- filter WARC-Type by conversion instead of by response https://github.com/lintool/warcbase/blob/8ba16e842c299c2955bedb1062b3ed8f1aa95190/warcbase-core/src/main/scala/org/warcbase/spark/matchbox/RecordLoader.scala#L37

What do you think?

[ianmilligan1]

Interesting! Do you want to give it a try, maybe altering RecordLoader and IngestFiles.java, see if it works with your WETs?

[dportabella]

Yup, I'll try that.