Closed ianmilligan1 closed 8 years ago
We might have a potential bug (or it could be a misunderstanding of the code base). This script works well with an ARC file, using loadArc:
loadArc
val r = RecordLoader.loadArc(arc, sc) .keepValidPages() .map(r => ExtractTopLevelDomain(r.getUrl)) .countItems() .take(10)
But WARC files fail when replacing loadArc with loadWarc:
loadWarc
val r = RecordLoader.loadWarc(warc, sc) .keepValidPages() .map(r => ExtractTopLevelDomain(r.getUrl)) .countItems() .take(10)
results in an empty array.
Using sample data uploaded here.
It was a bug indeed. I've merged the fix.
Great stuff, @aliceranzhou – just confirmed that it's all working as it should. Many thanks.
Thanks @aliceranzhou!
We might have a potential bug (or it could be a misunderstanding of the code base). This script works well with an ARC file, using
loadArc
:But WARC files fail when replacing
loadArc
withloadWarc
:results in an empty array.
Using sample data uploaded here.