loadWarc generating empty arrays

[ianmilligan1]

We might have a potential bug (or it could be a misunderstanding of the code base). This script works well with an ARC file, using loadArc:

val r = 
RecordLoader.loadArc(arc, 
sc) 
.keepValidPages() 
.map(r => ExtractTopLevelDomain(r.getUrl)) 
.countItems() 
.take(10) 

But WARC files fail when replacing loadArc with loadWarc:

val r = 
RecordLoader.loadWarc(warc, 
sc) 
.keepValidPages() 
.map(r => ExtractTopLevelDomain(r.getUrl)) 
.countItems() 
.take(10) 

results in an empty array.

Using sample data uploaded here.

[aliceranzhou]

It was a bug indeed. I've merged the fix.

[ianmilligan1]

Great stuff, @aliceranzhou – just confirmed that it's all working as it should. Many thanks.

[ruebot]

Thanks @aliceranzhou!