To support analysis mixed data, please add a new method RecordLoader.loadArcOrWarc(source, sc) which is identical to RecordLoader.loadArc(source, sc) and RecordLoader.loadWarc(source, sc), but decides for each file if it is treated as ARC or WARC file.
To distinguish ARC and WARC files one could use the file extensions (.arc and .arc.gz for ARC, .warc and .warc.gz for WARC).
Most of our harvested data is stored in WARC files, but some older is still there as ARC. When we try to perform an analysis using
it will fail when it reaches the first ARC file:
To support analysis mixed data, please add a new method
RecordLoader.loadArcOrWarc(source, sc)
which is identical toRecordLoader.loadArc(source, sc)
andRecordLoader.loadWarc(source, sc)
, but decides for each file if it is treated as ARC or WARC file.To distinguish ARC and WARC files one could use the file extensions (.arc and .arc.gz for ARC, .warc and .warc.gz for WARC).