search

webrecorder / warcio.js

JS Streaming WARC IO optimized for Browser and Node
MIT License
30 stars 6 forks source link

indexer: add IndexerOffsetLength type to decouple custom indexing without a WARCParser #60

Closed ikreymer closed 11 months ago

ikreymer commented 11 months ago

Allow for indexing without requiring a WARCParser, eg. if WARC records are generated dynamically / already parsed. The indexer just requires an IndexerOffsetLength object, which tracks 'offset' and 'recordLength'

WARCParser also derives from IndexerOffsetLength

http:<header> access fix: when access an http headers field, if headers are a map, convert to Headers to support case-insensitive get()