search

workeffortwaste / horseman

The detailed update and issue repository for the Horseman crawler.
https://gethorseman.app/
18 stars 0 forks source link

Big sites can't be exported as JSON #110

Closed workeffortwaste closed 1 year ago

workeffortwaste commented 1 year ago

Very large crawls are unable to be exported. Data needs to be streamed to files to overcome this.

workeffortwaste commented 1 year ago

Added NDJSON export in 0.2.15