search

LAW-Unimi / BUbiNG

The LAW next generation crawler.
http://law.di.unimi.it/software.php#bubing
Apache License 2.0
85 stars 24 forks source link

Multiwarc #9

Open guillaumepitel opened 7 years ago

guillaumepitel commented 7 years ago

New pull request with just the MultiWarc store : adds a new Store with multiple WARC files output instead of just one big.

Configurable by maxRecordsPerWarc and maxSecondsBetweenDumps

vigna commented 7 years ago

It is my understanding that this code has been by now either merged or implemented by us. Am I right?

mapio commented 7 years ago

Not yet merged. And now has a conflict, so it can't be merged as is.

vigna commented 7 years ago

Yep. The other changes were ported in differently (e.g., there's an option to bring back self-signed certificates). From what I see, we need this changeset only:

https://github.com/LAW-Unimi/BUbiNG/pull/9/commits/3eeac0af2c6163079e72cdc8adb9ceaf5401110f

The rest has been already integrated.