search

crawler-commons / url-frontier

API definition, resources and reference implementation of URL Frontiers
Apache License 2.0
47 stars 12 forks source link

Remove lock in putURLs for RocksDB service, implements #28 #29

Closed jnioche closed 3 years ago

jnioche commented 3 years ago

see #28 tested on a large crawl and has greatly improved the performance: no more threads waiting for hundreds of seconds on the lock