search

crawler-commons / url-frontier

API definition, resources and reference implementation of URL Frontiers
Apache License 2.0
47 stars 12 forks source link

Scan of table failing for queues which have been completed #16

Closed jnioche closed 3 years ago

jnioche commented 3 years ago

When resurrecting an existing Rockdb

Caused by: java.lang.RuntimeException: Scan of table found missing queue -.alienvault at crawlercommons.urlfrontier.service.rocksdb.RocksDBService.recoveryQscan(RocksDBService.java:151) at crawlercommons.urlfrontier.service.rocksdb.RocksDBService.(RocksDBService.java:103) ... 15 more

The reason for it is that the queue object wasn't created if nothing was scheduled - i.e. the queue was finished. This is a bug.