search

crawler-commons / url-frontier

API definition, resources and reference implementation of URL Frontiers
Apache License 2.0
44 stars 11 forks source link

Configs read.thread.num and write.thread.num are ignored for IgniteService #73

Closed michaeldinzinger closed 1 year ago

michaeldinzinger commented 1 year ago

When starting the IgniteService, the two configuration arguments read.thread.num and write.thread.num are ignored. In the IgniteService constructor (line 130), the configuration map is not forwarded to the underlying DistributedFrontierService and, subsequently, not forwarded to the underlying AbstractFrontierService. So specifying read.thread.num and write.thread.num makes no difference for the Ignite imlementation.

michael@pc:~/Desktop/Git/url-frontier/service$ java -Xmx2G -cp target/urlfrontier-service-*.jar crawlercommons.urlfrontier.service.URLFrontierServer implementation=crawlercommons.urlfrontier.service.ignite.IgniteService ignite.path=/some_path/data_frontier read.thread.num=1 write.thread.num=1
16:38:33.319 [main] INFO  c.u.service.AbstractFrontierService - Available processor(s) 12
16:38:33.321 [main] INFO  c.u.service.AbstractFrontierService - Using 3 threads for reading from queues
16:38:33.321 [main] INFO  c.u.service.AbstractFrontierService - Using 3 threads for writing to queues
...
michaeldinzinger commented 1 year ago

This issue is solved incidentally by the changes in PR #79