kProcessor 1 can be easily used, and it's a dependency in the current version of kSpider and other tools. The hashmap of the index and others can be optimized to use the parallel version of the flat_hash_map with the maximum number of sub-maps to easily handle very large indexes.
Increasing the number of sub-maps in kProcessor v2 will also be good optimization (even if it's used in single-threaded applications).
mr-eyes
commented
2 years ago
kProcessor 1 can be easily used, and it's a dependency in the current version of kSpider and other tools. The hashmap of the index and others can be optimized to use the parallel version of the flat_hash_map with the maximum number of sub-maps to easily handle very large indexes.
Increasing the number of sub-maps in kProcessor v2 will also be good optimization (even if it's used in single-threaded applications).