The crawler was unreasonably slow for how little data it actually pushed. This is one of the first crawlers and therefore used only single get functions, which can get pretty slow in nested loops. Now it uses batch functions.
Also removed the sys.exit in case of a failure and replaced with an exception.
How Has This Been Tested?
I compared that the created nodes/relationships are the same as with the old crawler.
Types of changes
[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
Crawler slow. Now crawler fast.
Description
The crawler was unreasonably slow for how little data it actually pushed. This is one of the first crawlers and therefore used only single get functions, which can get pretty slow in nested loops. Now it uses batch functions.
Also removed the
sys.exit
in case of a failure and replaced with an exception.How Has This Been Tested?
I compared that the created nodes/relationships are the same as with the old crawler.
Types of changes
Checklist: