Closed gauntface closed 6 years ago
@jeffmcaffer thanks for the review comments.
I've switched everything to Q where I was originally creating native promises. I haven't gone to the extent of wrapping the promises returned by the Google Cloud Storage package, let me know if you would prefer it though.
Added some tests for the new delta store - I didn't see a coverage reporting anywhere, I've just been running npm run test
to check everything is WAI.
Let me know if you'd like any other changes / tests / etc.
Regarding testing - I am running this on a container to make sure it's crawling and working as I'd expect so far, but obviously doesn't rule out stupid mistakes on my part, so feel free to question any of this PR.
Gentle nudge @jeffmcaffer :)
After talking with @jeffmcaffer a little last week this adds support for Google Cloud Storage to GHCrawler (Both as a storage provider and a delta provider).
I've tried to mimic the azure storage model as much as possible, couple of things I'm not sure of:
initializeSubsystemOptions()
in the crawlerFactory. TheCRAWLER_DELTA_PROVIDER
value was being picked up by thegetDefaultOptions()
method, but would get filtered out byinitializeSubsystemOptions()
. Commenting out the early return fixes this but I'm sure I'm missing the purpose of that if statement.Happy to change any of the approaches, code style etc, just let me know where the issues are :)