Closed aryanagrawal22 closed 8 months ago
Thanks for your PR. I left some suggestions. PTAL.
Thank you for going through my PR, and reviewing it. I think, I have done the relevant changes as asked. Please let me know in case any others are needed.
Yes, fair point. I have made the changes, changed the name to has_config_dependency from crawler_config
Pushed with the changes 😁
Description
crawler_config
in base classICrawler
which is returns bool value, wheredefault=False
crawler_config
in the base class needs to be overridden in the sub class to return True.Note : Currently as I was checking all the crawlers, I have just seen that config is being used only inside the
StorageBucketsCrawler
crawler. So only its value is set as True in its base class.Changes Made
ICrawler
has a new property called ascrawler_config
which returns the abstracted private variable_config_depndency
which is set to False by default.StorageBucketsCrawler
class has overriddencrawler_config
to return True.get_crawl
function in scanner.py has a check now calling thecrawler_config
to decide to send config or not.Testing
Local testing done with logs denoting that where config file is sent and where it is not. (Removed the logs after the testing)
This check denotes that only
StorageBucketsCrawler
is being sent the config file