Update crawl.py and scanner.py - Class Implementation

RahulDubey391 commented 1 year ago

Refactored crawl.py to have multiple classes separated by service type into separate files. Updated scanner.py to use crawler Classes.

Description

Refactoring applied to separate crawlers into separate classes based on Service-type in crawl.py. Each class is put into its own module and imported into a Composition Class in crawl.py. Further Composition Crawler Class is used in scanner.py

Changes Made

crawl.py - Separated each function under separate Classes based on Service-type.
crawl.py - Implemented a Composition Class for abstraction.
crawlers folder - all the separate classes into its own module file under crawlers folder
scanner.py - Imported and used crawler from Composition Class in crawl.py.

Checklist

[x] I have read and followed the contributing guidelines.
[ ] I have tested my changes thoroughly and they work as expected.
[ ] I have added necessary tests for the changes made.
[x] I have updated the documentation to reflect the changes made.
[ ] My code follows the project's coding style and standards.
[x] I have added appropriate commit messages and comments for my changes.

Related Issues

This update solves the issue #2 partially.

Additional Notes

Extensive testing is required to check classes and method implementations

Feel free to modify and customize this template as needed to fit the specific needs of the GCP Scanner project.

mshudrak commented 1 year ago

Hi Rahul, all tests are failing. Have you tested the new code? I would also recommend to first submit a proposal on what/how you want to refactor, what benefit it provides to us and discuss it on the corresponding issue. Perhaps, create a proposal.

RahulDubey391 commented 1 year ago

Hi Rahul, all tests are failing. Have you tested the new code? I would also recommend to first submit a proposal on what/how you want to refactor, what benefit it provides to us and discuss it on the corresponding issue. Perhaps, create a proposal.

Hi @mshudrak , These tests are failing because in unit test file, the changes are not made to consider the class implementation. But you are correct, I'll put this as a proposal for the final application, also I'll include Parallelization proposal in the same proposal. Meanwhile, I'll try to see if I can update the unit tests as well.

I'll keep this PR in queue so that I can mention the code structure in proposal application. Thanks for the guidance!!!

RahulDubey391 commented 1 year ago

@mshudrak Hi, I have a question regarding the proposal. For this refactoring PR do I have to include code as well in Proposal? If so should I prepare the proposal as Doc file or .Md file?

mshudrak commented 1 year ago

Nah, just provide a link/links to PRs you made. We accept proposals in any format you like.

RahulDubey391 commented 1 year ago

Nah, just provide a link/links to PRs you made. We accept proposals in any format you like.

Thanks!!! I'll mention this PR in the proposal. Good day @mshudrak

mshudrak commented 1 year ago

I am archiving this one. Noted on the application.

google / gcp_scanner