search

cloudviz / agentless-system-crawler

A tool to crawl systems like crawlers for the web
Apache License 2.0
116 stars 44 forks source link

Documentation on how to extend the crawler ecosystem #377

Open lamchakchan opened 5 years ago

lamchakchan commented 5 years ago

Description

This agentless crawler ecosystem looks very interesting. I would like to contribute to it but I'm new to both this project and to Python. Can someone familiar with this project add some information around how the plugin architecture works such as prerequisites and class interface contracts. How this project is structure. If it is structured like a standard Python app, maybe a link to some source of truth on what this looks like. And if there are any Python specific tools/concepts/libs that this project depends or focuses around, that would be good to detail out as well.

sahilsuneja1 commented 5 years ago

See Section 3.1.5 here: http://www.cs.toronto.edu/~sahil/suneja-mw17-2-TR.pdf That would give you some information about the crawler architecture. Also, some more arch info below: plugin-arch