Create storage crawling graph implementation for headless crawler

projectdiscovery / katana

A next-generation crawling and spidering framework.

MIT License

10.3k stars 539 forks source link

Open Ice3man543 opened 2 years ago

Ice3man543 commented 2 years ago

Implement a crawl graph with relevant method for navigation and events storage and retrieval

Ice3man543 commented 2 years ago

Either we can have a graph with States as Nodes and Events as Edges
Or we can have a graph with States as Edges containing event information in them. The first one sounds better

Create an interface for the storage crawl graph, decide on methods/algorithms to implement + choose an implementation in golang.

brenocss commented 1 year ago

call graph would be amazing for visualization as well

derekperkins commented 1 year ago

Is the plan to make a pluggable storage layer? A vote for Vitess (sharded MySQL) from me :)