Research Crawler Algo

[JeremyBarbosa]

Requirements

• Choose based on speed, but that can also run in the limited browser extension environment.
• Preferably single threaded, but if you find something requiring parallel work perhaps we can implement Web Workers: https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers

Defintion of Done Provide a white paper describing the algorithm that we will implement

[JeremyBarbosa]

It seems that based on http://people.cse.nitc.ac.in/sites/default/files/aviralnigam/files/web_crawling_algorithms.pdf A* Search is the best.

[mylee1995]

https://stackoverflow.com/questions/35042798/get-dom-tree-with-javascript

This seems like a good resource in configuring algorithm for extracting DOM tree

[anthonychu00]

Yeah and we can just extract the tags from that to get links. The A* algorithm is straightforward to implement serially if we are using that. I'm just trying to figure out how exactly we're grading the relevancy of a page like it says in the paper.

I don't assume we're iterating more than once to search for lyrics right? If so we can probably ignore the section on Adaptive A*.