search

vezaynk / Sitemap-Generator-Crawler

PHP script to recursively crawl websites and generate a sitemap. Zero dependencies.
https://www.bbss.dev
MIT License
241 stars 92 forks source link

Follow frames #39

Closed Thyra closed 7 years ago

Thyra commented 7 years ago

Hi, I really like this script and I'd like to use it on a current project of mine, but I discovered two bugs that have to be resolved first. The first one is the same as in #34 and the second one is that the script is currently ignoring frames. If you take the site http://rolf-herbold.de/ for example, the index page only consists of three frames that then contain further links. But if you run the crawler on that domain, it only crawls the index page and doesn't follow the links in the frames. I think this should be relatively easy to solve with another regular expression searching for frames and following their src attribute, but I haven't tested it.

vezaynk commented 7 years ago

Frames are deprecated in HTML5, consider getting a new website.

However, you are correct in asserting that it shouldn't be difficult. Going to take a look.

Thyra commented 7 years ago

I know, but it's not my website; I want to analyze a million sites and some of them still use frames. And they have to be crawled correctly, too.