http referer of the crawled url

spatie / crawler

An easy to use, powerful crawler implemented in PHP. Can execute Javascript.

https://freek.dev/308-building-a-crawler-in-php

MIT License

2.51k stars 357 forks source link

http referer of the crawled url #378

Closed aimagician closed 2 years ago

aimagician commented 2 years ago

Hello

Can I get an HTTP referer of the crawled URL, in this case, it is the parent URL?

parent page
- first child page(FCP)
  - child page (i am crawling child page, I need an URL of the FCP)

Thank you

Redominus commented 2 years ago

As you can check https://github.com/spatie/crawler/blob/9596cd0c9e8d61ab31ab5d9bd961b2671323bba5/src/CrawlObservers/CrawlObserver.php#L27-L31 An observer receives the url where the current url was found. You could create an observer to store this data.

spatie-bot commented 2 years ago

Dear contributor,

because this issue seems to be inactive for quite some time now, I've automatically closed it. If you feel this issue deserves some attention from my human colleagues feel free to reopen it.