issues
search
spatie
/
crawler
An easy to use, powerful crawler implemented in PHP. Can execute Javascript.
https://freek.dev/308-building-a-crawler-in-php
MIT License
2.49k
stars
356
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Check original URL against depth tree when visited link is a redirect
#467
superpenguin612
closed
5 days ago
5
Using a maximum depth prevents redirected links from being crawled
#466
superpenguin612
closed
5 days ago
1
Cookies in clientOptions
#465
davidgeisler1998
closed
5 days ago
2
Class "Spatie\Crawler\CrawlObservers\CrawlObserver" not found in production
#463
ahmadnajmdev
closed
3 months ago
0
Add request factory
#462
asbiin
closed
3 weeks ago
2
Add Laravel/Illuminate 11 Support
#461
Jubeki
closed
5 months ago
4
Fix wording in documentation
#460
adamtomat
closed
5 months ago
1
feat: custom link parser
#458
Velka-DEV
closed
6 months ago
4
Allow Symfony 7.x
#456
thecaliskan
closed
7 months ago
1
Cannot install under php 8.1.2
#453
maverickchangithub
closed
8 months ago
0
Fix return type
#452
riesjart
closed
8 months ago
1
Fix for crawler stopping issue
#451
buismaarten
closed
8 months ago
7
Crawler stops after `www`-version redirect to non-`www` version
#450
buismaarten
closed
6 months ago
0
Define only needed methods in observer implementation
#449
buismaarten
closed
8 months ago
1
Depth tree structure
#447
Xaelp
closed
9 months ago
1
Update dependencies
#446
koertho
closed
10 months ago
0
Check if rel attribute contains nofollow
#445
robbinbenard
closed
1 year ago
1
Add link text
#444
freekmurze
closed
1 year ago
0
Crawler simply stops
#442
mrtnmueller
closed
1 year ago
0
Modify URLs before crawl
#440
codealfa
closed
1 year ago
5
Add PHP 8.2 Support
#437
patinthehat
closed
1 year ago
1
Call to undefined method GuzzleHttp\Client::getConfig()
#434
tommusrhodus
closed
1 year ago
1
Make setParseableMimeTypes prevent URLs from being crawled
#432
arturzealves
closed
1 year ago
3
Custom/extendable `CrawlUrl`
#429
rudiedirkx
closed
1 year ago
0
`CrawlObserver` and `CrawlProfile` interfaces
#428
rudiedirkx
closed
1 year ago
0
Network.getAllCookies
#426
futureweb
closed
1 year ago
3
Extract link text
#423
simonschaufi
closed
1 year ago
0
Follow all redirects, including HTML meta and Javascript
#422
rzb
closed
1 year ago
0
Fix Puppeteer installation instructions link
#421
DvDty
closed
1 year ago
1
Request Type
#418
Dr-Maspix
closed
2 years ago
0
Update .gitattributes
#417
angeljqv
closed
2 years ago
1
Add the ability to change the default baseUrl scheme
#402
arnissolle
closed
2 years ago
1
Feat/convert phpunit tests to pest
#401
mansoorkhan96
closed
2 years ago
1
setConcurrency not working if using browsershoot
#399
lenusch
closed
2 years ago
0
Maximum depth not working in subsequent request
#397
dacdam
closed
1 year ago
3
Fix issue #395
#396
Rezyan
closed
2 years ago
1
Deprecations of implemented interfaces since PHP 8.1 for `CrawlObserverCollection` class
#395
Rezyan
closed
2 years ago
0
Keep only guzzlehttp/psr7 v2.0
#392
flangofas
closed
2 years ago
3
Replace stream_for with Utils::streamFor
#391
NicolasMica
closed
2 years ago
1
Update CrawlRequestFailed.php
#388
asefsoft
closed
2 years ago
1
Browsershot timeout unhandled
#387
MichaelNino
closed
2 years ago
1
Very poor documentation.
#384
bohdanbaida
closed
2 years ago
0
ConnectException's message is not transfered to crawlFailed
#382
jszczypk
closed
2 years ago
0
Honeypot
#380
mitmelon
closed
2 years ago
4
[v7.0.2] Undefined method stream_for
#379
flangofas
closed
2 years ago
1
http referer of the crawled url
#378
aimagician
closed
2 years ago
2
Laravel and Vapor
#377
Andrew-Hike
closed
2 years ago
1
Best way to limit similar urls
#376
aaronbauman
closed
2 years ago
2
Test against php 8.1
#375
sergiy-petrov
closed
2 years ago
1
Can't install with Laravel 8.57
#374
morawcik
closed
2 years ago
6
Next