spatie robots-txt issues

spatie / robots-txt

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers

https://spatie.be/en/opensource/php

MIT License

219 stars 36 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

Update phpunit/phpunit requirement from ^9.0|^10.0 to ^11.4.1

#51 dependabot[bot] opened 20 hours ago
1
Update phpunit/phpunit requirement from ^9.0|^10.0 to ^11.4.0

#50 dependabot[bot] closed 1 day ago
2
Update phpunit/phpunit requirement from ^9.0|^10.0 to ^11.3.6

#49 dependabot[bot] closed 3 weeks ago
2
Bump stefanzweifel/git-auto-commit-action from 4 to 5

#48 dependabot[bot] closed 3 weeks ago
1
Bump actions/checkout from 2 to 4

#47 dependabot[bot] closed 3 weeks ago
1
Fixes "case-sensitive" URI matching for Disallow rules in robots.txt

#46 mattfo0 closed 2 weeks ago
1
Robots.txt "Disallow" URI matching should be case-sensitive

#45 mattfo0 closed 2 weeks ago
0
Add missing use statement for InvalidArgumentException

#44 remcom closed 2 months ago
1
optionally allow partial matches and global groups

#43 resohead closed 5 months ago
1
Add some efficiencies to prevent unnecessary requests

#42 tsjason closed 5 months ago
1
Fix deprecation message

#41 buismaarten closed 10 months ago
1
Discussion: Would it make sense to replace file_get_contents with guzzle

#40 ivangrozni closed 1 year ago
3
Update PHPUnit config to latest schema

#39 patinthehat closed 1 year ago
1
Add PHP 8.2 Support

#38 patinthehat closed 1 year ago
1
php8.1-compatibility: Make package php8.1 compatible - do not pass nu…

#37 petarbasic closed 2 years ago
2
Update .gitattributes and ignore php_cs cache

#36 angeljqv closed 2 years ago
1
Match meta tag for single quotes, often used by Wordpress sites

#35 gjportegies closed 2 years ago
1
file_get_contents($source) throws an InvalidArgumentException on Websites with expired Certificates

#34 osthafen closed 2 years ago
1
Adding namespace to class

#33 dougblackjr closed 3 years ago
1
Added x-robots-tag: none. Ref #30

#32 ihah closed 3 years ago
1
Add PHP 8-only Support (v2)

#31 patinthehat closed 3 years ago
1
Not working properly with: x-robots-tag: none

#30 nnerijuss closed 3 years ago
3
Handle multiple user-agent

#29 Kixell-NicolasJardillier closed 3 years ago
1
Several user-agent for one Disallow directive does not work

#28 Kixell-NicolasJardillier closed 3 years ago
5
Support for PHP 8.0 + Move to GitHub actions

#27 pascalbaljet closed 3 years ago
1
user agent is converted to lowercase for matching

#26 muhci closed 4 years ago
2
Custom UserAgent mismatches due to parseUserAgent()

#25 muhci closed 4 years ago
4
Can't create `Robots` from string, must use url or file

#24 rezen closed 4 years ago
1
Fix find robots meta tag line if minified code

#23 stephane-monnot closed 4 years ago
0
Apply fixes from StyleCI

#22 freekmurze closed 4 years ago
0
fix headers checking (nofollow, noindex) for custom userAgent

#21 RobinDev closed 4 years ago
2
Prevent parsing of large HTML blobs to reach out-of-memory exceptions

#20 mattiasgeniar closed 4 years ago
5
Out of Memory exception on large pieces of HTML

#19 mattiasgeniar closed 4 years ago
8
Implement Allow directive

#18 BenMorel closed 4 years ago
3
Implement wildcard, end-of-string, query string

#17 BenMorel closed 5 years ago
1
Fix null/false confusion

#16 BenMorel closed 5 years ago
1
False positives on bare domains with no trailing slash

#15 mikemike closed 5 years ago
1
Feature/fix parsing robotstxt urls with keywords

#14 Redominus closed 5 years ago
1
Fixes (see commit msg)

#13 RobinDev closed 5 years ago
5
Fix Wildcard check in Robots.txt and Headers

#12 RobinDev closed 5 years ago
1
Fix nofollow or noindex check for Headers

#11 RobinDev closed 5 years ago
1
Fixes "case-insensitive"-Rule in X-robots-tag

#10 RobinDev closed 5 years ago
1
Fixes "case-insensitive"-Rule in robots.txt

#9 Enaah closed 5 years ago
2
Robots.txt fields should be case-insensitive

#8 Redominus closed 5 years ago
2
Test against PHP 7.3

#7 amitmerchant1990 closed 5 years ago
1
Credits linked to non existing user

#6 koenhoeijmakers closed 5 years ago
1
Apply fixes from StyleCI

#5 freekmurze closed 5 years ago
0
Test enhancement

#4 peter279k closed 6 years ago
1
Apply fixes from StyleCI

#3 freekmurze closed 6 years ago
0
Code review

#2 freekmurze closed 6 years ago
0