There are two changes in this PR that will help me use the package more efficiently.
1) Be able to provider Robots with a RobotsTxt object directly. I have access to the robots.txt file in memory and don't want to write it to disk first. With this change, I can create a RobotsTxt object from the string in memory and create a Robot with it.
2) Calling mayIndex() and mayFollowOn() on a Robot class causes two requests to the remote server. This change pulls the file_get_contents() request up one level and prevents RobotsMeta and RobotsHeader from each having to do their own requests.
freekmurze
commented
5 months ago
There are two changes in this PR that will help me use the package more efficiently.
1) Be able to provider Robots with a RobotsTxt object directly. I have access to the robots.txt file in memory and don't want to write it to disk first. With this change, I can create a RobotsTxt object from the string in memory and create a Robot with it.
2) Calling
mayIndex()andmayFollowOn()on a Robot class causes two requests to the remote server. This change pulls thefile_get_contents()request up one level and preventsRobotsMetaandRobotsHeaderfrom each having to do their own requests.