spatie / robots-txt

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers
https://spatie.be/en/opensource/php
MIT License
219 stars 36 forks source link

Can't create `Robots` from string, must use url or file #24

Closed rezen closed 4 years ago

rezen commented 4 years ago

In the Robots constructor it seems the source is assumed to be a uri, and can not be the content of the robots txt. In certain implementations, a library user may want to handle fetching of robots.txt. What if instead the logic for reading source was moved into the static create function and the constructor accepted RobotsTxt? It would mean nothing changes for users of create but the constructor would use a different signature.

Example

  public function __construct(string $userAgent = null, $robotsTxt = null)
    {
        $this->userAgent = $userAgent;
        $this->robotsTxt = $robotsTxt;
    }

    public static function create(string $userAgent = null, string $source = null): self
    {
       $robotsTxt = $source ? RobotsTxt::readFrom($source) : null;
        return new self($userAgent, $robotsTxt);
    }
spatie-bot commented 4 years ago

Dear contributor,

because this issue seems to be inactive for quite some time now, I've automatically closed it. If you feel this issue deserves some attention from my human colleagues feel free to reopen it.