roach-php / core

The complete web scraping toolkit for PHP.
https://roach-php.dev
1.37k stars 70 forks source link

How to login and then scrap data from a page that requires auth? #200

Closed skiopey closed 10 months ago

skiopey commented 10 months ago

I want to login into a website which is a Laravel10 app that has basic auth. and then access a page that requires the user to be logged in.

Thank you for this amazing package, you are the best man.

Edit:

Checkout the solution in the next comment. 👇 👇

skiopey commented 10 months ago

i figured it out.

public array $downloaderMiddleware = [
    RequestDeduplicationMiddleware::class,
    CookieMiddleware::class,
];

public function parse(Response $response): Generator
{
    $csrfToken = $response->filter('meta[name="csrf-token"]')->attr('content');

    $request = new Request(
        'POST',
        'http://127.0.0.1:8001/login',
        [$this, 'parseContent'],
        [
            'headers' => [
                'X-CSRF-TOKEN' => $csrfToken,
            ],
            'form_params' => [
                'email' => 'admin@demo.com',
                'password' => 'password',
            ],
        ],
    );

    yield ParseResult::fromValue($request);
}

public function parseContent(Response $response): Generator
{
    yield $this->request('GET', 'http://127.0.0.1:8001/billing', 'ParseBilling');
}

public function ParseBilling(Response $response): Generator
{
    dd($response->filter('h3')->text());
}