search

ilyasozkurt / mobilephone-brands-and-models

A database includes mobilephone manufacturers and their models.
GNU General Public License v3.0
78 stars 41 forks source link

Some URLs are not scraped #14

Open Ro-Den opened 1 year ago

Ro-Den commented 1 year ago

URLs with the following syntax are not scraped:

  1. .com/_ - don't know why, e.g. Umidigi F1. Maybe the scraper omits the part after / and downloads the main page.

  2. [g] - throws the following error: GuzzleHttp\Exception\TooManyRedirectsException. It's just one model: vivo Y20s [G]. I tried to change the URL to %5Bg%5D but to no avail.

P.S.: Glad you are back and with a cute website.

ilyasozkurt commented 1 year ago

Hello @Ro-Den,

Thank you for your feedback about my website.

Can you please share full URL with me that occurs this redirect loop?

Ro-Den commented 1 year ago

  1. www.gsmarena.com/_umidigi_f1-9524.php

  2. www.gsmarena.com/vivoy20s[g]-10847.php

Ro-Den commented 1 year ago

The following has resolved the issues:

  1. trim($name);

What a great idea to terminate instructions with smilies ):

  1. I replaced Illuminate\Support\Facades\Http with curl 'cause Laravel's HTTP Client is too tricky to debug ):