TikTok's webpages takes forever and end up failing the whole task

CreaZyp154 commented 2 years ago

It happens quite a lot; there's a task with tiktok in it, and it hangs. It's the one that blocks everything I've checked. It happened with different subdomains as well like vt. and vm.. TikTok is not blocked for me. It works perfectly when opening in the browser.

Screenshot

Details

Running on VirtualBox 6.1 Seesaw version: 0.10.3 Python version: 3.9.9 (main, Dec 21 2021, 10:35:05) [GCC 8.3.0]

Suggestions for fix

I don't have a full solution but I have some ideas:

It might be due to an anti-bot script
It may also NOT be an anti-bot script so, trying to figure out why it behaves like that
Add a timeout to each element and re-queue them to the tracker (or abandon idk)
Create custom code for tiktok's domain

madprogramer commented 1 year ago

bump

CreaZyp154 commented 1 year ago

I haven't seen this happen again, so it may have been fixed, idk...

Arkiver2 commented 1 year ago

From the image the problematic URL would be https://www.tiktok.com/@saki_15166/live?_r=1&sec_user_id=MS4wLjABAAAADFRCvXBvgK4_u-gP9zPzOvTwljFpMa3YrXZtPr0OYj9Dn7Lr8lT3nkg0EaAmujk4&share_app_id=1180&share_link_id=a29bab9d-02ca-43c7-87e3-baf57c5e7187&source=h5_t&ugbiz_name=Live&user_id=7064603045267506178

Arkiver2 commented 1 year ago

I can't reproduce this problem, and with comment https://github.com/ArchiveTeam/urls-grab/issues/13#issuecomment-1333342750, I'm going to close this for now.

ArchiveTeam / urls-grab