search

openzim / wikihow

WikiHow scraper
https://download.kiwix.org/zim/wikihow/
GNU General Public License v3.0
16 stars 2 forks source link

wikihow does fetch article CSS properly (at least it does not uses a retry mechanism) #163

Open benoit74 opened 6 months ago

benoit74 commented 6 months ago

Task: https://farm.openzim.org/pipeline/3481640e-bb1e-4e47-9a82-edb4aa2d13b2/debug

Logs:

[MainThread::2024-03-21 15:03:33,763] DEBUG:17394/269089
[MainThread::2024-03-21 15:03:33,763] INFO:>> Article:Start-a-Toyota-Prius-(US)
[MainThread::2024-03-21 15:03:34,887] DEBUG:URL `https://www.wikihow.com/extensions/wikihow/IMR/assets/chat_wH.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/f/f2/CommunityAvatar3.png/-crop-104-104-104px-CommunityAvatar3.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/f/f5/CommunityAvatar4.png/-crop-104-104-104px-CommunityAvatar4.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/e/e1/Disable-Reverse-Beep-in-a-Toyota-Prius-Step-20.jpg/-crop-342-184-245px-Disable-Reverse-Beep-in-a-Toyota-Prius-Step-20.jpg` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/a/aa/Check-the-Hybrid-System-on-a-Prius-Step-6.jpg/-crop-342-184-245px-Check-the-Hybrid-System-on-a-Prius-Step-6.jpg` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/e/ec/Glovebox_filter_8.JPG/-crop-342-184-204px-Glovebox_filter_8.JPG` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/3/3e/Change-the-HID-Headlights-on-a-2007-Prius-%28Without-Removing-Bumper%29-Step-27.jpg/-crop-342-184-245px-Change-the-HID-Headlights-on-a-2007-Prius-%28Without-Removing-Bumper%29-Step-27.jpg` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/f/fb/Turbo-Charge-a-Corolla-Step-21.jpg/-crop-342-184-245px-Turbo-Charge-a-Corolla-Step-21.jpg` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/b/be/Drive-a-Prius-in-Ice-or-Snow-Step-10-Version-2.jpg/-crop-342-184-245px-Drive-a-Prius-in-Ice-or-Snow-Step-10-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/f/f5/Tom_eisenberg.png/-crop-100-100-99px-Tom_eisenberg.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/f/f5/Tom_eisenberg.png/-crop-100-100-99px-Tom_eisenberg.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/b/b0/CommunityAvatar2.png/-crop-72-72-72px-CommunityAvatar2.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/0/0a/CommunityAvatar1.png/-crop-72-72-72px-CommunityAvatar1.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/4/4e/Empath.png/-crop-127-140-127px-Empath.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/7/76/Canva-love-me-love-me-not.jpg/-crop-127-140-127px-Canva-love-me-love-me-not.jpg` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/4/45/Aesthetic-Quiz.png/-crop-127-140-127px-Aesthetic-Quiz.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/e/ed/Formal_dance.jpg/-crop-127-140-127px-Formal_dance.jpg` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/4/48/Sleep-quiz.png/-crop-127-140-127px-Sleep-quiz.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/0/06/Zodiac-Dating-Quiz2.png/-crop-127-140-127px-Zodiac-Dating-Quiz2.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/e/e1/Disable-Reverse-Beep-in-a-Toyota-Prius-Step-20.jpg/-crop-127-140-127px-Disable-Reverse-Beep-in-a-Toyota-Prius-Step-20.jpg` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/a/aa/Check-the-Hybrid-System-on-a-Prius-Step-6.jpg/-crop-127-140-127px-Check-the-Hybrid-System-on-a-Prius-Step-6.jpg` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/e/ec/Glovebox_filter_8.JPG/-crop-127-140-127px-Glovebox_filter_8.JPG` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/1/14/LR23-D-Amirra-Fidgets-Happily.png/-crop-127-140-127px-LR23-D-Amirra-Fidgets-Happily.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/6/67/Contact-Your-MP-Step-2.jpg/-crop-127-140-127px-Contact-Your-MP-Step-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/2/2c/Love-You-vs-I-Love-You-Step-13-Version-2.jpg/-crop-127-140-127px-Love-You-vs-I-Love-You-Step-13-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/7/71/Accept-Blame-when-You-Deserve-It-Step-11-Version-2.jpg/-crop-127-140-127px-Accept-Blame-when-You-Deserve-It-Step-11-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/3/38/Use-a-Moka-Pot-Step-10.jpg/-crop-127-140-127px-Use-a-Moka-Pot-Step-10.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/6/62/Clean-White-Converse-Step-35-Version-2.jpg/-crop-127-140-127px-Clean-White-Converse-Step-35-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/6/62/Be-Less-Emotional-Step-17-Version-2.jpg/-crop-127-140-127px-Be-Less-Emotional-Step-17-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/7/7b/Know-if-You%27re-Dating-a-Toxic-Person-Step-12.jpg/-crop-127-140-127px-Know-if-You%27re-Dating-a-Toxic-Person-Step-12.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/d/da/Add-Sounds-to-Discord-Soundboard-Step-16.jpg/-crop-127-140-127px-Add-Sounds-to-Discord-Soundboard-Step-16.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/a/ae/Get-Free-Clothes-from-Shein-Step-10.jpg/-crop-127-140-127px-Get-Free-Clothes-from-Shein-Step-10.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/7/76/Eye_color_quiz.png/-crop-127-140-127px-Eye_color_quiz.png` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/9/9e/Phonetically-Spell-My-Name-Step-9.jpg/-crop-127-140-127px-Phonetically-Spell-My-Name-Step-9.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/b/bf/Access-YouTube-at-School-Step-32.jpg/-crop-127-140-127px-Access-YouTube-at-School-Step-32.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/d/db/Use-Watercolor-Pencils-Step-13-Version-2.jpg/-crop-127-140-127px-Use-Watercolor-Pencils-Step-13-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/e/ec/Read-an-Egg-Cleanse-Step-1.jpg/-crop-127-140-127px-Read-an-Egg-Cleanse-Step-1.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/9/91/Make-Your-Dog-Stop-Sleeping-in-Your-Bed-Step-15-Version-4.jpg/-crop-127-140-127px-Make-Your-Dog-Stop-Sleeping-in-Your-Bed-Step-15-Version-4.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/b/b8/Be-More-Spontaneous-Step-10-Version-2.jpg/-crop-127-140-127px-Be-More-Spontaneous-Step-10-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/9/92/Cut-Hair-Straight-Step-22.jpg/-crop-127-140-127px-Cut-Hair-Straight-Step-22.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/2/23/Get-in-Shape-Step-16.jpg/-crop-127-140-127px-Get-in-Shape-Step-16.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/e/e4/Cut-Side-Swept-Bangs-Step-20-Version-2.jpg/-crop-127-140-127px-Cut-Side-Swept-Bangs-Step-20-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/f/f2/Read-Body-Language-Step-13-Version-5.jpg/-crop-127-140-127px-Read-Body-Language-Step-13-Version-5.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/2/2f/Windows-11-Mobile-Hotspot-Save-New-Hotspot-Properties.png/-crop-127-140-127px-Windows-11-Mobile-Hotspot-Save-New-Hotspot-Properties.png` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/2/23/Exercise-at-Work-Every-Day-Step-1-Version-2.jpg/-crop-127-140-127px-Exercise-at-Work-Every-Day-Step-1-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/2/2a/Play-Basketball-Step-29-Version-4.jpg/-crop-127-140-127px-Play-Basketball-Step-29-Version-4.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/d/de/Fold-Boxer-Briefs-Step-10-Version-3.jpg/-crop-127-140-127px-Fold-Boxer-Briefs-Step-10-Version-3.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/d/df/Suspend-an-Object-in-Jello-Step-13.jpg/-crop-127-140-127px-Suspend-an-Object-in-Jello-Step-13.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/c/cb/Open-a-Durian-Step-7.jpg/-crop-127-140-127px-Open-a-Durian-Step-7.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/2/2c/Make-Ginger-Tea-or-Tisane-Step-9-Version-2.jpg/-crop-127-140-127px-Make-Ginger-Tea-or-Tisane-Step-9-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/5/50/Take-Lenses-Out-of-Glasses-Step-10.jpg/-crop-127-140-127px-Take-Lenses-Out-of-Glasses-Step-10.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/8/80/Cook-Chorizo-Step-24.jpg/-crop-127-140-127px-Cook-Chorizo-Step-24.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/5/52/Owl-Hooting-Meaning-Step-16.jpg/-crop-127-140-127px-Owl-Hooting-Meaning-Step-16.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/b/bc/Why-Do-Dogs-Sigh-Step-12-Version-3.jpg/-crop-127-140-127px-Why-Do-Dogs-Sigh-Step-12-Version-3.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/8/8a/Gryffindor-Traits-Step-17.jpg/-crop-127-140-127px-Gryffindor-Traits-Step-17.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/0/02/Know-when-Your-Computer-Was-Last-Used-Step-9.jpg/-crop-127-140-127px-Know-when-Your-Computer-Was-Last-Used-Step-9.jpg` already processed.
[MainThread::2024-03-21 15:03:34,890] DEBUG:URL `https://www.wikihow.com/images/thumb/3/39/Angel_Number_Quiz_image.png/-crop-127-140-127px-Angel_Number_Quiz_image.png` already processed.
[MainThread::2024-03-21 15:03:34,890] DEBUG:URL `https://www.wikihow.com/images/thumb/6/6a/Platonic-Marriage-Step-14.jpg/-crop-127-140-127px-Platonic-Marriage-Step-14.jpg` already processed.
[IMG-T-6::2024-03-21 15:03:35,062] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/f/f5/Start-a-Toyota-Prius-%28US%29-Step-3-Version-3.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-3-Version-3.jpg into ZIM::images/3139121941-v4-460px-Start-a-Toyota-Prius-(US)-Step-3-Version-3.webp
[IMG-T-0::2024-03-21 15:03:35,142] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/5/54/Start-a-Toyota-Prius-%28US%29-Step-4-Version-3.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-4-Version-3.jpg into ZIM::images/2405511860-v4-460px-Start-a-Toyota-Prius-(US)-Step-4-Version-3.webp
[IMG-T-5::2024-03-21 15:03:35,151] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/9/90/Start-a-Toyota-Prius-%28US%29-Step-5-Version-3.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-5-Version-3.jpg into ZIM::images/2441818810-v4-460px-Start-a-Toyota-Prius-(US)-Step-5-Version-3.webp
[IMG-T-2::2024-03-21 15:03:35,183] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/d/de/Start-a-Toyota-Prius-%28US%29-Step-2-Version-3.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-2-Version-3.jpg into ZIM::images/3455660863-v4-460px-Start-a-Toyota-Prius-(US)-Step-2-Version-3.webp
[IMG-T-3::2024-03-21 15:03:35,189] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/6/67/Start-a-Toyota-Prius-%28US%29-Step-6-Version-3.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-6-Version-3.jpg into ZIM::images/2454270653-v4-460px-Start-a-Toyota-Prius-(US)-Step-6-Version-3.webp
[IMG-T-9::2024-03-21 15:03:35,217] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/e/eb/Start-a-Toyota-Prius-%28US%29-Step-8-Version-2.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-8-Version-2.jpg into ZIM::images/3479384904-v4-460px-Start-a-Toyota-Prius-(US)-Step-8-Version-2.webp
[IMG-T-7::2024-03-21 15:03:35,223] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/e/e2/Start-a-Toyota-Prius-%28US%29-Step-1-Version-3.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-1-Version-3.jpg into ZIM::images/3090363148-v4-460px-Start-a-Toyota-Prius-(US)-Step-1-Version-3.webp
[IMG-T-1::2024-03-21 15:03:35,252] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/d/d2/Start-a-Toyota-Prius-%28US%29-Step-7-Version-2.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-7-Version-2.jpg into ZIM::images/3106222868-v4-460px-Start-a-Toyota-Prius-(US)-Step-7-Version-2.webp
[IMG-T-4::2024-03-21 15:03:35,277] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/b/b6/Start-a-Toyota-Prius-%28US%29-Step-10-Version-2.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-10-Version-2.jpg into ZIM::images/288633704-v4-460px-Start-a-Toyota-Prius-(US)-Step-10-Version-2.webp
[IMG-T-8::2024-03-21 15:03:35,284] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/f/f2/Start-a-Toyota-Prius-%28US%29-Step-9-Version-2.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-9-Version-2.jpg into ZIM::images/3147903772-v4-460px-Start-a-Toyota-Prius-(US)-Step-9-Version-2.webp
[IMG-T-9::2024-03-21 15:03:35,461] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/c/c1/Clean-Headlights-Step-22.jpg/-crop-127-140-127px-Clean-Headlights-Step-22.jpg into ZIM::images/3365283943--crop-127-140-127px-Clean-Headlights-Step-22.webp
[IMG-T-6::2024-03-21 15:03:35,567] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/d/df/Start-a-Toyota-Prius-%28US%29-Step-11-Version-2.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-11-Version-2.jpg into ZIM::images/683619230-v4-460px-Start-a-Toyota-Prius-(US)-Step-11-Version-2.webp
[IMG-T-2::2024-03-21 15:03:35,604] DEBUG:Attempting download of S3::https/www.wikihow.com/images/avatarOut/5/53/3565366.jpg?20160502110857 into ZIM::images/2537952873-3565366.webp
[IMG-T-5::2024-03-21 15:03:35,627] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/f/ff/Start-a-Toyota-Prius-%28US%29-Step-13-Version-2.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-13-Version-2.jpg into ZIM::images/725955494-v4-460px-Start-a-Toyota-Prius-(US)-Step-13-Version-2.webp
[IMG-T-0::2024-03-21 15:03:35,628] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/8/8b/Start-a-Toyota-Prius-%28US%29-Step-12-Version-2.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-12-Version-2.jpg into ZIM::images/4284691268-v4-460px-Start-a-Toyota-Prius-(US)-Step-12-Version-2.webp
[IMG-T-7::2024-03-21 15:03:35,693] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/6/66/Get-the-Best-Gas-Mileage-from-Your-Toyota-Prius-Step-30.jpg/-crop-342-184-245px-Get-the-Best-Gas-Mileage-from-Your-Toyota-Prius-Step-30.jpg into ZIM::images/1645690422--crop-342-184-245px-Get-the-Best-Gas-Mileage-from-Your-Toyota-Prius-Step-30.webp
[IMG-T-3::2024-03-21 15:03:35,729] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/c/c1/Clean-Headlights-Step-22.jpg/-crop-342-184-245px-Clean-Headlights-Step-22.jpg into ZIM::images/3384551535--crop-342-184-245px-Clean-Headlights-Step-22.webp
[MainThread::2024-03-21 15:19:17,911] ERROR:Interrupting process due to error: ('Connection aborted.', TimeoutError(110, 'Connection timed out'))
[MainThread::2024-03-21 15:19:17,912] ERROR:('Connection aborted.', TimeoutError(110, 'Connection timed out'))
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 404, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1058, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.8/ssl.py", line 1073, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.8/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 799, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 404, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1058, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.8/ssl.py", line 1073, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.8/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', TimeoutError(110, 'Connection timed out'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/wikihow2zim-1.2.3-py3.8.egg/wikihow2zim/scraper.py", line 986, in run
    self.scrape_articles()
  File "/usr/local/lib/python3.8/site-packages/wikihow2zim-1.2.3-py3.8.egg/wikihow2zim/scraper.py", line 506, in scrape_articles
    if not self.scrape_article(article):
  File "/usr/local/lib/python3.8/site-packages/wikihow2zim-1.2.3-py3.8.egg/wikihow2zim/scraper.py", line 672, in scrape_article
    page_linked_styles=self.get_style_urls(soup),
  File "/usr/local/lib/python3.8/site-packages/wikihow2zim-1.2.3-py3.8.egg/wikihow2zim/scraper.py", line 88, in get_style_urls
    self.add_css(url)
  File "/usr/local/lib/python3.8/site-packages/wikihow2zim-1.2.3-py3.8.egg/wikihow2zim/scraper.py", line 226, in add_css
    source = url if inline else self.get_from_cache(url).decode("UTF-8")
  File "/usr/local/lib/python3.8/site-packages/wikihow2zim-1.2.3-py3.8.egg/wikihow2zim/scraper.py", line 206, in get_from_cache
    content = self.session.get(url).content
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', TimeoutError(110, 'Connection timed out'))

We see that the call stack is:

However, get_from_cache is not using any retry mechanism, plus its docstring states that it should only be used for devel/debug purposes: https://github.com/openzim/wikihow/blob/275bac20a7aa2b040a65a5f0b0916a5d56b863a7/wikihow2zim/scraper.py#L195-L199

Either add_css is not using the proper fetch method, or the docstring + method behavior are inaccurate (we might use get_from_cache for prod but then it misses a proper backoff mechanism like the fetch method which is used for web scraping calls)

rgaudin commented 6 months ago

I think it wasn't meant for being used in production and was comited along with a big change.