[BUG] Not parsing arabic urls

[xoco70]

Describe the bug The script try to parse:

https://www.tutorialbar.com/%D8%A7%D9%83%D8%AA%D8%B4%D9%81-%D8%A7%D8%AD%D8%AF%D8%AB-30-%D8%A7%D8%AE%D8%AA%D8%B5%D8%A7%D8%B1-%D9%81%D9%89-%D8%A7%D9%84%D8%A7%D9%83%D8%B3%D9%8A%D9%84/

but he cannot.

Expected behavior Maybe make errors not fatal

Logs

➜  Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE git:(develop) ✗      python3 udemy_enroller.py --browser=chrome                        
Loading existing settings

[WDM] - Current google-chrome version is 87.0.4280
[WDM] - Get LATEST driver version for 87.0.4280
[WDM] - Get LATEST driver version for 87.0.4280
[WDM] - Trying to download new driver from http://chromedriver.storage.googleapis.com/87.0.4280.88/chromedriver_linux64.zip
[WDM] - Driver has been saved in cache [/home/julien/.wdm/drivers/chromedriver/linux64/87.0.4280.88]
tutorialbar scraper is running
comidoc scraper is running
comidoc scraper complete
Got 68 links from https://comidoc.net in 0.32 seconds
Page: 1 of 653 scraped from tutorialbar.com
Got 12 links from https://www.tutorialbar.com in 7.53 seconds
Successfully enrolled in: The Python Programming Comprehensive Bootcamp | Udemy
Successfully enrolled in: The Python Programming For Everyone Immersive Training | Udemy
Successfully enrolled in: Artificial Neural Networks (ANN) with Keras in Python and R | Udemy
Successfully enrolled in: DIY Website Design Bootcamp: WordPress Elementor Plugin | Udemy
Successfully enrolled in: Programming Fundamentals using Python and C++ | Udemy
Successfully enrolled in: Learn Servlets : Basics in a few hours | Udemy
Successfully enrolled in: Python 3: Análisis y visualización de datos | Udemy
Successfully enrolled in: Code Editors Master Course | Udemy
Successfully enrolled in: JDBC | Udemy
Successfully enrolled in: cPanel | Udemy
Successfully enrolled in: Learn Photo Editing with Photoshop 2020 | Udemy
Successfully enrolled in: Full Stack JavaScript Developer en Español | Udemy
tutorialbar scraper is running
Page: 2 of 653 scraped from tutorialbar.com
Got 12 links from https://www.tutorialbar.com in 1.67 seconds
tutorialbar scraper is running
Page: 3 of 653 scraped from tutorialbar.com
Got 12 links from https://www.tutorialbar.com in 1.76 seconds
tutorialbar scraper is running
Page: 4 of 653 scraped from tutorialbar.com
Error while running tutorialbar scrapper: 0, message='', url=URL('https://www.tutorialbar.com/%D8%A7%D9%83%D8%AA%D8%B4%D9%81-%D8%A7%D8%AD%D8%AF%D8%AB-30-%D8%A7%D8%AE%D8%AA%D8%B5%D8%A7%D8%B1-%D9%81%D9%89-%D8%A7%D9%84%D8%A7%D9%83%D8%B3%D9%8A%D9%84/')
Closing browser
Traceback (most recent call last):
  File "udemy_enroller.py", line 120, in <module>
    main()
  File "udemy_enroller.py", line 116, in main
    run(args.browser, tutorialbar_enabled, comidoc_enabled, args.max_pages)
  File "udemy_enroller.py", line 57, in run
    redeem_courses(dm.driver, settings, tutorialbar_enabled, comidoc_enabled, max_pages)
  File "/home/julien/Code/Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE/core/runner.py", line 89, in redeem_courses
    _redeem_courses(driver, settings, scrapers)
  File "/home/julien/Code/Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE/core/runner.py", line 36, in _redeem_courses
    udemy_course_links = loop.run_until_complete(scrapers.run())
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/julien/Code/Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE/core/scrapers/manager.py", line 28, in run
    await asyncio.gather(*map(lambda sc: sc.run(), enabled_scrapers)),
  File "/home/julien/Code/Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE/core/scrapers/base_scraper.py", line 72, in wrapper
    f"Got {len(response)} links from {self.DOMAIN} in {(end_time - start_time).total_seconds():.2f} seconds"

Desktop (please complete the following information):

• OS: Ubuntu 20.04
• Browser : Chrome

[fakeid30]

Don't think we can support non latin character in URL. I will post it as won't fix. Even my browser can not visit this url.

[fakeid30]

Actually , we can skip any non Latin url and not crash the program. Yes.

[fakeid30]

@xoco70 should be fixed now.

[xoco70]

Cool ! This is quite an awesome script you have here !!!