search

pgaref / HTTP_Request_Randomizer

Proxying Python Requests
http://pgaref.com/blog/python-proxy/
MIT License
148 stars 60 forks source link

FIX for PremProxy and FreeProxy #75

Open alsrua7222 opened 2 years ago

alsrua7222 commented 2 years ago

http_request_randomizer.requests.errors.ProxyListException.ProxyListException: list is empty -> FreeProxyParser.py: html parsing is list error. (now solved.)

reqeusts.exceptions.InvalidURL: Failed to parse: http://(ip):(port) | FreeProxy -> UnPack.py: headers add -> PremProxyParser.py: headers add

pgaref commented 2 years ago

Thanks for the PR @alsrua7222 ! Tested locally and fixes both PremProxyParser by adding headers on the requests as #74 and FreeProxyParser by updating the attributes. Can you please fix test_FreeProxyParser so we can merge this? I believe a simple update on free_proxy_mock is needed.

@urlmatch(netloc=r'(.*\.)?free-proxy-list\.net$') def free_proxy_mock(url, request): return """<table border="0" cellpadding="0" cellspacing="0" class="table table-striped table-bordered"

alsrua7222 commented 2 years ago

I didn't know if it was updated or not. It's already been updated. I wasted my time. However, libraries help a lot with proxies. thank you. After an error, when I traced back, the html parsing structure was changed. Also, I wanted to point out that "headers" should be added to the library.

As you said, we simply updated the html table structure of free_proxy_list.net. Is it correct to do this?

pgaref commented 2 years ago

I didn't know if it was updated or not. It's already been updated. I wasted my time. However, libraries help a lot with proxies. thank you. After an error, when I traced back, the html parsing structure was changed. Also, I wanted to point out that "headers" should be added to the library.

As you said, we simply updated the html table structure of free_proxy_list.net. Is it correct to do this?

Hey @alsrua7222 thanks for updating the PR! The new html table structure is fine but since you used different proxy exmamples the assertion of the Test should also be fixed.

Please take a look at:

  self.assertEqual(proxy_list_addr, free_proxy_expected)

E AssertionError: Lists differ: ['58.234.116.197:8193', '20.122.24.225:80',[40 chars]080'] != ['138.197.136.46:3128', '177.207.75.227:8080'] E
E First differing element 0: E '58.234.116.197:8193' E '138.197.136.46:3128' E
E First list contains 2 additional elements. E First extra element 2: E '154.236.177.100:1981' E
E + ['138.197.136.46:3128', '177.207.75.227:8080'] E - ['58.234.116.197:8193', E - '20.122.24.225:80', E - '154.236.177.100:1981', E - '54.37.160.92:1080']