search

REMitchell / apiscraper

123 stars 61 forks source link

Maintenance #3

Open dozham opened 4 years ago

dozham commented 4 years ago

Hey Ryan, I was wondering if you plan to ever maintain this project again? Or has it moved to another project? What's up with it? As far as I can tell, the current code is not a working one.

jambi00 commented 4 years ago

Bump on this, I have been working in the O'reilly book where this is an example. However, the main issue, as far as i can tell, is within the browser.py file. The proxy server, for whatever reason, is not initializing properly therefore searches are not yielding any results. Other parts seem ok. e.g. if you comment out browser.py > line 20 > #chrome_options.add_argument("--proxy-server={0}".format(url)) you wont get the 'data:,' displayed on the chrome head, and pages will scroll/navigate. I have mulled over all other details including chrome driver compatibility and all looks OK on that front.

I have spent a day(!)scratching my head, trying to string together an understanding of all the dependencies but it is taking too much time and I don't want to fall down a rabbit hole. I, like the rest of the community, are problem solvers, but my level experience would cause this to take a tremendous amount of effort if I could even fix it at all. It will niggle me until it is solved though.

If you can take 5 on this Ryan, I'm sure your legacy reader community would be eternally grateful!

--Jambi

rollue commented 3 years ago

@jambi00 Not sure if you still need this. But removing from proxy-server problem you're having can be solved by changing line 18 in browser.py.

Argument to --proxy-server requires address format like "11.456.448.110:8080", but currently url argument to the proxy-server is like "8080"(just the port without the hostname).

...
url = urlparse(self.proxy.proxy)  # remove ".path" at the end
...