enricobacis
commented
5 years ago Hi,
are you sure the problem is about web of science limits? You can try to add some debug output to the code to investigate the problem.
Unfortunately I can't help much if it's a wos limitation for your subscription, because trying to circumvent it would violate the terms of service.
I don't think that changing IP would actually help, because I don't think that your college assigns different public IPs to students, you are routed via the same exit node and changing IP will only affect the local network, so it would not reset the limits. But you can simply try, and if you think that changing the IP is a valid solution, and does not violate the terms of service for your school, there are easy ways to force a reconnection from python, so just add such a step every thousand requests or so.
I hope this helps you.
Enrico
Description
Hello, I'm a collage student.Our school bought the "web of science" database,so I can connect it through campus network.But when I crawl too often,the output is always like this:"urllib.error.HTTPError: HTTP Error 500: Internal Server Error".I am considering change my IP address when this problem occur.
What should I do to add such a function?Or maybe there is another way to solve this problem?
Thanks for reading!