How do I use proxy IP? - Githubissues

codelucas / newspaper

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

https://goo.gl/VX41yK

MIT License

14.2k stars 2.12k forks source link

How do I use proxy IP? #642

Open funnyway opened 6 years ago

funnyway commented 6 years ago

How do I use proxy IP?

Neromaru commented 5 years ago


from newspaper.configuration import Configuration
from bs4 import BeautifulSoup

PROXIES = {
    'http': "",
    'https': ""
}

USER_AGENT = "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"

config = Configuration()

config.proxies = PROXIES

articles_two = Article(URL_FOR_IP, config=config)
articles_two.download()
html_two = BeautifulSoup(articles_two.html, 'lxml')
print(html_two.find('div', {"id": 'ipv4'}).get_text())

Neromaru commented 5 years ago

I hope it helped!