codelucas / newspaper

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
https://goo.gl/VX41yK
MIT License
14.14k stars 2.12k forks source link

CNN articles are not correctly parsed #560

Open jensdebruijn opened 6 years ago

jensdebruijn commented 6 years ago

Parsing CNN links yields Read More in the article text.

url = "http://www.cnn.com/2017/01/30/politics/trump-immigration-ban-refugees-trnd/index.html"
article = Article(url)
article.download()
article.parse()
print(article.text)

yields

Being a refugee isn't easy. Neither is getting into the United States. Just gaining refugee status is hard; most of the millions who seek it don't get it. Of those who do, only a fraction end up resettled in the US. The process takes an average of 18 to 24 months, and most of it's conducted outside the US. Here's how it works: It's a long and winding road Read More

while I expect it to yield


(...) 

refugees to the US last year was the Democratic Republic of Congo. Trump also banned all citizens of six countries from entering the U.S. for three months: Iran, Libya, Somalia, Sudan, Syria, and Yemen. Of the six countries targeted by Trump, only three were in the top 10 countries that refugees come from. One of the countries in the list, Libya, sent just one. How many refugees came from the countries on Trump's list`Just gaining refugee status is hard; most of the millions who seek it don't get it. Of those who do, only a fraction end up resettled in the US. The process takes an average of 18 to 24 months, and most of it's conducted outside the US. : Refugees are people who are forced to flee their home country to escape persecution, war or violence. The UN decides the most vulnerable cases and refers them for resettlement. : The UN only refers those whose life, liberty or health are at significant risk. If someone is found to have committed a crime, that person doesn't qualify for resettlement. The US does its own vetting: This process involves eight federal agencies, six different databases, five separate background checks, four fingerprint and biometric checks, three in-person interviews and two inter-agency checks. Then, it vets some more : If allowed to come to the US, refugees face another screening before embarking, and another security check at a US airport. : If everything works out fine, the State Department assigns refugees to an NGO that helps them find work and housing. Refugees flee terror, they don't bring terror to countries. Their arrival is very carefully vetted, so there should be no fear. Historically, the US has been very hospitable Since 1975, the US has resettled more than 3.3 million refugees. Last year, it took in 84,994, mostly women and children. But that's just a drop in the bucket. The UN estimates there are 21.3 million refugees in the world. Turkey hosts the most: 2.8 million. Most refugees come to the US from just 10 countries The country that sent the most refugees to the US last year was the Democratic Republic of Congo. Trump also banned all citizens of six countries from entering the U.S. for three months: Iran, Libya, Somalia, Sudan, Syria, and Yemen. Of the six countries targeted by Trump, only three were in the top 10 countries that refugees come from. One of the countries in the list, Libya, sent just one. How many refugees came from the countries on Trump's list
pgrzesik commented 6 years ago

This PR is solving this issue: https://github.com/codelucas/newspaper/pull/633