Using Python to Access Web Data_Coursera_Scraping HTML Data with BeautifulSoup

[fairypp]

Scraping Numbers from HTML using BeautifulSoup In this assignment you will write a Python program similar to http://www.pythonlearn.com/code/urllink2.py. The program will use urllib to read the HTML from the data files below, and parse the data, extracting numbers and compute the sum of the numbers in the file.

We provide two files for this assignment. One is a sample file where we give you the sum for your testing and the other is the actual data you need to process for the assignment.

Sample data: http://python-data.dr-chuck.net/comments_42.html (Sum=2553) Actual data: http://python-data.dr-chuck.net/comments_247549.html (Sum ends with 68) You do not need to save these files to your folder since your program will read the data directly from the URL. Note: Each student will have a distinct data url for the assignment - so only use your own data url for analysis. Data Format The file is a table of names and comment counts. You can ignore most of the data in the file except for lines like the following:

Modu90 Kenzie88 Hubert87

You are to find all the tags in the file and pull out the numbers from the tag and sum the numbers. Look at the sample code provided. It shows how to find all of a certain kind of tag, loop through the tags and extract the various aspects of the tags.

...

Retrieve all of the anchor tags

tags = soup('a') for tag in tags:

Look at the parts of a tag

print 'TAG:',tag print 'URL:',tag.get('href', None) print 'Contents:',tag.contents[0] print 'Attrs:',tag.attrs You need to adjust this code to look for span tags and pull out the text content of the span tag, convert them to integers and add them up to complete the assignment. Sample Execution

$ python solution.py Enter - http://python-data.dr-chuck.net/comments_42.html Count 50 Sum 2... Turning in the Assignment

Enter the sum from the actual data and your Python code below: Sum: (ends with 68) Submit Assignment

My Solution:

import urllib
from bs4 import BeautifulSoup

url = 'http://python-data.dr-chuck.net/comments_247549.html'
html = urllib.urlopen(url).read()

soup = BeautifulSoup(html,"html.parser")

# Retrive 
tags = soup('span')

spans = [int(tag.contents[0]) for tag in tags]
summation = sum(spans)
print summation

[aayushanand1]

This code works perfectly `import urllib.request, urllib.parse, urllib.error from bs4 import BeautifulSoup import ssl

Ignore SSL certificate errors

ctx = ssl.create_default_context() ctx.check_hostname = False ctx.verify_mode = ssl.CERT_NONE

url = input('Enter - ') html = urllib.request.urlopen(url, context=ctx).read() soup = BeautifulSoup(html, 'html.parser') list1=[]

Retrieve all of the anchor tags

tags = soup('a') for tag in tags: list1.append(tag.get('href', None)) reps=int(input('enter number of reps:')) count=int(input('enter the count:'))-1 url = list1[count] html = urllib.request.urlopen(url, context=ctx).read() soup = BeautifulSoup(html, 'html.parser') list2=[]

Retrieve all of the anchor tags

i=1 while i<reps: url = list1[count] list1.clear() html = urllib.request.urlopen(url, context=ctx).read() soup = BeautifulSoup(html, 'html.parser') tags = soup('a') for tag in tags: list1.append(tag.get('href', None)) if i==(reps-1): list2.append(tag.contents[0]) i=i+1 print(list2[count]) `

[sheersh2001]

import urllib.request, urllib.parse, urllib.error from bs4 import BeautifulSoup

url = input('Enter - ')

html = urllib.request.urlopen(url).read() soup = BeautifulSoup(html)

Retrieve all of the anchor tags

tags = soup('span') count=0 sum=0 for tag in tags: y=int(tag.text) count=count+1 sum=sum+y print(sum) print(count)

what is the problem in this code. I nam3d it bhaisahab.py and i am getting traceback. Just few minutes before it was running in command prompt but the code there was something wrong in code but when i coreected it i got the trace back

[rejothomaspy4e]

SIMPLIFIED VERSION

from urllib.request import urlopen from bs4 import BeautifulSoup import ssl

Ignore SSL certificate errors

ctx = ssl.create_default_context() ctx.check_hostname = False ctx.verify_mode = ssl.CERT_NONE

url = input('Enter - ') html = urlopen(url, context=ctx).read() soup = BeautifulSoup(html, "html.parser")

Retrieve all of the SPAN tags

tags = soup('span') y=0 for tag in tags:

Look at the parts of a tag

x=int(tag.contents[0])
y=x+y

print(y)