search

tidyverse / rvest

Simple web scraping for R
https://rvest.tidyverse.org
Other
1.49k stars 340 forks source link

Discrepancy of results between using XML and rvest #21

Closed notesofdabbler closed 9 years ago

notesofdabbler commented 9 years ago

I am not sure what I am doing wrong but when I scrape a page from realtor.com, I am not getting the full list with rvest. I have listed my R code below.


#
# Scrape data on house listings from realtor.com
#

# set working directory
setwd("~/notesofdabbler/Rspace/dayoh_housing/")

# load libraries
library(rvest)
library(XML)

#
# Search URL with following filter applied
#3+ bedrooms, 2+ baths, 1800+ sqft, 0-20 years old
#
srchurl="http://www.realtor.com/realestateandhomes-search/Centerville_OH/beds-3/baths-2/sqft-8/pfbm-10/show-hide-pending"

# using XML library
housedoc=htmlTreeParse(srchurl,useInternalNodes=TRUE)
ns_id=getNodeSet(housedoc,"//ul[@class='listing-summary']//li[@class='listing-location']//a[@href]") 
id=sapply(ns_id,function(x) xmlAttrs(x)["href"])
id

# using rvest library
housedoc = html(srchurl) 
houselist = housedoc %>% html_node(".listing-summary")
id =  houselist %>% html_node(".listing-location a") %>% html_attr("href")
id

The actual run version of the code with output is here.

hadley commented 9 years ago

Maybe you need html_nodes? html_node only returns a single node.

On Sunday, October 12, 2014, notesofdabbler notifications@github.com wrote:

I am not sure what I am doing wrong but when I scrape a page from realtor.com, I am not getting the full list with rvest. I have listed my R code below.

#

Scrape data on house listings from realtor.com

#

set working directory

setwd("~/notesofdabbler/Rspace/dayoh_housing/")

load libraries

library(rvest) library(XML)

#

Search URL with following filter applied

3+ bedrooms, 2+ baths, 1800+ sqft, 0-20 years old

# srchurl="http://www.realtor.com/realestateandhomes-search/Centerville_OH/beds-3/baths-2/sqft-8/pfbm-10/show-hide-pending"

using XML library

housedoc=htmlTreeParse(srchurl,useInternalNodes=TRUE) ns_id=getNodeSet(housedoc,"//ul[@class='listing-summary']//li[@class='listing-location']//a[@href]") id=sapply(ns_id,function(x) xmlAttrs(x)["href"]) id

using rvest library

housedoc = html(srchurl) houselist = housedoc %>% html_node(".listing-summary") id = houselist %>% html_node(".listing-location a") %>% html_attr("href") id

The actual run version of the code with output is here http://notesofdabbler.github.io/other/scrape_housingdata.html.

— Reply to this email directly or view it on GitHub https://github.com/hadley/rvest/issues/21.

http://had.co.nz/

notesofdabbler commented 9 years ago

Thanks very much. changing to html_nodes fixed it. Just for reference I have put the corrected version of the code here