Danielsloth
commented
8 years ago library("rvest") library("plyr") 'css.selector="td"' 'css.selector="td:nth-child(6) , .filtered" including date' css.selector=".filtered" link <-" http://www.ft.dk/dokumenter/vis_efter_type/spoergsmaal/spoergsmaal.aspx?questionSearchtype=1&startDate=20101006&endDate=20161001&session=&minister=-1&inquirer=-1&committee=-1&statusAnswer=-1&sortColumn=caseNumber&sortOrder=desc&startRecord=1&numberOfRecords=150000&totalNumberOfRecords=#dok " ft_data=read_html(link, encoding = "UTF-8") %>% html_nodes(css=css.selector) %>% html_text() ft_data df<-data.frame(ft_data)
fulddata <- matrix(ncol=1) for (i in 1:length(df[,1])){ link_spm <-"http://www.ft.dk/samling/20111/spoergsmaal/" link1 <- paste(paste(link_spm, df[i,1], sep = ""),"/index.htm", sep ="") link1<-gsub(" ", "", link1) fulddata <- rbind(fulddata, link1) 'Sys.sleep(1)' } fulddata<-as.data.frame(fulddata[-1,]) fulddata<-data.frame(lapply(fulddata, as.character), stringsAsFactors = FALSE) first.link = as.data.frame(fulddata[1,])
first.link.text=read_html(first.link, encoding = "UTF-8") %>% html_nodes(".tableTitle+ p , p a , h1") %>% html_text()
2015-12-06 10:32 GMT+01:00 ChrisNygaard notifications@github.com:
title: "Eksamen" output: html_document
The paragraph 20 questions asked in the Danish Parliament (Folketinget), are questions directed at a secretary of the government from a regular member of parliament. They are used to directly aquire information from the particular secretary, and are usually rather critical of the law they are directed at.
First we need all our packages
Libraries
library("readr") library("plyr") library("ggplot2") library("dplyr") library("rvest") library("stringr") library("lubridate")
First we need to acquire the data. This is done by scraping the website of the Danish Parliament. Many of the issues can be solved simply by scraping the main pages with the question headline, author, department directed at, date and so on. This is done by a scraper functio that we define as such.
scrape <- function(link){
First we load the website
ft_data <- read_html(link, encoding="UTF-8")
Then we get the titles of the questions
ft_titler <- as.matrix(ft_data %>% html_nodes("h3 a") %>% html_text())
ft_titler <- as.matrix(str_trim(ft_titler, "both")) ft_titler <- gsub("Endeligt svar", "", ft_titler)
Then we acquire the wanted information
ft_systematisk = matrix(ft_data %>% html_nodes("div p") %>% html_text()) ft_systematisk <- as.matrix(str_trim(ft_systematisk, "both"))
ft_systematisk <- gsub("Endeligt svar", "", ft_systematisk)
ft_systematisk <-
return(ft_systematisk) }
Then we need to us our function on the link of our choosing. The link will be set in a loop, to ensure we get all the data right away. Thus we start with 200 views per page at page 1
ft_link <- "http://www.ft.dk/Search.aspx?q=&SearchDepthFull=&Udvalg=&ParagrafTyveType=&StilletTil=&StilletAf=&PeriodenBesvaretFrom=&PeriodenBesvaretTo=&Status=&Emneord=&sf=ps&msf=ps&as=1&Samling=20111&PeriodenStilletFrom=2010-07-01&PeriodenStilletTo=2015-11-21&SortBy=SortDate&SortOrder=asc&pageSize=200&pageNr=1#search" x1 <- scrape(ft_link) print(x1)
Note that the echo = FALSE parameter was added to the code chunk to prevent printing of the R code that generated the plot.
— Reply to this email directly or view it on GitHub https://github.com/ChrisNygaard/Eksamensgruppe19/issues/1.
ChrisNygaard
title: "Eksamen"
output: html_document
The paragraph 20 questions asked in the Danish Parliament (Folketinget), are questions directed at a secretary of the government from a regular member of parliament. They are used to directly aquire information from the particular secretary, and are usually rather critical of the law they are directed at.
First we need all our packages
First we need to acquire the data. This is done by scraping the website of the Danish Parliament. Many of the issues can be solved simply by scraping the main pages with the question headline, author, department directed at, date and so on. This is done by a scraper functio that we define as such.
ændring Then we need to us our function on the link of our choosing. The link will be set in a loop, to ensure we get all the data right away. Thus we start with 200 views per page at page 1
Note that the
echo = FALSEparameter was added to the code chunk to prevent printing of the R code that generated the plot.