softloud
commented
3 years ago Might need to build in a delay so that it doesn't block. Ping too many times too fast on a website and it can fail, but sometimes a delay of a second or two will do the trick. Slows it down, of course, so is only good for smaller scrapes. Writing this down so I don't forget.
The API is limited on how many records are returned per page so I need to work out a way to send several requests to the API to get each page and then join them together. When you call the API you can get an indication of how many pages there are (see below). What I need to do is to use the number of pages to set up a list of dataframes and then fill up this list of dataframes with each and every page of data. The url will need to be appended each time with the next page number in the sequence.
`url<-paste0("https://unstats.un.org/SDGAPI/v1/sdg/Indicator/Data?indicator=1.1.1")
datcall <- jsonlite::fromJSON(url)
names(datcall)
> [1] "size" "totalElements" "totalPages" "pageNumber"
> [5] "attributes" "dimensions" "data"
datcall$totalElements
> [1] 11584
datcall$totalPages
> [1] 464
datcall$pageNumber
> [1] 1
head(datcall$data)
> goal target indicator series
> 1 1 1.1 1.1.1 SI_POV_DAY1
> 2 1 1.1 1.1.1 SI_POV_DAY1
> 3 1 1.1 1.1.1 SI_POV_DAY1
> 4 1 1.1 1.1.1 SI_POV_DAY1
> 5 1 1.1 1.1.1 SI_POV_DAY1
> 6 1 1.1 1.1.1 SI_POV_DAY1
> seriesDescription seriesCount
> 1 Proportion of population below international poverty line (%) 2053
> 2 Proportion of population below international poverty line (%) 2053
> 3 Proportion of population below international poverty line (%) 2053
> 4 Proportion of population below international poverty line (%) 2053
> 5 Proportion of population below international poverty line (%) 2053
> 6 Proportion of population below international poverty line (%) 2053
> geoAreaCode geoAreaName timePeriodStart value valueType time_detail
> 1 2 Africa 1990 46 Float NA
> 2 2 Africa 1993 50 Float NA
> 3 2 Africa 1996 49 Float NA
> 4 2 Africa 1999 49 Float NA
> 5 2 Africa 2002 47 Float NA
> 6 2 Africa 2005 43 Float NA
> timeCoverage upperBound lowerBound basePeriod
> 1 NA NA NA NA
> 2 NA NA NA NA
> 3 NA NA NA NA
> 4 NA NA NA NA
> 5 NA NA NA NA
> 6 NA NA NA NA
> source geoInfoUrl footnotes
> 1 World Development Indicators database, World Bank NA
> 2 World Development Indicators database, World Bank NA
> 3 World Development Indicators database, World Bank NA
> 4 World Development Indicators database, World Bank NA
> 5 World Development Indicators database, World Bank NA
> 6 World Development Indicators database, World Bank NA
> attributes.Nature attributes.Units Reporting Type
> 1 G PERCENT G
> 2 G PERCENT G
> 3 G PERCENT G
> 4 G PERCENT G
> 5 G PERCENT G
> 6 G PERCENT G`