search

TWJolly / fundraising_data_pull

Pulls data from the JustGiving API for a defined set of charities and highlights new pages
1 stars 1 forks source link

Trying to download 'all fundraisers for a selected charity' in chunks (because of 9999 limit) #21

Open daaronr opened 4 years ago

daaronr commented 4 years ago

In function get_charity_fundraising_pages in functions.R

The third line below is key:

  charity_search_name <- gsub(' ', '%20', charity_name)
  uri <- paste('/v1/onesearch?q=', charity_search_name, '&i=Fundraiser&limit=9999', sep = '')

We want to get all the pages for each charity. Thus we need to edit this line to take an argument limiting this to a subset of those pages, and then iterate this over all subsets.

This JG page describes the onesearch tool

Can we add something like "campaignId starts with letter {{letter}}" and then iterate this over all alphanumeric characters, and then join them?

Also commented here

@TWJolly

daaronr commented 4 years ago

I thought the 'offset' would help but it doesn't... doesn't seem to let you offset past 9999

https://api.justgiving.com/%7Bb83e9729%7D/v1/onesearch?q=Macmillan%20Cancer%20Support&i=Fundraiser&limit=9&offset=9980

https://api.justgiving.com/%7Bb83e9729%7D/v1/onesearch?q=Macmillan%20Cancer%20Support&i=Fundraiser&limit=9&offset=9997