Read abs api

[Annabel-Kennedy]

Hi Matt,

I apologise for the delay in creating this pull request; however, I hope these functions are useful to your package! The request contains all 7 functions that ingest, tidy and describe flat format data from the ABS' SDMX-JSON API. For convenience I have pasted a copy of their descriptions and examples below again (as a few of the functions were renamed):

Exported functions:

• read_abs_api takes the API URL as its only parameter and returns a tidy tibble of the data queried. Both the ABS.Stat URL structure and the new API URL structure (as outlined on api.gov.au) are accepted. The following parameters would just need to be included in the URLs to organise the data in the flat view and JSON format: dimensionAtObservation=allDimensions (for the ABS.Stat SDMX-JSON API) and format=jsondata&dimensionAtObservation=AllDimensions (for the new API).

• abs_clipboard assigns a really long URL (4087+ characters) to a variable, which gets around the RStudio console’s maximum string input of 4087 characters.

• unique_chars takes the results of an API call and shows you all the unique values for all character columns. (Useful to inform filtering, plotting etc.)

Helper functions:

• tidy_api_data tidies the raw data queried using the ABS’ API and returns a tibble.

• api_split_url splits one large URL into smaller URLs to increase efficiency.

• api_split_variables splits each dimension in the API’s data key into batches with a maximum of 50 dimension members each.

• api_join is a unique join function that helps to tidy the data.

read_abs_api examples: ABS.Stat SDMX-JSON API URL:

old_url <- “http://stat.data.abs.gov.au/sdmx-json/data/ALC/all/all?startTime=2000&endTime=2016&dimensionAtObservation=allDimensions”
read_abs_api(old_url)

New API URL:

new_url <- “https://api.data.abs.gov.au/data/ABS,RES_DWELL/all?startPeriod=2019-Q4&endPeriod=2020-Q1&format=jsondata&dimensionAtObservation=AllDimensions”
read_abs_api(new_url)

I'm happy to answer any questions/change anything relating to these functions :) Thanks!

Kind regards, Annabel

[MattCowgill]

Hi @Annabel-Kennedy, thank you so much for this contribution. Sorry for the delay in getting back to you. I will review as soon as I can.

[MattCowgill]

Hi @Annabel-Kennedy and @baslat, thanks again for all your work on this. I'm going to merge this PR into a read_abs_api branch here and do some work on it myself.

What's the benefit of adding the abs_clipboard() wrapper around readClipboard()?

[Annabel-Kennedy]

Hi @MattCowgill,

No worries! I would say the benefit of the abs_clipboard() wrapper is its added documentation for users; however, obviously this can be removed :)

Some of the changes made to fix the previous errors have caused conflicts, which I have not had the opportunity to look at yet with my uni load and work. Sorry about this. Thank you very much!

[MattCowgill]

No worries, thanks again @Annabel-Kennedy. Sorry again for the delay reviewing this

[baslat]

Thanks both, and I agree with @Annabel-Kennedy re documentation. Having abs_clipboard() exported along with abs_api() (noting I think the function name has chagned for this PR) meant that the user saw the two abs_* functions and helped remind them of the weird process needed with the really long URLs, so there was also an element of convenience.

[MattCowgill]

Thank you both! I've merged into the read_abs_api branch here; any further changes will be made there before merging to master via this PR https://github.com/MattCowgill/readabs/pull/168