mjwestgate
commented
2 years ago This is an interesting one, as it may require a rethink of how atlas_media works. The current workflow for atlas_media is as follows:
- Filters are first passed to
atlas_occurrences, with a bespokeselectfunction to add 'media' fields to the result. Each of these fields contains unique identifiers for media associated with an individual occurrence. If there is more that one file, those identifiers are separated by '|'. - Metadata for each ID is downloaded from the images API and joined to the occurrence download - this is the tibble returned by
atlas_mediacontaining relevant information such as file size and licence type. Unlike BioCache queries, there is no pre-filtering option with this API, makinggalah_filteran odd mechanism to interact with this dataset - Loop over all image URLs and download to the cache
atlas_media is unusual, therefore, in that it returns images as a side-effect; what is actually returned to the workspace is a tibble (as in atlas_occurrences and elsewhere), not the images themselves. Further, the current function design ensures that the resulting tibble can only be filtered once the images have already been downloaded, which is inefficient (and prevents filtering by licence type as per issue #151). A final problem is that if you have a set of media IDs, there is no way to bypass the atlas_occurrences step within atlas_media and just download the files you want.
Some useful additions would be:
- allow media categories to be passed via
galah_select, and thereby toatlas_occurrences, perhaps including a newgroup = mediaoption - prevent
atlas_mediafrom downloading any images, and instead just return atibble - add new function (e.g.
get_media) that does the actual download - [lower priority]
show_all_licences()could be created to hit https://images.ala.org.au/ws/LicenceMapping
A harder problem is how atlas_media should behave.
Option 1 is to force the user to do their own occurrence download first, and pass the results to atlas_media to get image metadata, i.e.
galah_call() |>
galah_filter(year == 2022) |>
galah_select(group = c("basic", "media")) |>
atlas_occurrences() |>
atlas_media() |>
dplyr::filter(sizeInBytes < 10^6) |> # optional extra filtering stage
get_media()This is more modular than the current version, and doesn't require many new function names; but greatly changes the behaviour of atlas_media by forcing an intermediate call to galah_select and atlas_occurrences.
Option 2 is to keep the filtering behaviour of atlas_media unchanged, but create a new function for users who want a more modular workflow, e.g.
# basic usage
galah_call() |>
galah_filter(year == 2022) |>
atlas_media() |> # returns a tibble only, but doesn't require atlas_occurrences first
dplyr::filter(sizeInBytes < 10^6, licence == "http://creativecommons.org/licenses/by-sa/4.0/") |>
get_media(cache = "my_cache", type = "thumbnail") # downloads images
# advanced usage
galah_call() |>
galah_filter(year == 2022) |>
galah_select(group = c("basic", "media")) |>
atlas_occurrences() |>
show_all_media() |> # get metadata on images
dplyr::filter(sizeInBytes < 10^6) |>
get_media()
It was suggested in a separate issue to add a fileSize filter for downloading media. This is potentially a good idea worth discussing, and I don't imagine it being too difficult to implement for
atlas_media