search

Public-Health-Scotland / source-linkage-files

This repo is for the syntax used for the PHS Source Linkage File project
https://public-health-scotland.github.io/source-linkage-files/
Other
4 stars 2 forks source link

Review `read_sc_extract_xxx` for reading from the social care platform #963

Open Jennit07 opened 6 months ago

Jennit07 commented 6 months ago

Currently, we use this: 

  at_full_data <- dplyr::tbl(
    sc_dvprod_connection,
    dbplyr::in_schema("social_care_2", "equipment_snapshot")
  ) %>% dplyr::collect()

and then we do a selection on the variables but this takes too long in targets ~23 minutes. We should try change this to use SQL for selecting variables while we read from the platform to speed this up