search

Public-Health-Scotland / source-linkage-files

This repo is for the syntax used for the PHS Source Linkage File project
https://public-health-scotland.github.io/source-linkage-files/
Other
4 stars 2 forks source link

Review copy to hscdiip script #888

Closed Jennit07 closed 7 months ago

Jennit07 commented 10 months ago

Process for copying over final episode and individual files from sourcedev to hscdiip – script first deletes exiting file then writes to hscdiip but gets stuck early on before writing file. Alternative is to use PUTTY, but need to install on laptop

lizihao-anu commented 8 months ago 
> for (i in 1:length(folders)) {
+   timer <- Sys.time()
+   print(stringr::str_glue("{folders[i]} starts at {Sys.time()}"))
+   folder_path <- file.p .... [TRUNCATED] 
1920 starts at 2024-02-27 10:10:15
[1] "source-episode-file-1920.parquet"    "source-individual-file-1920.parquet"
file size is 931.924534797668.
[1] 7.223492
2021 starts at 2024-02-27 10:17:29
[1] "source-episode-file-2021.parquet"    "source-individual-file-2021.parquet"
file size is 839.647627830505.
[1] 6.240084

The move_to_hscdiip script works fine. I guess that it looks stuck because it takes approximately 7 minutes to finish writing files for data in one year.

lizihao-anu commented 7 months ago

Use putty instead now for faster file moving