Trouble importing zipped csv into R

[lizacoyer]

Hi,

I´m trying to download the https://github.com/robert-koch-institut/SARS-CoV-2-Sequenzdaten_aus_Deutschland/blob/master/SARS-CoV-2-Entwicklungslinien_Deutschland.csv.xz file programmatically into R, but Github blocks the download of binary files. Would it be possible to have this file as a CSV rather than zipping it? Additionally this would prevent your repository from growing too large.

Just FYI, what I am trying to do is

URL <- "https://github.com/robert-koch-institut/SARS-CoV-2-Sequenzdaten_aus_Deutschland/blob/master/SARS-CoV-2-Entwicklungslinien_Deutschland.csv.xz?raw=true"
x <- getURL(URL, ssl.verifypeer=FALSE)
out <- read.csv(untar(textConnection(x)))

but the getURL() cannot read this properly.

Thanks in advance!

[icestorm972]

Maybe let R first download it as local file and then import the data?

getURL seems to have problems in R with zipped contents, but other users found workaround that, for example something like this: https://stackoverflow.com/questions/64431076/r-cannot-download-gz-file-from-ftp-server

or like this: https://stackoverflow.com/questions/9548630/read-gzipped-csv-directly-from-a-url-in-r

[HannesWuensche]

Hallo @lizacoyer,

wie wäre es mit diesem Link:

https://github.com/robert-koch-institut/SARS-CoV-2-Sequenzdaten_aus_Deutschland/raw/master/SARS-CoV-2-Entwicklungslinien_Deutschland.csv.xz (verbirgt sich hinter dem Download Button)

Würden wir die Entwicklungslinien unverpackt bereitstellen wären es cs 180 MB. Wir müssten dann das git-lfs für die Daten verwenden, wofür eine Traffic-Beschränkung besteht. Wir wollen daher nur bei einer absoluten Notwendigkeit drauf zurückgreifen.

Beste Grüße @HannesWuensche für das Team RKI | Open Data

[lizacoyer]

@icestorm972 @HannesWuensche Thank you both for answering so quickly. Neither of the proposed solutions gave me a workable option, but I managed to solve it. Just in case someone else wants to use it too:

library(micropan)

url <- "https://github.com/robert-koch-institut/SARS-CoV-2-Sequenzdaten_aus_Deutschland/raw/master/SARS-CoV-2-Entwicklungslinien_Deutschland.csv.xz"

downloaded <- tempfile(fileext = "csv.xz")
unpacked <- tempfile(fileext = "csv")

download.file(url, downloaded, "auto")

xzuncompress(filename = downloaded, destname = unpacked)

csv <- read.csv(file = unpacked)