Closed cfbeuchel closed 8 months ago
Just for reference, creating a temporary index column gets around this, but depending on the file size, this might cause some overhead, but would be a pragmatic solution that might work for most, if not all, cases.
dat <- data.table(snp = paste0("rs", 1:10))
dat[1:5, snp := NA]
dat[, i := .I]
miss_dat <- dat[!grep("^rs", snp), ]
setkey(miss_dat, i)
setkey(dat, i)
dat[miss_dat, IMPUTATION_SNP := TRUE]
dat
dat$i <- NULL
Edit: Since this becomes an issue already when only a few entries are NA
(expanding to n=sum(is.na(miss_rs$SNP_old_temp)^2
additional rows, a simple check to avoid creating the additional column could be useful. E.g.:
if(any(is.na(miss_rs$SNP_old_temp)) {
[...]
}
I'm actually not too sure if what you identified is causing the ongoing issue you referenced but nevertheless, it is something that should be fixed. I've made a fix in push of v1.11.6. Can you pull from Github and let me know if this addresses your issue?
Hi! I've just successfully formatted a large file that previously produced the error with the current version. That would resolve this issue. Thanks for the patch! Cheers
Hi,
alright, I've been struggling with an edge-case again for a few days and I think I have found the reason for it. I hope at least...
If you think this should be implemented differently I'd gladly submit a PR in case we figure something out. Like simply matching against the index or something.
Edit: I guess this is a duplicate of #164
1. Bug description
Whenever many entries in the
SNP
column are NA, using the joining by key feature from data.table, e.g. incheck_no_rs_snp()
will result in data.table matching allNA
entries insumstats_dt
to allNA
s inmiss_rs
, leading to a quickly inflated number of rows.For example, when
imputation_ind = TRUE
, the following segment will cause a crash (line 415):Console output
Expected behaviour
I think having NA as
SNP
entry is rare and often CHR:BP:REF:ALT or something is used (I guess GWAS Catalog and openGWAS do this?). I am currently munging data from various other sources and my current problems come from the GBMI summary statistics, a large and prominent GWAS meta analysis consortium that sadly decided against using one of the suggested standard formats.Still, it might be desirable to handle this case more robustly.
2. Reproducible example
Code
Data
A summary statistics file that leads to errors can be downloaded from here: https://docs.google.com/spreadsheets/d/1sSU_JfPKs6EZLcY9t3gXsSGPZA1LsP_z/edit
3. Session info
(Add output of the R function
utils::sessionInfo()
below. This helps us assess version/OS conflicts which could be causing bugs.)