oneilsh
commented
3 years ago I managed to get this to run via some hints in #57, with some tweaks to tableName RE #86:
LoadVocabFromCsv <- function (connectionDetails, cdmDatabaseSchema, vocabFileLoc)
{
#csvList <- c("concept.csv","vocabulary.csv","concept_ancestor.csv","concept_relationship.csv","relationship.csv","concept_synonym.csv","domain.csv","concept_class.csv", "drug_strength.csv")
csvList <- c("vocabulary.csv")
conn <- DatabaseConnector::connect(connectionDetails)
for (csv in csvList) {
vocabTable <- data.table::fread(file = paste0(vocabFileLoc, "/", csv), stringsAsFactors = FALSE, header = TRUE, sep = "\t", na.strings = NULL)
# Format Dates for tables that need it
if (base::identical(csv,"concept.csv") || base::identical(csv,"concept_relationship.csv") || base::identical(csv,"drug_strength.csv")) {
vocabTable$valid_start_date <- as.Date(as.character(vocabTable$valid_start_date),"%Y%m%d")
vocabTable$valid_end_date <- as.Date(as.character(vocabTable$valid_end_date),"%Y%m%d")
}
writeLines(paste0("Loading: ",csv))
DatabaseConnector::insertTable(conn,tableName=paste0(cdmDatabaseSchema,".",strsplit(csv,"[.]")[[1]][1]), data=as.data.frame(vocabTable), dropTableIfExists = FALSE, createTable = FALSE, progressBar = TRUE)
}
on.exit(DatabaseConnector::disconnect(conn))
}
LoadVocabFromCsv(cd, cdmSchema, vocabFileLoc)
mpreusse
AnthonyMolinaro
Hello again :) I'm still working on ETLing the Synthea COVID 100K dataset, running into an error in the LoadVocabFromCsv step.
I did diverge from the suggested code slightly by generating the indexing scripts and running the index one (in a previous run the LoadEventTables step ran for > 4 hours on the condition_occurance table I think it was, after generating the indices, so I'm trying again from scratch with the latest codebase...)
some settings:
I get an error for
vocabulary.csv, but the others appear to have loaded successfully:Thank you for looking!