Running DUS fails

[eric-fey-hus]

When running the DUS (took a long time) I got this error (sorry for the image, cannot copy text from my analytics env.):

Unfortunately, not a very helpful error message. My backend is SQL Server.

See traceback and sessioninfo screenshots below.

Maybe I can try the odbc_1.5 version (I had 1.3.5, prob from the lockfile), as mentioned in another post, but it will take time to run.

Is there any way to modularize the code for debugging? Like I have the created tables on the database, e.g covid but cannot reference upon restart like this cdm$covid, meaning I need to rerun everything. If we could do something like this cdm$covid = ref_to_exsisting_database_table, that could be speed things up...

[tiozab]

@eric-fey-hus thanks. can you send me the log.txt from the storage as well please?

furthermore, can you give me the versions you are using package.version("CDMConnector") package.version("DrugUtilisation") package.version("IncidencePrevalence") package.version("PatientProfiles") package.version("omopgenerics")

[eric-fey-hus]

For package versions are CDMConnector 1.4.0 DrugUtilisation 0.6.1 IncidencePrevalence 0.7.4 PatientProfiles 1.1.0 omopgenerics 0.2.3

log.txt - can do that later.

[tiozab]

@eric-fey-hus the renv lock may not have worked (the packages are in a very delicate balance to one another). Please make sure that the renv is activated and restored :-)

[tiozab]

@eric-fey-hus the code runs for a long time indeed, if you want to speed things up until they run properly, you can always take a sample from the cdm https://github.com/oxford-pharmacoepi/MegaStudy/issues/46

[eric-fey-hus]

@tiozab OK. Previous env was based on IncPrev step. Any reason why the DUS step renv is so different? Anyway, restored renv from DUS, see screenshot of package versions attached. But now the run fails even earlier:

Package versions match:

...but a few odd ones still do not match:

BTW, it seems a few packages are in the lockfile, but not used at all. RPostgres for example.

[tiozab]

@eric-fey-hus the renv is different because the packages have evolved (there were updated packages like every month). can you send me the log again please?

[tiozab]

@eric-fey-hus if it is not failing at COVID but at another indication then you can use this updated version R script and renv.lock from #53

[eric-fey-hus]

@tiozab Thanks. I have tried the 1point4 version. This one fails at getDrugIngredientCodes:

> concept_drugs <- getDrugIngredientCodes(
+   cdm,
+   name = c("tenecteplase",
+            "alteplase",
[...]
Created a temporary table named #dbplyr_3i2XvLJR8t
Error in `db_copy_to()`:
! Can't copy data to table "#dbplyr_3i2XvLJR8t".
Caused by error in `dplyr::db_write_table()`:
! Can't write table table "#dbplyr_3i2XvLJR8t".
Caused by error in `name@name[["table"]]`:
! subscript out of bounds
Run `rlang::last_trace()` to see where the error occurred.

Logs below. Screen output: log_tmp.txt
Storage log: HUS_log_dus4point1.txt
Package versions: log_packageversions_dus4point1.txt
sessionInfo: sessoin_info_dus4point1.txt
Command that caused the error: log_code_that_caused_error.txt

Debugging still a pain. Even if i subset the cdm with a sample, recreating the cohorts takes ages, eps. the covid cohort. Is there another way to pick up where we left after an error? My omop connection might also time out. And when recreating the cdm reference, all references to th created tables are lost, but these tables are still on the omop database. Any way to add them back to the cdm without (the slow) recreating?

[tiozab]

@eric-fey-hus maybe you can liaise with @elyne who was able to overcome this Problem in issue #63 Or/and @festerman See issue #72

[tiozab]

@eric-fey-hus For inserting existing Tables into the cdm you can use the Function insertTable() from omopgenerics.

[tiozab]

@eric-fey-hus just read your posts, cdmconnector 1.5.0 is not mentioned anywhere, @festerman was talking about odbc 1.5.0? Have you tried that?

[eric-fey-hus]

@tiozab oouh! :confounded: Thanks! Missed that. ...but yes, was trying this now just as I see your message.

[eric-fey-hus]

Now its running. Using odbc 1.5.0 worked. Needed to go back to DatabaseConnector 1.4; version 1.5.0 was not compatible due to an depreciated compute query function in omopgenerics. Fingers crossed 😆

[tiozab]

@eric-fey-hus did it finish? dont see it yet in the uploads

[eric-fey-hus]

@tiozab yes it finished. I had to run in steps. For this I used this trick: The cohort tables were already on the omop database. To create the cdm object including those cohort tables I could use:

cdm <- cdmFomCon(
  con = db,
  cdmSchema = c(schema = cdmSchema),
  writeSchema = c(schema = writeSchema, prefix = writePrefix),
  cohortTables = c('covid', 'neutropenia', [...], 
                   'inc_pat'),
  cdmName = dbName
)

After that referencing to the cohort tables (e.g. cdm$inc_pat) works without the need to recreate them :-)