OHDSI / Andromeda

AsynchroNous Disk-based Representation of MassivE DAta: An R package aimed at replacing ff for storing large data objects.
https://ohdsi.github.io/Andromeda/
11 stars 13 forks source link

Error / Stack smashing /segfault when calling andromeda from CohortMethod #66

Open DrTTrousers opened 2 months ago

DrTTrousers commented 2 months ago

Hi guys

Not sure if you want this here, CohortMethod, Or CohortMethod Module. Andromeda is getting it because this is the last referenced sub package before the stack smashing crash.

I've got an OHDSI strategus based package I'm trying to run on the big openclaims database. It reaches cohort method and then segfault / stack smashes consistently at the first fetch from the database:

getResultsDataModelSpecifications, migrateDataModel, uploadResults 2024-09-25 11:03:33 [Main thread] DEBUG CohortMethod FUN Calling getDbCohortMethodData() for targetId 324, comparatorId 325 2024-09-25 11:03:33 [Main thread] INFO DatabaseConnector connectSnowflake Connecting using Snowflake driver 2024-09-25 11:03:34 [Main thread] INFO CohortMethod getDbCohortMethodData Constructing target and comparator cohorts 2024-09-25 11:03:40 [Main thread] INFO DatabaseConnector executeSql Executing SQL took 5.72 secs 2024-09-25 11:03:40 [Main thread] INFO CohortMethod getDbCohortMethodData Fetching cohorts from server 2024-09-25 11:03:47 [Main thread] DEBUG CohortMethod getDbCohortMethodData Fetched cohort total rows in target is 166752, total rows in comparator is 957244 2024-09-25 11:03:47 [Main thread] INFO CohortMethod getDbCohortMethodData Fetching cohorts took 6.81 secs 2024-09-25 11:03:47 [Main thread] WARN FeatureExtraction getDbCovariateData The 'oracleTempSchema' argument is deprecated. Use 'tempEmulationSchema' instead. This warning is displayed once every 8 hours. 2024-09-25 11:03:48 [Main thread] INFO FeatureExtraction Sending temp tables to server 2024-09-25 11:03:49 [Main thread] INFO DatabaseConnector insertTable.default Inserting data took 1.09 secs 2024-09-25 11:03:49 [Main thread] INFO FeatureExtraction Constructing features on server 2024-09-25 11:11:24 [Main thread] INFO DatabaseConnector executeSql Executing SQL took 7.57 mins 2024-09-25 11:11:24 [Main thread] INFO FeatureExtraction Fetching data from server

(segfault doesn't capture in the log)


Thread: Main
Message:  VECTOR_ELT() can only be applied to a 'list', not a 'NULL'
Level:  FATAL
Time:  2024-09-11 13:47:46

Stack trace:
17: h(simpleError(msg, call))
16: .handleSimpleError(function (condition) 
{
    if (is(condition, "error")) {

15: tryCatchOne(expr, names, parentenv, handlers[[1]])
14: tryCatchList(expr, classes, parentenv, handlers)
13: tryCatch({
    andromeda <- lowLevelQuerySqlToAndromeda(connection = connect
12: DatabaseConnector::querySqlToAndromeda(connection = connection, sql = sql, 
11: (function (connection, oracleTempSchema = NULL, cdmDatabaseSchema, cohortTa
10: do.call(eval(parse(text = fun)), args)
9: DataLoadingSaving.R#321: FeatureExtraction::getDbCovariateData(connection =
8: getDbCohortMethodData(connectionDetails = list("snowflake", NULL, "thin", "
7: RunAnalyses.R#768: do.call("getDbCohortMethodData", params$args)
6: FUN(X[[i]], ...)
5: lapply(x, fun, ...)
4: RunAnalyses.R#352: ParallelLogger::clusterApply(cluster, objectsToCreate, d
3: (function (connectionDetails, cdmDatabaseSchema, tempEmulationSchema = getO
2: do.call(CohortMethod::runCmAnalyses, args)
1: execute(jobContext)

R version:
R version 4.3.2 (2023-10-31)

Platform:
x86_64-pc-linux-gnu

Attached base packages:
- stats
- graphics
- grDevices
- datasets
- utils
- methods
- base

Other attached packages:
- CohortMethod (5.3.0)
- FeatureExtraction (3.7.0)
- Andromeda (0.6.7)
- dplyr (1.1.4)
- Cyclops (3.4.1)
- RSQLite (2.3.7)
- SqlRender (1.18.1)
- ParallelLogger (3.3.1)
- keyring (1.3.2)
- CohortGenerator (0.8.1)
- R6 (2.5.1)
- DatabaseConnector (6.3.2)
`

At this point I've tried upgrading/recompiling every package and library running the workbench to no success. Is there any insight hades can offer as to how this is managing to error at such a fundamental point? Given its a strategus implementation its quite challenging to debug as I can't (e.g.) breakpoint the code along the way, unless thats been implemented?