search

OHDSI / CohortDiagnostics

An R package for performing various cohort diagnostics.
https://ohdsi.github.io/CohortDiagnostics
40 stars 45 forks source link

Error running cohort diagnostics on Eunomia #1072

Closed ablack3 closed 1 week ago

ablack3 commented 10 months ago

I'm not sure if I'm doing something wrong here or what this error is about but it seems like my small example code should work.

# run cohort diagnostics and view the app.

library(DatabaseConnector)

connectionDetails <- Eunomia::getEunomiaConnectionDetails()

con <- connect(connectionDetails)
#> Connecting using SQLite driver
DatabaseConnector::getTableNames(con)
#>  [1] "care_site"             "cdm_source"            "cohort"               
#>  [4] "cohort_attribute"      "concept"               "concept_ancestor"     
#>  [7] "concept_class"         "concept_relationship"  "concept_synonym"      
#> [10] "condition_era"         "condition_occurrence"  "cost"                 
#> [13] "death"                 "device_exposure"       "domain"               
#> [16] "dose_era"              "drug_era"              "drug_exposure"        
#> [19] "drug_strength"         "fact_relationship"     "location"             
#> [22] "measurement"           "metadata"              "note"                 
#> [25] "note_nlp"              "observation"           "observation_period"   
#> [28] "payer_plan_period"     "person"                "procedure_occurrence" 
#> [31] "provider"              "relationship"          "source_to_concept_map"
#> [34] "specimen"              "visit_detail"          "visit_occurrence"     
#> [37] "vocabulary"
disconnect(con)

library(CohortDiagnostics)
#> Loading required package: FeatureExtraction
#> Loading required package: Andromeda
#> Loading required package: dplyr
#> 
#> Attaching package: 'dplyr'
#> 
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> 
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
cohortDefinitionSet <- CohortGenerator::getCohortDefinitionSet(
  settingsFileName = "Cohorts.csv",
  jsonFolder = "cohorts",
  sqlFolder = "sql/sql_server",
  packageName = "CohortDiagnostics"
)
#> Loading cohortDefinitionSet

cohortTableNames <- CohortGenerator::getCohortTableNames(cohortTable = "cohort")

# Next create the tables on the database
CohortGenerator::createCohortTables(
  connectionDetails = connectionDetails,
  cohortTableNames = cohortTableNames,
  cohortDatabaseSchema = "main",
  incremental = FALSE
)
#> Connecting using SQLite driver
#> Creating cohort tables
#> - Created table main.cohort
#> - Created table main.cohort_inclusion
#> - Created table main.cohort_inclusion_result
#> - Created table main.cohort_inclusion_stats
#> - Created table main.cohort_summary_stats
#> - Created table main.cohort_censor_stats
#> Creating cohort tables took 0.2secs

# Generate the cohort set
CohortGenerator::generateCohortSet(
  connectionDetails = connectionDetails,
  cdmDatabaseSchema = "main",
  cohortDatabaseSchema = "main",
  cohortTableNames = cohortTableNames,
  cohortDefinitionSet = cohortDefinitionSet,
  incremental = FALSE
)
#> Connecting using SQLite driver
#> Initiating cluster consisting only of main thread
#> 1/4- Generating cohort: [PL 192671002] Gastrointestinal hemorrhage referent concept prevalent cohort: First occurrence of referent concept + descendants
#>   |                                                                              |                                                                      |   0%  |                                                                              |==                                                                    |   2%  |                                                                              |===                                                                   |   5%  |                                                                              |=====                                                                 |   7%  |                                                                              |=======                                                               |  10%  |                                                                              |=========                                                             |  12%  |                                                                              |==========                                                            |  15%  |                                                                              |============                                                          |  17%  |                                                                              |==============                                                        |  20%  |                                                                              |===============                                                       |  22%  |                                                                              |=================                                                     |  24%  |                                                                              |===================                                                   |  27%  |                                                                              |====================                                                  |  29%  |                                                                              |======================                                                |  32%  |                                                                              |========================                                              |  34%  |                                                                              |==========================                                            |  37%  |                                                                              |===========================                                           |  39%  |                                                                              |=============================                                         |  41%  |                                                                              |===============================                                       |  44%  |                                                                              |================================                                      |  46%  |                                                                              |==================================                                    |  49%  |                                                                              |====================================                                  |  51%  |                                                                              |======================================                                |  54%  |                                                                              |=======================================                               |  56%  |                                                                              |=========================================                             |  59%  |                                                                              |===========================================                           |  61%  |                                                                              |============================================                          |  63%  |                                                                              |==============================================                        |  66%  |                                                                              |================================================                      |  68%  |                                                                              |==================================================                    |  71%  |                                                                              |===================================================                   |  73%  |                                                                              |=====================================================                 |  76%  |                                                                              |=======================================================               |  78%  |                                                                              |========================================================              |  80%  |                                                                              |==========================================================            |  83%  |                                                                              |============================================================          |  85%  |                                                                              |=============================================================         |  88%  |                                                                              |===============================================================       |  90%  |                                                                              |=================================================================     |  93%  |                                                                              |===================================================================   |  95%  |                                                                              |====================================================================  |  98%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.148 secs
#> 2/4- Generating cohort: [PL 192671001] Gastrointestinal hemorrhage referent concept incident cohort: First occurrence of referent concept + descendants with >=365d prior observation
#>   |                                                                              |                                                                      |   0%  |                                                                              |==                                                                    |   2%  |                                                                              |===                                                                   |   5%  |                                                                              |=====                                                                 |   7%  |                                                                              |=======                                                               |  10%  |                                                                              |=========                                                             |  12%  |                                                                              |==========                                                            |  15%  |                                                                              |============                                                          |  17%  |                                                                              |==============                                                        |  20%  |                                                                              |===============                                                       |  22%  |                                                                              |=================                                                     |  24%  |                                                                              |===================                                                   |  27%  |                                                                              |====================                                                  |  29%  |                                                                              |======================                                                |  32%  |                                                                              |========================                                              |  34%  |                                                                              |==========================                                            |  37%  |                                                                              |===========================                                           |  39%  |                                                                              |=============================                                         |  41%  |                                                                              |===============================                                       |  44%  |                                                                              |================================                                      |  46%  |                                                                              |==================================                                    |  49%  |                                                                              |====================================                                  |  51%  |                                                                              |======================================                                |  54%  |                                                                              |=======================================                               |  56%  |                                                                              |=========================================                             |  59%  |                                                                              |===========================================                           |  61%  |                                                                              |============================================                          |  63%  |                                                                              |==============================================                        |  66%  |                                                                              |================================================                      |  68%  |                                                                              |==================================================                    |  71%  |                                                                              |===================================================                   |  73%  |                                                                              |=====================================================                 |  76%  |                                                                              |=======================================================               |  78%  |                                                                              |========================================================              |  80%  |                                                                              |==========================================================            |  83%  |                                                                              |============================================================          |  85%  |                                                                              |=============================================================         |  88%  |                                                                              |===============================================================       |  90%  |                                                                              |=================================================================     |  93%  |                                                                              |===================================================================   |  95%  |                                                                              |====================================================================  |  98%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.146 secs
#> 3/4- Generating cohort: [PL 201826003] Type 2 diabetes mellitus prevalent cohort:  First condition occurrence of 'Type 2 diabetes mellitus' conceptset or observation of 'History of diabetes'
#>   |                                                                              |                                                                      |   0%  |                                                                              |==                                                                    |   2%  |                                                                              |===                                                                   |   5%  |                                                                              |=====                                                                 |   7%  |                                                                              |=======                                                               |  10%  |                                                                              |=========                                                             |  12%  |                                                                              |==========                                                            |  15%  |                                                                              |============                                                          |  17%  |                                                                              |==============                                                        |  20%  |                                                                              |===============                                                       |  22%  |                                                                              |=================                                                     |  24%  |                                                                              |===================                                                   |  27%  |                                                                              |====================                                                  |  29%  |                                                                              |======================                                                |  32%  |                                                                              |========================                                              |  34%  |                                                                              |==========================                                            |  37%  |                                                                              |===========================                                           |  39%  |                                                                              |=============================                                         |  41%  |                                                                              |===============================                                       |  44%  |                                                                              |================================                                      |  46%  |                                                                              |==================================                                    |  49%  |                                                                              |====================================                                  |  51%  |                                                                              |======================================                                |  54%  |                                                                              |=======================================                               |  56%  |                                                                              |=========================================                             |  59%  |                                                                              |===========================================                           |  61%  |                                                                              |============================================                          |  63%  |                                                                              |==============================================                        |  66%  |                                                                              |================================================                      |  68%  |                                                                              |==================================================                    |  71%  |                                                                              |===================================================                   |  73%  |                                                                              |=====================================================                 |  76%  |                                                                              |=======================================================               |  78%  |                                                                              |========================================================              |  80%  |                                                                              |==========================================================            |  83%  |                                                                              |============================================================          |  85%  |                                                                              |=============================================================         |  88%  |                                                                              |===============================================================       |  90%  |                                                                              |=================================================================     |  93%  |                                                                              |===================================================================   |  95%  |                                                                              |====================================================================  |  98%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.0881 secs
#> 4/4- Generating cohort: [Cohort Diagnostics] Gastrointestinal hemorrhage for Eunomia
#>   |                                                                              |                                                                      |   0%  |                                                                              |==                                                                    |   2%  |                                                                              |===                                                                   |   5%  |                                                                              |=====                                                                 |   7%  |                                                                              |=======                                                               |  10%  |                                                                              |=========                                                             |  12%  |                                                                              |==========                                                            |  15%  |                                                                              |============                                                          |  17%  |                                                                              |==============                                                        |  20%  |                                                                              |===============                                                       |  22%  |                                                                              |=================                                                     |  24%  |                                                                              |===================                                                   |  27%  |                                                                              |====================                                                  |  29%  |                                                                              |======================                                                |  32%  |                                                                              |========================                                              |  34%  |                                                                              |==========================                                            |  37%  |                                                                              |===========================                                           |  39%  |                                                                              |=============================                                         |  41%  |                                                                              |===============================                                       |  44%  |                                                                              |================================                                      |  46%  |                                                                              |==================================                                    |  49%  |                                                                              |====================================                                  |  51%  |                                                                              |======================================                                |  54%  |                                                                              |=======================================                               |  56%  |                                                                              |=========================================                             |  59%  |                                                                              |===========================================                           |  61%  |                                                                              |============================================                          |  63%  |                                                                              |==============================================                        |  66%  |                                                                              |================================================                      |  68%  |                                                                              |==================================================                    |  71%  |                                                                              |===================================================                   |  73%  |                                                                              |=====================================================                 |  76%  |                                                                              |=======================================================               |  78%  |                                                                              |========================================================              |  80%  |                                                                              |==========================================================            |  83%  |                                                                              |============================================================          |  85%  |                                                                              |=============================================================         |  88%  |                                                                              |===============================================================       |  90%  |                                                                              |=================================================================     |  93%  |                                                                              |===================================================================   |  95%  |                                                                              |====================================================================  |  98%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.115 secs
#> Generating cohort set took 0.95 secs

CohortGenerator::getCohortCounts(
  connectionDetails = connectionDetails,
  cohortDatabaseSchema = "main",
  cohortTable = "cohort")
#> Connecting using SQLite driver
#> Counting cohorts took 0.018 secs
#>   cohortId cohortEntries cohortSubjects
#> 1    14909           479            479
#> 2    17492           479            479
#> 3    17493           479            479

executeDiagnostics(cohortDefinitionSet,
                   connectionDetails = connectionDetails,
                   cohortTable = "cohort",
                   cohortDatabaseSchema = "main",
                   cdmDatabaseSchema = "main",
                   exportFolder = here::here("export"),
                   databaseId = "gibleed",
                   minCellCount = 5
)
#> Run Cohort Diagnostics started at 2023-09-01 13:46:06
#>  - Databasename was not provided. Using CDM source table
#>  - Databasedescription was not provided. Using CDM source table
#> Created folder at /private/var/folders/xx/01v98b6546ldnm1rg1_bvk000000gn/T/RtmppgVE8Y/reprex-a49eada76eb-lucid-carp/export
#> Currently in a tryCatch or withCallingHandlers block, so unable to add global calling handlers. ParallelLogger will not capture R messages, errors, and warnings, only explicit calls to ParallelLogger. (This message will not be shown again this R session)
#> The following fields found in the cohortDefinitionSet will be exported in JSON format as part of metadata field of cohort table:
#>     name
#>  - Unexpected fields found in table cohort - name. These fields will be ignored.
#> Connecting using SQLite driver
#> Saving database metadata
#> Saving database metadata took 0.0247 secs
#> Counting cohort records and subjects
#> Counting cohorts took 0.0188 secs
#> - Censoring 0 values (0%) from cohortEntries because value below minimum
#> - Censoring 0 values (0%) from cohortSubjects because value below minimum
#> Found 3 of 4 (75.00%) submitted cohorts instantiated. Beginning cohort diagnostics for instantiated cohorts. 
#> Fetching inclusion statistics from files
#> Exporting inclusion rules with CohortGenerator
#> Warning in CohortGenerator::insertInclusionRuleNames(connection = connection, :
#> No inclusion rules found in the cohortDefinitionSet
#> - Fetching data from cohort_inclusion
#> - Fetching data from cohort_inclusion_result
#> - Fetching data from cohort_inclusion_stats
#> - Fetching data from cohort_inclusion_stats
#> - Fetching data from cohort_summary_stats
#> - Fetching data from cohort_censor_stats
#> - Censoring 0 values (0%) from personCount because value below minimum
#> - Censoring 0 values (0%) from baseCount because value below minimum
#> - Censoring 0 values (0%) from finalCount because value below minimum
#> Exporting cohort concept sets to csv
#>  - Unexpected fields found in table concept_sets - databaseId. These fields will be ignored.
#> Starting concept set diagnostics
#> Instantiating concept sets
#>   |                                                                              |                                                                      |   0%  |                                                                              |==============                                                        |  20%  |                                                                              |======================================================================| 100%
#> Creating internal concept counts table
#>   |                                                                              |                                                                      |   0%  |                                                                              |=======================                                               |  33%  |                                                                              |===============================================                       |  67%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.275 secs
#> Fetching included source concepts
#>   |                                                                              |                                                                      |   0%  |                                                                              |=========                                                             |  12%  |                                                                              |==================                                                    |  25%  |                                                                              |==========================                                            |  38%  |                                                                              |===================================                                   |  50%  |                                                                              |============================================                          |  62%  |                                                                              |====================================================                  |  75%  |                                                                              |=============================================================         |  88%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.121 secs
#> - Censoring 0 values (0%) from conceptSubjects because value below minimum
#> - Censoring 0 values (0%) from conceptCount because value below minimum
#> Finding source codes took 0.238 secs
#> Breaking down index events
#> - Breaking down index events for cohort '[Cohort Diagnostics] Gastrointestinal hemorrhage for Eunomia'
#> - Breaking down index events for cohort '[PL 192671002] Gastrointestinal hemorrhage referent concept prevalent cohort: First occurrence of referent concept + descendants'
#> - Breaking down index events for cohort '[PL 192671001] Gastrointestinal hemorrhage referent concept incident cohort: First occurrence of referent concept + descendants with >=365d prior observation'
#> - Breaking down index events for cohort '[PL 201826003] Type 2 diabetes mellitus prevalent cohort:  First condition occurrence of 'Type 2 diabetes mellitus' conceptset or observation of 'History of diabetes''
#> Index event breakdown results were not returned for: 17720
#> - Censoring 0 values (0%) from conceptCount because value below minimum
#> - Censoring 0 values (0%) from subjectCount because value below minimum
#> - Censoring 0 values (0%) from conceptCount because value below minimum
#> - Censoring 0 values (0%) from subjectCount because value below minimum
#> Breaking down index event took 0.785 secs
#> Finding orphan concepts
#> - Finding orphan concepts for concept set 'Gastrointestinal hemorrhage ConceptSet'
#>   |                                                                              |                                                                      |   0%  |                                                                              |====                                                                  |   6%  |                                                                              |========                                                              |  11%  |                                                                              |============                                                          |  17%  |                                                                              |================                                                      |  22%  |                                                                              |===================                                                   |  28%  |                                                                              |=======================                                               |  33%  |                                                                              |===========================                                           |  39%  |                                                                              |===============================                                       |  44%  |                                                                              |===================================                                   |  50%  |                                                                              |=======================================                               |  56%  |                                                                              |===========================================                           |  61%  |                                                                              |===============================================                       |  67%  |                                                                              |===================================================                   |  72%  |                                                                              |======================================================                |  78%  |                                                                              |==========================================================            |  83%  |                                                                              |==============================================================        |  89%  |                                                                              |==================================================================    |  94%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.0128 secs
#> - Finding orphan concepts for concept set 'Gastrointestinal hemorrhage ConceptSet'
#>   |                                                                              |                                                                      |   0%  |                                                                              |====                                                                  |   6%  |                                                                              |========                                                              |  11%  |                                                                              |============                                                          |  17%  |                                                                              |================                                                      |  22%  |                                                                              |===================                                                   |  28%  |                                                                              |=======================                                               |  33%  |                                                                              |===========================                                           |  39%  |                                                                              |===============================                                       |  44%  |                                                                              |===================================                                   |  50%  |                                                                              |=======================================                               |  56%  |                                                                              |===========================================                           |  61%  |                                                                              |===============================================                       |  67%  |                                                                              |===================================================                   |  72%  |                                                                              |======================================================                |  78%  |                                                                              |==========================================================            |  83%  |                                                                              |==============================================================        |  89%  |                                                                              |==================================================================    |  94%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.0118 secs
#> - Finding orphan concepts for concept set 'Gastrointestinal hemorrhage ConceptSet'
#>   |                                                                              |                                                                      |   0%  |                                                                              |====                                                                  |   6%  |                                                                              |========                                                              |  11%  |                                                                              |============                                                          |  17%  |                                                                              |================                                                      |  22%  |                                                                              |===================                                                   |  28%  |                                                                              |=======================                                               |  33%  |                                                                              |===========================                                           |  39%  |                                                                              |===============================                                       |  44%  |                                                                              |===================================                                   |  50%  |                                                                              |=======================================                               |  56%  |                                                                              |===========================================                           |  61%  |                                                                              |===============================================                       |  67%  |                                                                              |===================================================                   |  72%  |                                                                              |======================================================                |  78%  |                                                                              |==========================================================            |  83%  |                                                                              |==============================================================        |  89%  |                                                                              |==================================================================    |  94%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.0129 secs
#> - Finding orphan concepts for concept set 'Type 2 Diabetes Mellitus'
#>   |                                                                              |                                                                      |   0%  |                                                                              |====                                                                  |   6%  |                                                                              |========                                                              |  11%  |                                                                              |============                                                          |  17%  |                                                                              |================                                                      |  22%  |                                                                              |===================                                                   |  28%  |                                                                              |=======================                                               |  33%  |                                                                              |===========================                                           |  39%  |                                                                              |===============================                                       |  44%  |                                                                              |===================================                                   |  50%  |                                                                              |=======================================                               |  56%  |                                                                              |===========================================                           |  61%  |                                                                              |===============================================                       |  67%  |                                                                              |===================================================                   |  72%  |                                                                              |======================================================                |  78%  |                                                                              |==========================================================            |  83%  |                                                                              |==============================================================        |  89%  |                                                                              |==================================================================    |  94%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.00932 secs
#> - Finding orphan concepts for concept set 'History of diabetes'
#>   |                                                                              |                                                                      |   0%  |                                                                              |====                                                                  |   6%  |                                                                              |========                                                              |  11%  |                                                                              |============                                                          |  17%  |                                                                              |================                                                      |  22%  |                                                                              |===================                                                   |  28%  |                                                                              |=======================                                               |  33%  |                                                                              |===========================                                           |  39%  |                                                                              |===============================                                       |  44%  |                                                                              |===================================                                   |  50%  |                                                                              |=======================================                               |  56%  |                                                                              |===========================================                           |  61%  |                                                                              |===============================================                       |  67%  |                                                                              |===================================================                   |  72%  |                                                                              |======================================================                |  78%  |                                                                              |==========================================================            |  83%  |                                                                              |==============================================================        |  89%  |                                                                              |==================================================================    |  94%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.00966 secs
#> Warning: There were 2 warnings in `dplyr::summarise()`.
#> The first warning was:
#> ℹ In argument: `conceptCount = max(conceptCount)`.
#> Caused by warning in `max()`:
#> ! no non-missing arguments to max; returning -Inf
#> ℹ Run `dplyr::last_dplyr_warnings()` to see the 1 remaining warning.
#> Finding orphan concepts took 0.579 secs
#> Running concept set diagnostics took 2.13 secs
#> Retrieving visit context for index dates
#>   |                                                                              |                                                                      |   0%  |                                                                              |=======================                                               |  33%  |                                                                              |===============================================                       |  67%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.00846 secs
#> Retrieving visit context took 0.0574 secs
#> - Censoring 0 values (0%) from subjects because value below minimum
#> Computing incidence rates
#>   Computing incidence rate for cohort '[Cohort Diagnostics] Gastrointestinal hemorrhage for Eunomia'
#> Calculating incidence rate per year by age and gender
#> Inserting data took 0.0125 secs
#>   |                                                                              |                                                                      |   0%  |                                                                              |=====                                                                 |   7%  |                                                                              |=========                                                             |  13%  |                                                                              |==============                                                        |  20%  |                                                                              |===================                                                   |  27%  |                                                                              |=======================                                               |  33%  |                                                                              |============================                                          |  40%  |                                                                              |=================================                                     |  47%  |                                                                              |=====================================                                 |  53%  |                                                                              |==========================================                            |  60%  |                                                                              |===============================================                       |  67%  |                                                                              |===================================================                   |  73%  |                                                                              |========================================================              |  80%  |                                                                              |=============================================================         |  87%  |                                                                              |=================================================================     |  93%  |                                                                              |======================================================================| 100%
#> Executing SQL took 2.04 secs
#> Computing incidence rates took 2.19 secs
#>   Computing incidence rate for cohort '[PL 192671002] Gastrointestinal hemorrhage referent concept prevalent cohort: First occurrence of referent concept + descendants'
#> Calculating incidence rate per year by age and gender
#> Inserting data took 0.0294 secs
#>   |                                                                              |                                                                      |   0%  |                                                                              |=====                                                                 |   7%  |                                                                              |=========                                                             |  13%  |                                                                              |==============                                                        |  20%  |                                                                              |===================                                                   |  27%  |                                                                              |=======================                                               |  33%  |                                                                              |============================                                          |  40%  |                                                                              |=================================                                     |  47%  |                                                                              |=====================================                                 |  53%  |                                                                              |==========================================                            |  60%  |                                                                              |===============================================                       |  67%  |                                                                              |===================================================                   |  73%  |                                                                              |========================================================              |  80%  |                                                                              |=============================================================         |  87%  |                                                                              |=================================================================     |  93%  |                                                                              |======================================================================| 100%
#> Executing SQL took 2.05 secs
#> Computing incidence rates took 2.18 secs
#>   Computing incidence rate for cohort '[PL 192671001] Gastrointestinal hemorrhage referent concept incident cohort: First occurrence of referent concept + descendants with >=365d prior observation'
#> Calculating incidence rate per year by age and gender
#> Inserting data took 0.00514 secs
#>   |                                                                              |                                                                      |   0%  |                                                                              |=====                                                                 |   7%  |                                                                              |=========                                                             |  13%  |                                                                              |==============                                                        |  20%  |                                                                              |===================                                                   |  27%  |                                                                              |=======================                                               |  33%  |                                                                              |============================                                          |  40%  |                                                                              |=================================                                     |  47%  |                                                                              |=====================================                                 |  53%  |                                                                              |==========================================                            |  60%  |                                                                              |===============================================                       |  67%  |                                                                              |===================================================                   |  73%  |                                                                              |========================================================              |  80%  |                                                                              |=============================================================         |  87%  |                                                                              |=================================================================     |  93%  |                                                                              |======================================================================| 100%
#> Executing SQL took 2.02 secs
#> Computing incidence rates took 2.12 secs
#> - Censoring 1095 values (16%) from cohortCount because value below minimum
#> - Censoring 657 values (9.6%) from personYears because value below minimum
#> - Censoring 1089 values (15.9%) from incidenceRate because value below minimum
#> Running Incidence Rate took 6.6 secs
#> Computing Cohort Relationship
#>   |                                                                              |                                                                      |   0%  |                                                                              |==========                                                            |  14%  |                                                                              |====================                                                  |  29%  |                                                                              |==============================                                        |  43%  |                                                                              |========================================                              |  57%  |                                                                              |==================================================                    |  71%  |                                                                              |============================================================          |  86%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.216 secs
#>   |                                                                              |                                                                      |   0%  |                                                                              |==========                                                            |  14%  |                                                                              |====================                                                  |  29%  |                                                                              |==============================                                        |  43%  |                                                                              |========================================                              |  57%  |                                                                              |==================================================                    |  71%  |                                                                              |============================================================          |  86%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.225 secs
#>   |                                                                              |                                                                      |   0%  |                                                                              |==========                                                            |  14%  |                                                                              |====================                                                  |  29%  |                                                                              |==============================                                        |  43%  |                                                                              |========================================                              |  57%  |                                                                              |==================================================                    |  71%  |                                                                              |============================================================          |  86%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.201 secs
#>   |                                                                              |                                                                      |   0%  |                                                                              |==========                                                            |  14%  |                                                                              |====================                                                  |  29%  |                                                                              |==============================                                        |  43%  |                                                                              |========================================                              |  57%  |                                                                              |==================================================                    |  71%  |                                                                              |============================================================          |  86%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.217 secs
#>   |                                                                              |                                                                      |   0%  |                                                                              |==========                                                            |  14%  |                                                                              |====================                                                  |  29%  |                                                                              |==============================                                        |  43%  |                                                                              |========================================                              |  57%  |                                                                              |==================================================                    |  71%  |                                                                              |============================================================          |  86%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.217 secs
#>   |                                                                              |                                                                      |   0%  |                                                                              |==========                                                            |  14%  |                                                                              |====================                                                  |  29%  |                                                                              |==============================                                        |  43%  |                                                                              |========================================                              |  57%  |                                                                              |==================================================                    |  71%  |                                                                              |============================================================          |  86%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.201 secs
#>   |                                                                              |                                                                      |   0%  |                                                                              |==========                                                            |  14%  |                                                                              |====================                                                  |  29%  |                                                                              |==============================                                        |  43%  |                                                                              |========================================                              |  57%  |                                                                              |==================================================                    |  71%  |                                                                              |============================================================          |  86%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.214 secs
#>   |                                                                              |                                                                      |   0%  |                                                                              |==========                                                            |  14%  |                                                                              |====================                                                  |  29%  |                                                                              |==============================                                        |  43%  |                                                                              |========================================                              |  57%  |                                                                              |==================================================                    |  71%  |                                                                              |============================================================          |  86%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.205 secs
#>   |                                                                              |                                                                      |   0%  |                                                                              |==========                                                            |  14%  |                                                                              |====================                                                  |  29%  |                                                                              |==============================                                        |  43%  |                                                                              |========================================                              |  57%  |                                                                              |==================================================                    |  71%  |                                                                              |============================================================          |  86%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.164 secs
#>   |                                                                              |                                                                      |   0%  |                                                                              |==========                                                            |  14%  |                                                                              |====================                                                  |  29%  |                                                                              |==============================                                        |  43%  |                                                                              |========================================                              |  57%  |                                                                              |==================================================                    |  71%  |                                                                              |============================================================          |  86%  |                                                                              |======================================================================| 100%
#> Executing SQL took 0.163 secs
#> - Censoring 0 values (0%) from subjects because value below minimum
#> - Censoring 0 values (0%) from subCsBeforeTs because value below minimum
#> - Censoring 0 values (0%) from recCsBeforeTs because value below minimum
#> - Censoring 0 values (0%) from subCsOnTs because value below minimum
#> - Censoring 0 values (0%) from recCsOnTs because value below minimum
#> - Censoring 0 values (0%) from subCsAfterTs because value below minimum
#> - Censoring 0 values (0%) from recCsAfterTs because value below minimum
#> - Censoring 0 values (0%) from subCsBeforeTe because value below minimum
#> - Censoring 0 values (0%) from recCsBeforeTe because value below minimum
#> - Censoring 0 values (0%) from subCsOnTe because value below minimum
#> - Censoring 0 values (0%) from recCsOnTe because value below minimum
#> - Censoring 0 values (0%) from subCsAfterTe because value below minimum
#> - Censoring 0 values (0%) from recCsAfterTe because value below minimum
#> - Censoring 0 values (0%) from subCsWindowT because value below minimum
#> - Censoring 0 values (0%) from recCsWindowT because value below minimum
#> - Censoring 0 values (0%) from subCeWindowT because value below minimum
#> - Censoring 0 values (0%) from recCeWindowT because value below minimum
#> - Censoring 0 values (0%) from subCsWindowTs because value below minimum
#> - Censoring 0 values (0%) from recCsWindowTs because value below minimum
#> - Censoring 0 values (0%) from subCsWindowTe because value below minimum
#> - Censoring 0 values (0%) from recCsWindowTe because value below minimum
#> - Censoring 0 values (0%) from subCeWindowTs because value below minimum
#> - Censoring 0 values (0%) from recCeWindowTs because value below minimum
#> - Censoring 0 values (0%) from subCeWindowTe because value below minimum
#> - Censoring 0 values (0%) from recCeWindowTe because value below minimum
#> - Censoring 0 values (0%) from subCWithinT because value below minimum
#> - Censoring 0 values (0%) from recCWithinT because value below minimum
#> - Censoring 0 values (0%) from cDaysBeforeTs because value below minimum
#> - Censoring 0 values (0%) from cDaysBeforeTe because value below minimum
#> - Censoring 0 values (0%) from cDaysWithinTDays because value below minimum
#> - Censoring 0 values (0%) from cDaysAfterTs because value below minimum
#> - Censoring 0 values (0%) from cDaysAfterTe because value below minimum
#> - Censoring 0 values (0%) from tDays because value below minimum
#> - Censoring 0 values (0%) from cDays because value below minimum
#>     - Running Cohort Relationship iteration with batchsize 500 from row number 1 to 12 took 3.83 secs
#>  - Computing cohort relationships took 3.84 secs
#> Running Temporal Cohort characterization
#> Starting large scale characterization of 3 cohort(s)
#> Sending temp tables to server
#> Inserting data took 0.00609 secs
#> Constructing features on server
#>   |                                                                              |                                                                      |   0%  |                                                                              |                                                                      |   1%  |                                                                              |=                                                                     |   1%  |                                                                              |=                                                                     |   2%  |                                                                              |==                                                                    |   2%  |                                                                              |==                                                                    |   3%  |                                                                              |===                                                                   |   4%  |                                                                              |===                                                                   |   5%  |                                                                              |====                                                                  |   5%  |                                                                              |====                                                                  |   6%  |                                                                              |=====                                                                 |   6%  |                                                                              |=====                                                                 |   7%  |                                                                              |=====                                                                 |   8%  |                                                                              |======                                                                |   8%  |                                                                              |======                                                                |   9%  |                                                                              |=======                                                               |   9%  |                                                                              |=======                                                               |  10%  |                                                                              |=======                                                               |  11%  |                                                                              |========                                                              |  11%  |                                                                              |========                                                              |  12%  |                                                                              |=========                                                             |  12%  |                                                                              |=========                                                             |  13%  |                                                                              |=========                                                             |  14%  |                                                                              |==========                                                            |  14%  |                                                                              |==========                                                            |  15%  |                                                                              |===========                                                           |  15%  |                                                                              |===========                                                           |  16%  |                                                                              |============                                                          |  17%  |                                                                              |============                                                          |  18%  |                                                                              |=============                                                         |  18%  |                                                                              |=============                                                         |  19%  |                                                                              |==============                                                        |  19%  |                                                                              |==============                                                        |  20%  |                                                                              |==============                                                        |  21%  |                                                                              |===============                                                       |  21%  |                                                                              |===============                                                       |  22%  |                                                                              |================                                                      |  22%  |                                                                              |================                                                      |  23%  |                                                                              |=================                                                     |  24%  |                                                                              |=================                                                     |  25%  |                                                                              |==================                                                    |  25%  |                                                                              |==================                                                    |  26%  |                                                                              |===================                                                   |  26%  |                                                                              |===================                                                   |  27%  |                                                                              |===================                                                   |  28%  |                                                                              |====================                                                  |  28%  |                                                                              |====================                                                  |  29%  |                                                                              |=====================                                                 |  29%  |                                                                              |=====================                                                 |  30%  |                                                                              |=====================                                                 |  31%  |                                                                              |======================                                                |  31%  |                                                                              |======================                                                |  32%  |                                                                              |=======================                                               |  32%  |                                                                              |=======================                                               |  33%  |                                                                              |=======================                                               |  34%  |                                                                              |========================                                              |  34%  |                                                                              |========================                                              |  35%  |                                                                              |=========================                                             |  35%  |                                                                              |=========================                                             |  36%  |                                                                              |==========================                                            |  37%  |                                                                              |==========================                                            |  38%  |                                                                              |===========================                                           |  38%  |                                                                              |===========================                                           |  39%  |                                                                              |============================                                          |  39%  |                                                                              |============================                                          |  40%  |                                                                              |============================                                          |  41%  |                                                                              |=============================                                         |  41%  |                                                                              |=============================                                         |  42%  |                                                                              |==============================                                        |  42%  |                                                                              |==============================                                        |  43%  |                                                                              |===============================                                       |  44%  |                                                                              |===============================                                       |  45%  |                                                                              |================================                                      |  45%  |                                                                              |================================                                      |  46%  |                                                                              |=================================                                     |  46%  |                                                                              |=================================                                     |  47%  |                                                                              |=================================                                     |  48%  |                                                                              |==================================                                    |  48%  |                                                                              |==================================                                    |  49%  |                                                                              |===================================                                   |  49%  |                                                                              |===================================                                   |  50%  |                                                                              |===================================                                   |  51%  |                                                                              |====================================                                  |  51%  |                                                                              |====================================                                  |  52%  |                                                                              |=====================================                                 |  52%  |                                                                              |=====================================                                 |  53%  |                                                                              |=====================================                                 |  54%  |                                                                              |======================================                                |  54%  |                                                                              |======================================                                |  55%  |                                                                              |=======================================                               |  55%  |                                                                              |=======================================                               |  56%  |                                                                              |========================================                              |  57%  |                                                                              |========================================                              |  58%  |                                                                              |=========================================                             |  58%  |                                                                              |=========================================                             |  59%  |                                                                              |==========================================                            |  59%  |                                                                              |==========================================                            |  60%  |                                                                              |==========================================                            |  61%  |                                                                              |===========================================                           |  61%  |                                                                              |===========================================                           |  62%  |                                                                              |============================================                          |  62%  |                                                                              |============================================                          |  63%  |                                                                              |=============================================                         |  64%  |                                                                              |=============================================                         |  65%  |                                                                              |==============================================                        |  65%  |                                                                              |==============================================                        |  66%  |                                                                              |===============================================                       |  66%  |                                                                              |===============================================                       |  67%  |                                                                              |===============================================                       |  68%  |                                                                              |================================================                      |  68%  |                                                                              |================================================                      |  69%  |                                                                              |=================================================                     |  69%  |                                                                              |=================================================                     |  70%  |                                                                              |=================================================                     |  71%  |                                                                              |==================================================                    |  71%  |                                                                              |==================================================                    |  72%  |                                                                              |===================================================                   |  72%  |                                                                              |===================================================                   |  73%  |                                                                              |===================================================                   |  74%  |                                                                              |====================================================                  |  74%  |                                                                              |====================================================                  |  75%  |                                                                              |=====================================================                 |  75%  |                                                                              |=====================================================                 |  76%  |                                                                              |======================================================                |  77%  |                                                                              |======================================================                |  78%  |                                                                              |=======================================================               |  78%  |                                                                              |=======================================================               |  79%  |                                                                              |========================================================              |  79%  |                                                                              |========================================================              |  80%  |                                                                              |========================================================              |  81%  |                                                                              |=========================================================             |  81%  |                                                                              |=========================================================             |  82%  |                                                                              |==========================================================            |  82%  |                                                                              |==========================================================            |  83%  |                                                                              |===========================================================           |  84%  |                                                                              |===========================================================           |  85%  |                                                                              |============================================================          |  85%  |                                                                              |============================================================          |  86%  |                                                                              |=============================================================         |  86%  |                                                                              |=============================================================         |  87%  |                                                                              |=============================================================         |  88%  |                                                                              |==============================================================        |  88%  |                                                                              |==============================================================        |  89%  |                                                                              |===============================================================       |  89%  |                                                                              |===============================================================       |  90%  |                                                                              |===============================================================       |  91%  |                                                                              |================================================================      |  91%  |                                                                              |================================================================      |  92%  |                                                                              |=================================================================     |  92%  |                                                                              |=================================================================     |  93%  |                                                                              |=================================================================     |  94%  |                                                                              |==================================================================    |  94%  |                                                                              |==================================================================    |  95%  |                                                                              |===================================================================   |  95%  |                                                                              |===================================================================   |  96%  |                                                                              |====================================================================  |  97%  |                                                                              |====================================================================  |  98%  |                                                                              |===================================================================== |  98%  |                                                                              |===================================================================== |  99%  |                                                                              |======================================================================|  99%  |                                                                              |======================================================================| 100%
#> Executing SQL took 6.46 secs
#> Fetching data from server
#> Fetching data took 0.142 secs
#> Error in `covariates$"timeId"`:
#> ! The `$` method of <tbl_lazy> is for internal use only.
#> ℹ Use `dplyr::pull()` to get the values in a column.
#> Backtrace:
#>     ▆
#>  1. └─CohortDiagnostics::executeDiagnostics(...)
#>  2.   ├─CohortDiagnostics:::timeExecution(...)
#>  3.   │ └─base::eval(expr)
#>  4.   └─CohortDiagnostics:::executeCohortCharacterization(...)
#>  5.     └─CohortDiagnostics:::getCohortCharacteristics(...)
#>  6.       ├─covariates$timeId
#>  7.       └─dbplyr:::`$.tbl_lazy`(covariates, "timeId")
#>  8.         └─cli::cli_abort(...) at dbplyr/R/tbl-lazy.R:123:4
#>  9.           └─rlang::abort(...)

sessionInfo()
#> R version 4.2.2 (2022-10-31)
#> Platform: x86_64-apple-darwin17.0 (64-bit)
#> Running under: macOS Big Sur ... 10.16
#> 
#> Matrix products: default
#> BLAS:   /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRblas.0.dylib
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib
#> 
#> locale:
#> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] CohortDiagnostics_3.2.3 FeatureExtraction_3.3.0 Andromeda_0.6.3        
#> [4] dplyr_1.1.2             DatabaseConnector_6.2.3
#> 
#> loaded via a namespace (and not attached):
#>  [1] lubridate_1.9.2       here_1.0.1            tidyr_1.3.0          
#>  [4] SqlRender_1.15.2      rprojroot_2.0.3       digest_0.6.33        
#>  [7] utf8_1.2.3            R6_2.5.1              backports_1.4.1      
#> [10] reprex_2.0.2          RSQLite_2.3.1         Eunomia_1.0.2        
#> [13] evaluate_0.21         pillar_1.9.0          rlang_1.1.1          
#> [16] rstudioapi_0.15.0     blob_1.2.4            R.utils_2.12.2       
#> [19] R.oo_1.25.0           checkmate_2.2.0       rmarkdown_2.23       
#> [22] styler_1.10.1         CohortGenerator_0.8.0 readr_2.1.4          
#> [25] stringr_1.5.0         bit_4.0.5             munsell_0.5.0        
#> [28] compiler_4.2.2        xfun_0.39             pkgconfig_2.0.3      
#> [31] htmltools_0.5.5       tidyselect_1.2.0      tibble_3.2.1         
#> [34] fansi_1.0.4           dbplyr_2.3.3.9000     crayon_1.5.2         
#> [37] tzdb_0.4.0            withr_2.5.0           R.methodsS3_1.8.2    
#> [40] jsonlite_1.8.7        lifecycle_1.0.3       DBI_1.1.3            
#> [43] magrittr_2.0.3        scales_1.2.1          cli_3.6.1            
#> [46] stringi_1.7.12        vroom_1.6.3           cachem_1.0.8         
#> [49] fs_1.6.3              generics_0.1.3        vctrs_0.6.3          
#> [52] tools_4.2.2           RJSONIO_1.3-1.8       bit64_4.0.5          
#> [55] R.cache_0.16.0        glue_1.6.2            ParallelLogger_3.3.0 
#> [58] purrr_1.0.2           hms_1.1.3             parallel_4.2.2       
#> [61] fastmap_1.1.1         yaml_2.3.7            timechange_0.2.0     
#> [64] colorspace_2.1-0      memoise_2.0.1         rJava_1.0-6          
#> [67] knitr_1.43

Created on 2023-09-01 with reprex v2.0.2

ablack3 commented 10 months ago

Also had an error when running this on the ohdsi test postgres database.

image
azimov commented 10 months ago

@ablack3 In this context covariates is an Andromeda object so I'm wondering if this is related to any recent changes there and if we need to make alterations to support them? I had problems with the shelved Arrow backend usage, are there any changes to the interface with regard to this?

ablack3 commented 10 months ago

No changes to Andromeda but I think dbplyr had a release recently so maybe that is the cause. I'll try to dig into it and post my findings.

ablack3 commented 9 months ago

Well I have not had much time to dig into why this is failing (lots to do 🚀 ) but I did download the current release and run the devtools::test() on sqlite only. There were a couple test failures but a large number of warnings so I was surprised that this is passing R check. But actually when I check the Hades website I see it is failing R check according to the dashboard.

I know I have some strong opinions, maybe too many (and I absolutely don't mean to be overly critical and am so appreciative of the Hades codebase and maintainers and I'm as bad as anyone about not passing tests), but I really think that putting R packages on CRAN is a good idea because it prevents this situation. If you don't pass R check (which includes your tests) you get kicked off after a short period of time. So if you want to be on CRAN you have to pass your own tests. It's a good thing to enforce I think. In Hades you can fail R check and remain as the current release which I don't think is good practice.

Backtrace:
     ▆
  1. ├─CohortDiagnostics::uploadResults(...) at test-1-ResultsDataModel.R:163:4
  2. │ └─ResultModelManager::uploadResults(...) at CohortDiagnostics/R/ResultsDataModel.R:135:2
  3. │   └─base::lapply(unique(specifications$tableName), uploadTable)
  4. │     └─ResultModelManager (local) FUN(X[[i]], ...)
  5. │       └─readr::read_csv_chunked(...)
  6. │         └─readr:::read_delimited_chunked(...)
  7. │           └─readr:::read_tokens_chunked(...)
  8. │             └─readr:::read_tokens_chunked_(...)
  9. └─readr (local) `<fn>`(`<spc_tbl_[,7]>`, 1L)
 10.   └─private$callback(data, index)

[ FAIL 2 | WARN 23 | SKIP 0 | PASS 103 ]
ginberg commented 5 months ago

@ablack3 when using the latest version of OHDSI CohortDiagnostics, I am not getting the error from your reprex anymore. This seems to be the fix

azimov commented 1 week ago

Appears to be fixed