search

ctsit / redcapcustodian

Simplified, automated data management on REDCap systems
Other
12 stars 6 forks source link

Alternative to using SQLite in testing tools #37

Open ChemiKyle opened 2 years ago

ChemiKyle commented 2 years ago

While REDCap uses MySQL/MariaDB, we've used an in-memory SQLite database for ease of use in testing https://github.com/ctsit/redcapcustodian/blob/9c643e3fc67b25ace85c52aee4c4732f5b3a22f1/tests/testthat/test-redcap.R#L1

Syntax mismatches have already complicated schemas and are now making porting existing functions difficult (https://github.com/ctsit/redcapcustodian/issues/12#issuecomment-1050074644).

I'm looking into seeing if we can stand up a MariaDB database entirely in memory to avoid the need for local or containerized solutions. Another alternative may be to port all SQL commands to R equivalents and let DBI handle the command fabrication.

RAM MariaDB

Ideally we would be able to create an entirely in-memory MariaDB instance, this unfortunately is not as simple as dbConnect(RMariaDB::MariaDB(), dbname = ":memory:").

It appears MariaDB supports in-memory tables (see: https://stackoverflow.com/a/54849686/7418735 and https://mariadb.com/kb/en/memory-storage-engine/).

pbchase commented 7 months ago

I have been using duckdb for this purpose. it's data types are richer than SQLite and closer to MySQL's. I did a lot of this conversion in PR #143 and is eliminated 4 long-standing intermittent test failures. They were all caused by sqlite DBs accidentally appear on disk instead of in memory. Running library(dotenv) would change the sqlite behavior and tests would persist data on disk, then read it, and trip up on the unexpected database content.