Provide a no-brainer docker-compose build

MIT-LCP / mimic-omop

Mapping the MIMIC-III database to the OMOP schema

MIT License

123 stars 47 forks source link

Provide a no-brainer docker-compose build #80

Open parisni opened 4 months ago

parisni commented 4 months ago

After few years of hibernation, we tried an install from scratch and noticed few issues, leading to very painful experience. This PR solve the issues, and also provide a no brain docker-compose script to build the mimic-omop database. The build duration is less than 1 hour on a laptop, and it is compatible with both mimic demo or latest mimic 3 export.

Some of the major issues:

postgresql > 11 handle CTE differently leading to hang the ETL for eternity
indexing and analyze lack on concepts tables made the ETL also last way to long

Painful aspects:

manual instruction is quite vague on some parts
the R script needs local installation with no instructions
some scripts were missing
the makefile has broken references

briangow commented 3 months ago

@parisni , thanks for this!

I've attempted to run the Docker MIMIC-III demo build. I added the files to the mimic/data-mimicdemo and extras/athena folders as described in the docker/README. I get numerous errors and ultimately docker-compose up fails with exited with code 2. Please see the attached docker.log for details.

When initially setting things up, I also noticed that a lot of the links in the primary README are broken.

parisni commented 3 months ago

Hi @briangow thanks for your attempt. From the logs most of the errors you have are no big deal, for example missing schemas that are then created.

Eventually the data is created, as this example shows:

etl-1  | INSERT 0 31010
etl-1  | Time: 10634.488 ms (00:10.634)

But then this one retains my attention:

sh: 1: cannot create etl/Result/attribute_definition.csv.gz: Directory nonexistent

You might have to create a Result folder in etl to fix this. Tell me if this help, I would then fix this PR accordingly