Apache Spark Containerization

sidataplus / omop-core

Part of OMOP NHSO TCELS project

0 stars 0 forks source link

Closed ThanePi closed 1 year ago

ThanePi commented 1 year ago

Used as E-L backbone in CDM linkage and to push from CDM instance to analytics instance.

The containerization of this Spark will be customized with additional requirement include: Python + Dependency and JDBC Driver.

Most of container work will be foundation support for package of "crosspipe" which will be described on next issue, or visit its early proposal at: https://github.com/sidataplus/omop-core/blob/d8d35349e2d44502689eb8a1aa6f136de398119c/container/module/TCEL/Spark/README

ThanePi commented 1 year ago

Backlog:

commit https://github.com/sidataplus/omop-core/commit/0fe8ca3c9e86c39d43c088598bd111079be0631b
Quick check was trying to add JDBC driver of Postgres into Spark's Scala classpath but appeared as already exist.

ThanePi commented 1 year ago

Backlog:

commit https://github.com/sidataplus/omop-core/commit/6b680611b2e57db3198933a7dea0a9d715cca3ee
Add wod.sh script to start PySpark container, The wod.sh script is used to start the PySpark container and run the crosspipe application. This script is added to the container during the build process.

ThanePi commented 1 year ago

Closed by #54