search

sul-dlss-deprecated / dataOps

data operations ("dataOps") repo for issue queues & any version-controlled documentation
1 stars 1 forks source link

Work with Dev-Ops & Ops to Spin Up Spark & Possibly Kafka Dev Infrastructure #2

Closed cmharlow closed 6 years ago

cmharlow commented 7 years ago

There is an over-arching need to have test, development, and production infrastructure in place to deploy Spark, possibly Kafka or another stream log system, applications to. This is probably in AWS but that is open to discussion.

This originally emerged in LD4P Kafka Spark Conversion Pipeline Work Cycle 1, but is a larger need across projects wanting to employ data engineering approaches. Work with @eefahy to figure out requirements and how to proceed functionally.

cmharlow commented 7 years ago

Being clarified via the Logs Aggregation to AWS Pipeline experiment / project going on currentl.