enabling multiple job submissions and ucx classpath fix

cylondata / twister2

A composable framework for fast and scalable data analytics

https://twister2.org

Apache License 2.0

57 stars 32 forks source link

enabling multiple job submissions and ucx classpath fix #895

Closed ahmet-uyar closed 4 years ago

ahmet-uyar commented 4 years ago

Enabled multiple concurrent job submissions in LocalFileSystemUploader.java. Copied job package to a job specific folder under .twister2/repository/jobID Removed job package after the job ends. Added ucx lib folder to classpath in k8s docker container. Added an empty directory for ucx in lib/ucx/ucx