Fix for Databricks ES-257648 to fix WSFS dir issues within Databricks Container Services
Added image and layer for ganglia support within Databricks Container Services
How is this patch tested?
[X] Unit tests
[X] Integration tests
[X] Manual tests
To fix Databricks ES-257648, we added a dcs-wsfs.init file to the dbfsfuse image which we install to /databricks/scripts/. This enables the WSFS dir to be freely changed in subsequent operations within Databricks Container Services
To enable Ganglia, we employed a containerised ganglia deployment in a "ganglia" image which has been configured to collect container level metrics for the Spark cluster driver and workers.
What changes are proposed in this pull request?
Fix for Databricks ES-257648 to fix WSFS dir issues within Databricks Container Services
Added image and layer for
gangliasupport within Databricks Container ServicesHow is this patch tested?
To fix Databricks ES-257648, we added a
dcs-wsfs.initfile to thedbfsfuseimage which we install to/databricks/scripts/. This enables the WSFS dir to be freely changed in subsequent operations within Databricks Container ServicesTo enable Ganglia, we employed a containerised
gangliadeployment in a "ganglia" image which has been configured to collect container level metrics for the Spark cluster driver and workers.