EC528-Fall-2023 / Kata-Containers-for-SPARK

Apache License 2.0
7 stars 1 forks source link

Implement KataContainerRuntime for Apache YARN Node Manager #23

Closed jwz16 closed 12 months ago

jwz16 commented 1 year ago

Eventually if we want to make YARN works with Kata Container, we need to:

  1. Clone the YARN source codes to local.
  2. Implement a KataContainerRuntime class (inherits from OCIContainerRuntime ) inside the current YARN Node Manager source codes.
  3. Change YARN codes to allow kata configuration that indicates the Node Manager should use KataContainerRuntime.
  4. Rebuild the YARN Node Manager based on the changed source codes.
  5. Deploy Spark + YARN with KataContainerRuntime enabled.

This will possibly end up with a PR to the Hadoop repo, but also very challenging.

jwz16 commented 1 year ago

Current properties may look like:

<property>
    <name>yarn.nodemanager.runtime.linux.allowed-runtimes</name>
    <value>default,docker</value>
    <description>
      Comma separated list of runtimes that are allowed when using
      LinuxContainerExecutor. The allowed values are default, docker, and
      javasandbox.
    </description>
</property>

We need to support kata in <value>

<property>
    <name>yarn.nodemanager.runtime.linux.allowed-runtimes</name>
    <value>default,docker,kata</value>
    <description>
      Comma separated list of runtimes that are allowed when using
      LinuxContainerExecutor. The allowed values are default, docker, and
      javasandbox.
    </description>
</property>