Add support for dynamic placeholders in the POD_MANIFEST_TEMPLATE

[fw8]

Is your feature request related to a problem?

Yes, the current implementation of the POD_MANIFEST_TEMPLATE lacks support for dynamic placeholders, making it challenging to handle varying workspace paths (and other settings) in the devpod container.

Which solution do you suggest?

I propose adding support for interpolation/templating in the pod config. This enhancement would enable users to use placeholders or variables, allowing for dynamic insertion of values such as the path to the workspace.

Which alternative solutions exist?

Currently, a separate provider instance must be created for each project, which then points to the corresponding static template.

Additional context

Example use case:

In scenarios where bind mounts in the workspace of the devpod container with a dind sidecar are necessary, the workspace must also be mounted in the dind container. The path to the workspace varies based on the workspace name, making it challenging to set up consistent configurations.

Introducing templating or placeholders in the POD_MANIFEST_TEMPLATE would greatly enhance flexibility and ease of configuration, benefiting users facing similar challenges.

For example:

volumeMounts:
- mountPath: /workspaces/${WORKSPACE}
  name: dind
  subPath: devpod/0

[pascalbreuninger]

Moved to provider repository, hopefully it'll get more visibility here