Explicitly setting "destination_path_format" in path mapping spec

[Mathieson]

Description

I am working on Windows locally for development, but when deployed the host machine will be Linux. My tasks also involve running Linux-based Docker containers. This makes the local development workflow slightly tricky as I'm already faced with needing to consider two different contexts in which my paths could exist.

The path-mapping feature for OpenJD could lend itself well to this scenario. It is essentially the same problem it is already solving, but Docker (and especially) on Windows shifts the problem earlier. With the bind mount for my Docker container, I figure I can declare the same mapping in the OpenJD path mapping and it will bridge the gap between the environments. I can then remain in the context of Windows pathing and openjd run can remap for me to what would be expected in my Dockerized worker.

The problem I am running into seems to be that path mapping implicitly determines the destination path format. The command still evaluates on Windows to call docker run and, therefore, it makes sense that it is producing backslashes instead of forward slashes,

I am currently working my way through the tutorial, modifying the approach slightly to this docker-style for tasks, which more closely aligns with how we will eventually work. Here is a portion of what I have so far for my files, to provide more insight into the approach.

Here is my current job template, tutorial.template.yaml

specificationVersion: jobtemplate-2023-09
name: "{{Param.JobName}}"

parameterDefinitions:
  - name: BindRoot
    type: PATH
    dataFlow: IN
    objectType: DIRECTORY

  - name: RenderContainer
    type: STRING
    default: blender-render:latest

  - name: SceneFile
    type: PATH
    dataFlow: IN
    objectType: FILE

  - name: JobName
    type: STRING
    minLength: 1
    default: "DemoJob"

steps:
  - name: BlenderRender
    script:
      actions:
        onRun:
          command: "docker"
          args:
            - "run"
            - "-v"
            - "{{RawParam.BindRoot}}:/app"
            - "{{Param.RenderContainer}}"
            - "{{Param.SceneFile}}"
          timeout: 60

Here is my path mapping, path-mapping.json...

{
  "version": "pathmapping-1.0",
  "path_mapping_rules": [
    {
      "source_path_format": "WINDOWS",
      "source_path": "D:\\open-jd-example\\tutorial_wip\\mfacer",
      "destination_path": "/app"
    }
  ]
}

My call is

openjd run --step BlenderRender .\tutorial.template.yaml -p BindRoot=$(pwd) -p SceneFile=$(pwd)\\3d\\pavillon_barcelone_v1.2.blend --path-mapping-rules file://path-mapping.json

The output logs look as follows...

...
Wed Aug 14 14:41:40 2024        Running command "C:\Program Files\Docker\Docker\resources\bin\docker.EXE" run -v D:\open-jd-example\tutorial_wip\mfacer:/app blender-render:latest \app\3d\pavillon_barcelone_v1.2.blend \app\output_frames 1 2
...
Wed Aug 14 14:41:40 2024        Error: Cannot read file "/app/\app\3d\pavillon_barcelone_v1.2.blend": No such file or directory
...

As you can see in the first line above, the paths \app\3d\pavillon_barcelone_v1.2.blend and \app\output_frames end up with backslashes. I suspect this is because the command is running on Windows still, being that I am in a local iterative workflow currently.

The 2nd line showing the path /app/\app\3d\pavillon_barcelone_v1.2.blend I suspect is a side effect of the path getting passed to the container containing backslashes. It doesn't recognize it as a valid path when in Linux and attempts to append it to $pwd instead.

Proposed Solution

Checking the path mapping specification, there is an option for source_path_format but nothing for destination_path_format. Is destination_path_format something that can be added so that if it is provided, the path will get remapped to it? This would allow forcing to POSIX paths even when running on a Windows OS so the paths will be primed for the Docker container.

{
  "version": "pathmapping-1.0",
  "path_mapping_rules": [
    {
      "source_path_format": "WINDOWS",
      "source_path": "D:\\open-jd-example\\tutorial_wip\\mfacer",
      "destination_path_format": "POSIX",
      "destination_path": "/app"
    }
  ]
}

[jusiskin]

Hello @Mathieson. Thanks for reaching out and letting us know about this issue you are encountering. Your detailed and well-written write up of the problem is appreciated and helps understand your context and motivation here.

I'm wondering if this can be approached in a different way that may work for your use-case. Have you considered the possibility of adding the OpenJD CLI inside the container? It would require adding a line to your Dockerfile such as :

RUN pip install openjd-cli

You'd then need the job template and path mapping files to be made available inside the container (e.g. using bind mount).

The workflow on your development machine would have to switch from

openjd run ...

to:

docker run <CONTAINER_IMAGE> openjd run ...

This might be a good way to achieve the desired workflow. Let me know what you think?

[Mathieson]

Hi @jusiskin, thanks for responding! I had previously considered that option but decided not to proceed because of DinD and some unknowns there. Thanks for the suggestion; it might be worth exploring again when I have time. I've found success currently with a combination of WSL and Hatch.