Control over spawned processes

[milad2golnia]

Thank you for this awesome product.

Assume I have following configurations for mirrord:

{
  "target": "pod/bear-pod",
  "feature": {
    "env": true,
    "fs": "write",
    "network": true
  }
}

And here is interested part of code:

const child = spawn('mkvmerge', [
        '--default-duration',
        `0:${framerate}fps`,
        sourcePath,
        '-o',
        destinationPath,
      ])

Currently, all spawned processes are running on local machine and since all files (including sourcePath file) is just available in remote machine (considering "fs": "write" in configuration), It produces an error and we are not able to debug such programs.

It would be great if you offer a solution which gives us ability to run these spawned commands on remote machine. Is there any solution?

[linear[bot]]

MBE-504 Control over spawned processes

[aviramha]

hi @milad2golnia - Thank you for the compliments. Unfortunately, this is not possible atm, and I don't think we will support it in the near future since it creates a lot of complexities.

[milad2golnia]

Hi @aviramha -

Thank you for your fast response. I agree that this feature will create a lot of complexities. Now I’m just wondering if my understanding of mirrord is correct. Could you guide me?

I noticed that mirrord handles the file system with the feature.fs option specifically for direct I/O access. In this example, which spawns a separate process, it cannot manage file system access because I’m accessing it indirectly (through another process).

Am I correct?

[aviramha]

mirrord propogates to any sub process, it's just that it can't remote-execute the process itself. if you'd have mkvmerge locally, it should have access to remote files.

[milad2golnia]

Hmmm Actually I have mkvmerge installed but process exists with code 2.

$ mkvmerge --version                                                                                                                 ✔ 
mkvmerge v87.0 ('Black as the Sky') 64-bit

[aviramha]

What is the source path / destination path? also can you do which mkvmerge and then file MKVMERGEPATH (replace with path)

[milad2golnia]

Here is the full command which is executing:

mkvmerge --default-duration 0:25fps ./media/da86527f-c75f-4e52-a98a-59186cbe0ba4/add17fb6-b571-434b-aaf8-7f1d45cc830f-1730724272420/recording-merged.webm -o ./media/da86527f-c75f-4e52-a98a-59186cbe0ba4/add17fb6-b571-434b-aaf8-7f1d45cc830f-1730724272420/add17fb6-b571-434b-aaf8-7f1d45cc830f-fixed-framerate.webm

Please note that media directory is not part of my committed code but it exists in Kubernetes pod (aka my-pod-name) which is already running. It's a Persistent Volume which is managed by Kubernetes. Here is my main configuration for mirrord:

{
  "target": {
    "path": {
      "deployment": "my-pod-name"
    },
    "namespace": "services"
  },
  "feature": {
    "env": {
      "override": {
        "ID": "1",
      }
    },
    "fs": "write",
    "network": true
  },
  "agent": {
    "namespace": "services",
    "startup_timeout": 360
  },
  "kubeconfig": "~/.kube/test",
  "kube_context": "test"
}

And here is output of commands you asked me (executed in local machine):

$ which mkvmerge
/usr/bin/mkvmerge
$ file /usr/bin/mkvmerge                                                      
/usr/bin/mkvmerge: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=83856843391ed34be5919fa0f6ff5318e46ca7a0, for GNU/Linux 4.4.0, stripped

Besides this, I tried to run ls command to see if I really have remote file system or not, here is my code:

  exec('ls', (err, stdout, stderr) => {
    if (err) {
      console.error(err)
      process.exit(1)
    } else {
      console.log(`The stdout Buffer from shell: ${stdout.toString()}`)
      console.log(`The stderr Buffer from shell: ${stderr.toString()}`)
    }
  })

And here is output:

dist Dockerfile nest-cli.json node_modules package.json README.md src test tsconfig.build.json tsconfig.json yarn.lock

As you can see there is no media directory.

[aviramha]

Thank you - the problem is probably because of the use of relative path - mirrord automatically sets paths that are relative to be local. Can you try the mkvmerge to have such command line?

mkvmerge --default-duration 0:25fps /media/da86527f-c75f-4e52-a98a-59186cbe0ba4/add17fb6-b571-434b-aaf8-7f1d45cc830f-1730724272420/recording-merged.webm -o /media/da86527f-c75f-4e52-a98a-59186cbe0ba4/add17fb6-b571-434b-aaf8-7f1d45cc830f-1730724272420/add17fb6-b571-434b-aaf8-7f1d45cc830f-fixed-framerate.webm

(I removed . from the path, so maybe you need to add the current directory instead)

[milad2golnia]

mirrord propogates to any sub process,

I really doubt it or maybe it's a bug with mirrord, even running ls / returns my root filesystem instead of remote machine root directory.

Here is my code to reproduce:

  exec('ls /', (err, stdout, stderr) => {
    if (err) {
      console.error(err)
      process.exit(1)
    } else {
      console.log(`The stdout Buffer from shell: ${stdout.toString()}`)
      console.log(`The stderr Buffer from shell: ${stderr.toString()}`)
    }
  })

And here is output:

bin boot data desktopfs-pkgs.txt dev etc home lib lib64 lost+found media mnt opt postgresImportingFILES proc root rootfs-pkgs.txt run sbin srv svr sys tmp usr var

These files all are those who located in my root system (indicating by desktopfs-pkgs.txt and postgresImportingFILES).

But also as you told me, I tried without relative path and now I get a strange error. Complete Code:

  async merge(
    framerate: number,
    sourcePath: string,
    destinationPath: string,
  ): Promise<string> {
    sourcePath = sourcePath.replace(/^\.\//, '/app/')
    destinationPath = destinationPath.replace(/^\.\//, '/app/')
    if (framerate <= 0) throw new Error(`Received ${framerate} as framerate.`)
    this.logger.verbose(
      `Merging matroska multimedia with: mkvmerge --default-duration 0:${framerate}fps ${sourcePath} -o ${destinationPath}`,
    )
    return new Promise((resolve, reject) => {
      const child = spawn('mkvmerge', [
        '--default-duration',
        `0:${framerate}fps`,
        sourcePath,
        '-o',
        destinationPath,
      ])

      child.stderr.on('data', (chunk) => {
        this.logger.warn(chunk)
      })
      child.on('error', (err) => {
        this.logger.error(
          `Merging matroska failed: ${err.name} | ${err.message}`,
          err.stack,
        )
        reject(err)
      })
      child.on('close', (code, signal) => {
        if (code !== 0) {
          this.logger.error(
            `Merging matroska video faild with code ${code} and signal ${signal}`,
          )
          reject(code)
        } else {
          return resolve(destinationPath)
        }
      })
    })
  }

Here is output:

VERBOSE [Matroska] Merging matroska multimedia with: mkvmerge --default-duration 0:25fps /app/media/da86527f-c75f-4e52-a98a-59186cbe0ba4/add17fb6-b571-434b-aaf8-7f1d45cc830f-1730724272420/recording-merged.webm -o /app/media/da86527f-c75f-4e52-a98a-59186cbe0ba4/add17fb6-b571-434b-aaf8-7f1d45cc830f-1730724272420/add17fb6-b571-434b-aaf8-7f1d45cc830f-fixed-framerate.webm
[Nest] 36046  - 11/04/2024, 9:41:24 PM    WARN [Matroska] terminate called after throwing an instance of 'boost::filesystem::filesystem_error'
  what():  boost::filesystem::status: Function not implemented [system:38]: "/app/media/da86527f-c75f-4e52-a98a-59186cbe0ba4/add17fb6-b571-434b-aaf8-7f1d45cc830f-1730724272420"

[milad2golnia]

Also, in case if it is useful, here is full output if I use relative paths:

VERBOSE [Matroska] Merging matroska multimedia with: mkvmerge --default-duration 0:25fps ./media/da86527f-c75f-4e52-a98a-59186cbe0ba4/add17fb6-b571-434b-aaf8-7f1d45cc830f-1730724272420/recording-merged.webm -o ./media/da86527f-c75f-4e52-a98a-59186cbe0ba4/add17fb6-b571-434b-aaf8-7f1d45cc830f-1730724272420/add17fb6-b571-434b-aaf8-7f1d45cc830f-fixed-framerate.webm
[Nest] 37923  - 11/04/2024, 9:45:23 PM   ERROR [Matroska] Merging matroska video faild with code 2 and signal null

[aviramha]

We resolve / locally by default as well https://github.com/metalbear-co/mirrord/blob/main/mirrord/layer/src/file/filter/read_local_by_default.rs

Based on the error, it might be that mkvmerge is calling a file syscall we don't support yet. You can open another issue for that and we that's more feasible then the original issue.

[milad2golnia]

Hmm, You're right.

We resolve / locally by default as well

I verified your claim by listing /app/ content and it returned the remote files. I just wonder why there are some directory patterns that always resolves locally?

Anyway, I agree with you, It's about a file syscall support and not propagating mirrord. I'll open another issue based on result of this discussion.

Thank you for helping me to narrow down cause of problem.

[milad2golnia]

I found the relative documentation:

mirrord will relay file access (except for some exceptions) to the target pod by default.

Link to documentation: https://mirrord.dev/docs/reference/fileops/