Multi remote feature

[courentin]

Hello, we have a dvc repository with 2 remotes and intentionally no default remote. We need that because we have a data residency constraint by region.

Would it be possible to specify multiple data remote by projects?

[shcheklein]

@courentin qq - do you specify remote names in each .dvc file / dvc.yaml stage outputs?

[courentin]

Yes I do

[courentin]

Hello, do you have any news on this features? Is there a chance that it gets prioritized or do you have a workaround? :)

Thank you!

[shcheklein]

@courentin no updates yet.

Question. It's the same repo, but do you use separate remotes for different projects within that repository or it's within one repo, within one pipeline you have multiple remotes?

[courentin]

It's one pipeline with multiple remotes.

Basically we have the same pipeline that runs twice thanks to the foreach feature, and each run has a different remote.

Something like:

stages:
  train:
    foreach: languages
    do:
      cmd: python ..
      outs:
       - my_out:
         remote: ${key}

[shcheklein]

@courentin would it be enough if we allow specifying a remote name (and for now attach only one remote per project still). This way you would be able to create two projects (aka views). Would it solve the problem for now?

[courentin]

That would solve the issue yes!

I'd hope a solution where we'll have only one project for both remotes, so that we avoid duplicating settings and we can easily compare models from different remotes on the long run. But that's a sufficient workaround for now :)

[Suor]

@courentin so we are working on it. No release promises yet, but still should be a good news)

[jellebouwman]

@courentin I just merged the front-end part of this feature, I'll let you know when we make a new release that adds this functionality!

[jellebouwman]

@courentin we just released this functionality on production. Please check out the Data remotes / cloud storage credentials section of your Project settings page! Let us know if this works for you.

[courentin]

Super, thank you very much! Perfect timing, we planned to work on the multi-region/remote next week, I'll be able to confirm if it solve our issue.

[courentin]

I can confirm that metrics are showing up in my other remote, thank you very much!