opentofu: plugins not found due to missing registry override

[KiaraGrouwstra]

Describe the bug

opentofu.withPlugins still fails to override registry.terraform.io to registry.opentofu.org. as a result of this, it will fail to find used plugins, searching in a nix store path containing registry.terraform.io rather than registry.opentofu.org.

Steps To Reproduce

Steps to reproduce the behavior:

1. use opentofu.withPlugins

2. use a plugin in a .tf, e.g.:

   terraform {
     required_providers {
       nomad = {
         source = "hashicorp/nomad"
       }
     }
   }
   
   provider "nomad" {
     address = "http://127.0.0.1:4646"
   }

3. run tofu init

4. find error not unlike:

   │ Error: Failed to query available provider packages
   │ 
   │ Could not retrieve the list of available versions for provider hashicorp/nomad: provider registry.opentofu.org/hashicorp/nomad was not found in any of
   │ the search locations
   │ 
   │   - /nix/store/gaybr4b1wqlkfyppd0bnb0qwszki3az5-opentofu-1.6.0-rc1/libexec/terraform-providers

Expected behavior

plugins work using opentofu as well

Screenshots

-

Additional context

• note this technically applies to both withPlugins as well as its derivatives like full and passthru.
• while a workaround might be patching this to straight-up override the plugin domains, it would be nice to get a proper solution here i.e. that would not break the plugins for terraform.

Notify maintainers

@gmemstr @nickcao @zowoq

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.7.0-zen2, NixOS, 24.05 (Uakari), 24.05.20240114.dd5621d`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - channels(kiara): `"home-manager"`
 - channels(root): `""`
 - nixpkgs: `/nix/store/sl31dgnpwinzvzhh7ak2rx3h6w7kbb30-source`

[NickCao]

A workaround is to specify the full path of the plugin, e.g. source = "registry.terraform.io/hashicorp/nomad".

[NickCao]

Do you know about the current status of the opentofu registry? Is is a mirror of the terraform registry or something else?

[adamcstephens]

They have a separate submission process, so it must not be just a mirror: https://github.com/opentofu/registry/tree/main#adding-providers-modules-or-gpg-keys-to-the-opentofu-registry

[NickCao]

A workaround is to specify the full path of the plugin, e.g. source = "registry.terraform.io/hashicorp/nomad".

Then I would propose this as a solution (instead of just a workaround).

[KiaraGrouwstra]

hm, long-term that would seem a bit like making opentofu a second-class citizen in nixpkgs, whereas it would seem more likely given its license and broad backing it will outlive terraform

[NickCao]

hm, long-term that would seem a bit like making opentofu a second-class citizen in nixpkgs, whereas it would seem more likely given its license and broad backing it will outlive terraform

Having a default registry other than it's own is not making it a second-class citizen. Take podman as an example, despite having it's own (partly so) registry quay.io, still mostly uses docker.io as the default.

[KiaraGrouwstra]

opentofu seems to use its own registry by default rather than terraform's

[Sorixelle]

Unfortunately, it seems like OpenTofu doesn't have a choice - the Terraform Registry's ToS doesn't allow OpenTofu to use it (source).

It doesn't look like we're downloading providers from the registry, so we shouldn't be running afoul of that clause. We just need to use a different folder name for Terraform (registry.terraform.io) and OpenTofu (registry.opentofu.org). I don't think that's impossible to implement (at least, after a brief glance at the terraform-providers implementation).

[zowoq]

https://github.com/NixOS/nixpkgs/pull/269106

I've got a draft PR that will drop the terraform registry in favour of just using the provider github repos directly, it'll be simple to include an option to use either registry.terraform.io or registry.opentofu.org in that PR.

I plan to finish the PR sometime in the next few weeks.

[yajo]

A workaround I found for now, which implies rebuilding the providers:

{
  lib,
  opentofu,
}: let
  # HACK https://github.com/NixOS/nixpkgs/issues/283015
  tofuProvider = provider:
    provider.override (oldArgs: {
      provider-source-address =
        lib.replaceStrings
        ["https://registry.terraform.io/providers"]
        ["registry.opentofu.org"]
        oldArgs.homepage;
    });
in
  opentofu.withPlugins (ps:
    builtins.map tofuProvider [
      ps.aws
      ps.hcloud
    ])

[justinas]

A workaround is to specify the full path of the plugin, e.g. source = "registry.terraform.io/hashicorp/nomad".

Does not seem to work anymore. Due to the aforementioned T&Cs, OpenTofu seem to be forcing the providers to their own registry. https://github.com/opentofu/opentofu/issues/1478

[NickCao]

Guess we have to package the opentofu registry now.