search

hashicorp / terraform-provider-nomad

Terraform Nomad provider
https://registry.terraform.io/providers/hashicorp/nomad/latest
Mozilla Public License 2.0
144 stars 101 forks source link

Resource `nomad_volume`: EnvAccessKeyNotFound: failed to find credentials in the environment #376

Closed timurkhisamov closed 1 year ago

timurkhisamov commented 1 year ago

Hi there,

Terraform Version

Terraform v1.5.4
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v5.13.0
+ provider registry.terraform.io/hashicorp/nomad v1.4.20
+ provider registry.terraform.io/hashicorp/vault v3.19.0

Nomad Version

nomad server 
Name                Address      Port  Status  Leader  Raft Version  Build  Datacenter      Region
hashi-0.infra-jp    10.0.11.239  4648  alive   true    3             1.3.5  infra-jp-aws    infra-jp
hashi-1.infra-jp    10.0.12.186  4648  alive   false   3             1.3.5  infra-jp-aws    infra-jp
hashi-2.infra-jp    10.0.13.84   4648  alive   false   3             1.3.5  infra-jp-aws    infra-jp

Provider Configuration

provider "nomad" {
  address = "https://nomad.domainname.com"
}

Environment Variables

NOMAD_REGION=infra-jp
NOMAD_ADDR=https://nomad.domainname.com
NOMAD_NAMESPACE=services

AWS_ACCESS_KEY_ID=AKIA***
AWS_SECRET_ACCESS_KEY=oNLF***

Affected Resource(s)

Expected Behavior

Create the nomad_volume

resource "aws_ebs_volume" "gitea" {
  availability_zone = "ap-northeast-1a"
  size              = 30
  type              = "gp3"
}

data "nomad_plugin" "ebs" {
  plugin_id        = "ebs-0"
  wait_for_healthy = true
}

resource "nomad_volume" "gitea" {
  type        = "csi"
  namespace   = "services"
  plugin_id   = data.nomad_plugin.ebs.plugin_id
  volume_id   = "gitea"
  name        = "gitea"
  external_id = aws_ebs_volume.gitea.id

  capability {
    access_mode     = "single-node-writer"
    attachment_mode = "file-system"
  }

  mount_options {
    fs_type     = "ext4"
    mount_flags = ["noatime"]
  }
}

Actual Behavior

 Error: error registering volume: Unexpected response code: 500 (rpc error: rpc error: controller validate volume: rpc error: controller validate volume: CSI.ControllerValidateVolume: controller plugin returned an internal error, check the plugin allocation logs for more information: rpc error: code = Internal desc = Could not get volume with ID "vol-015ce5234a8f11230b": NoCredentialProviders: no valid providers in chain
│ caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
│ SharedCredsLoad: failed to load profile, .
│ EC2RoleRequestError: no EC2 instance role found
│ caused by: EC2MetadataError: failed to make EC2Metadata request
│ <?xml version="1.0" encoding="iso-8859-1"?>
│ <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
│        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
│ <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
│  <head>
│   <title>404 - Not Found</title>
│  </head>
│  <body>
│   <h1>404 - Not Found</h1>
│  </body>
│ </html>
│
│   status code: 404, request id:)
│
│   with nomad_volume.gitea,
│   on volumes.tf line 12, in resource "nomad_volume" "gitea":
│   12: resource "nomad_volume" "gitea" {

Steps to Reproduce

terraform apply

It worked on hashicorp/nomad v1.4.18. I've tried to use hashicorp/nomad v1.4.18, hashicorp/nomad v1.4.19, hashicorp/nomad v1.4.20 and hashicorp/nomad 2.0.0-rc.2 but it doesn't work now. EBS disk vol-015ce5234a8f11230b is present and available.

lgfa29 commented 1 year ago

Hi @timurkhisamov 👋

It seems like your CSI plugin is missing the proper credentials? Could you check if the plugin alloc logs have anything relevant?

timurkhisamov commented 1 year ago

@lgfa29, I found in node container (aws-ebs-csi-driver with args node) same error. I've updated the base image to ubuntu 22.04 and aws-ebs-csi-driver to latest version too. And it works. Thank you very much!

lgfa29 commented 1 year ago

Nice! I am going to close this issue then. Feel free to reach out again if you find any other problem!