Unable to login into ECS instance

spokofono commented 4 years ago

Terraform Version

1.12.24

Terraform Configuration Files


locals {
  # openstack --insecure --os-volume-api-version 2 catalog list
  endpoints = "${map(
    "ecs", "https://ecs-endpoint/v2/2849a2f26ff845d288b00155e32af891",
    "vpc", "https://vpc-endpoint",
    "ims", "https://ims-endpoint",
    "evs", "https://evs-endpoint/v2/2849a2f26ff845d288b00155e32af891"
  )}"
}

provider "huaweicloudstack" {
  user_name = "user"
  password = "pass"
  domain_name = "domain"
  region = "region"
  tenant_id = "2849a2f26ff845d288b00155e32af891"
  auth_url    = "https://auth-endpoint:443/v3/"
  insecure = true
  endpoints = local.endpoints
}

resource "huaweicloudstack_blockstorage_volume_v2" "volume_1" {
  availability_zone = "az1.dc0"
  name        = "volume_1"
  description = "first test volume"
  size        = 10
  image_id    = "143034d7-dddc-4a3e-bf37-6c9fd96e914d"
  #image_id  = "4ef76506-13cb-41ce-94d6-9cf523caf6b4"
  volume_type = "SSD01"
}

resource "huaweicloudstack_compute_instance_v2" "testsrv" {
  #name              = "basic"
  flavor_name       = "s3.small.1"
  key_pair          = "authorized_keys_prod"
  security_groups   = ["default", "base"]
  availability_zone = "az1.dc0"
  region            = "region"
  user_data         = file("user_data.yml")

  network {
    uuid = "0882a9e9-6465-45c8-a904-96dbc916f226"
  }

  block_device {
    uuid                  = huaweicloudstack_blockstorage_volume_v2.volume_1.id
    source_type           = "volume"
    destination_type      = "volume"
    boot_index            = 0
    delete_on_termination = false
  }
}

and the user-data.yml:

#cloud-config

#hostname: basic

#fqdn: basic

users:
  - name: vagrant
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: false
    ssh_authorized_keys:
      - ssh-ed25519 publickeyhash user@localhost.localdomain

Debug Output

Problems I found in debug:

2020/05/19 10:47:54 [WARN] Provider "registry.terraform.io/-/huaweicloudstack" produced an invalid plan for huaweicloudstack_compute_instance_v2.testsrv, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .user_data: planned value cty.StringVal("9d22cfda7aaf5d84ec041f3a8450263870bb3063") does not match config value cty.StringVal("#cloud-config\n\n#hostname: basic\n\n#fqdn: basic\n\nusers:\n  - name: vagrant\n    sudo: ALL=(ALL) NOPASSWD:ALL\n    lock_passwd: false\n    ssh_authorized_keys:\n      - ssh-ed25519 publickeyhash user@localhost.localdomain")
      - .stop_before_destroy: planned value cty.False does not match config value cty.NullVal(cty.Bool)
      - .network[0].access_network: planned value cty.False does not match config value cty.NullVal(cty.Bool)

Instance was created:

2020-05-19T10:48:31.823+0200 [DEBUG] plugin.terraform-provider-huaweicloudstack: 
2020/05/19 10:48:31 [DEBUG] Retrieved Server 2af44c4e-d351-421e-9604-63e90c372a25: &{ID:2af44c4e-d351-421e-9604-63e90c372a25 TenantID:2849a2f26ff845d288b00155e32af891 UserID:5ec4608164894831af37282a1b638807 Name:basic Updated:2020-05-19 08:46:32 +0000 UTC Created:2020-05-19 08:46:10 +0000 UTC HostID:95ba94f1434e48f605b801922b59a697c9f3d3f6d9213887d6960eda Status:ACTIVE Progress:0 AccessIPv4: AccessIPv6: Image:map[id:143034d7-dddc-4a3e-bf37-6c9fd96e914d links:[map[href:https://ecs-endpoint/2849a2f26ff845d288b00155e32af891/images/143034d7-dddc-4a3e-bf37-6c9fd96e914d rel:bookmark]]] Flavor:map[id:bf92db71-4968-41a2-9d3a-3c1972529365 links:[map[href:https://ecs-endpoint/2849a2f26ff845d288b00155e32af891/flavors/bf92db71-4968-41a2-9d3a-3c1972529365 rel:bookmark]]] Addresses:map[soc-test-services:[map[OS-EXT-IPS-MAC:mac_addr:fa:16:3e:8d:16:77 OS-EXT-IPS:type:fixed addr:172.16.1.146 version:4]]] Metadata:map[cascaded.instance_extrainfo:current_mem:1024,max_mem:4194304,max_cpu:1,cpu_num_for_one_plug:1,org_cpu:1,xml_support_live_resize:False,num_of_mem_plug:0,org_mem:1024,iohang_timeout:720,current_cpu:1,pcibridge:2] Links:[map[href:https://ecs-endpoint/v2/2849a2f26ff845d288b00155e32af891/servers/2af44c4e-d351-421e-9604-63e90c372a25 rel:self] map[href:https://ecs-endpoint/2849a2f26ff845d288b00155e32af891/servers/2af44c4e-d351-421e-9604-63e90c372a25 rel:bookmark]] KeyName:authorized_keys_prod AdminPass: SecurityGroups:[map[name:base] map[name:default]] Fault:{Code:0 Created:0001-01-01 00:00:00 +0000 UTC Details: Message:} VolumesAttached:[map[id:4fa48ec3-bd27-4117-8cad-70ccdd96a525]]}
b00155e32af891/servers/2af44c4e-d351-421e-9604-63e90c372a25

But on the end there is a WARN:

T10:48:33.561+0200 [DEBUG] plugin.terraform-provider-huaweicloudstack: }
2020/05/19 10:48:33 [WARN] Provider "registry.terraform.io/-/huaweicloudstack" produced an unexpected new value for huaweicloudstack_compute_instance_v2.testsrv, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .tags: was null, but now cty.SetVal([]cty.Value{cty.StringVal("ce034d13-1957-4b0f-87ef-6b92baa6f19b")})
      - .block_device[0].guest_format: was null, but now cty.StringVal("")
      - .block_device[0].volume_size: was null, but now cty.NumberIntVal(0)
      - .block_device[0].volume_type: was null, but now cty.StringVal("")
      - .block_device[0].device_name: was null, but now cty.StringVal("")

Crash Output

There is no crash, instance is created.

Expected Behavior

I can login with ssh key.

Actual Behavior

I can't login with ssh key.

Notes

If I create instance manually with same key, I can login without problems. I tried centos8 and centos7 with same result. It seems that whatever I write in user-data.yml has no impact on instance. Interesting is that webconsole show correct key on instance.

What can I do now?

Thank you for help.

spokofono commented 4 years ago

I've got some new foundings:

for ubuntu 18.04 and 20.04 ssh key authentication works, but admin_pass doesn't work, user root has no password inside instance, I also tried user-data with
```
- name: root
lock_passwd: false
passwd: "$1$saltedpassword"
```
Saltedpassword generated throuh openssl password -1 -salt saltsalt password. Still root password remains empty.
machines created through terraform have no cdrom, I did not find a way how to add it (although it should not be problem)
if I set delete_on_termination = true in block_device section of compute instance resource, instance and volume will be destroyed, but terraform still thinks the volume exists:
```
Error: Error creating HuaweiCloudStack server: Resource not found: [POST https://ecs-endpoint/v2/2849a2f26ff845d288b00155e32af891/os-volumes_boot], error message: {"itemNotFound": {"message": "Volume f4a1a63b-3cc6-4976-a41f-8d5f84cc3b54 could not be found.", "code": 404}}
```
On next run terraform refreshes volumes and then the volume will be created. But this is not directly connected with my issue.
I tried these images https://cloud.centos.org/centos/8/x86_64/images/ - genericCloud and ec2 - none worked

Now trying to find out how cloudinit differs between ubuntu and centos.

ShiChangkuo commented 4 years ago

@spokofono sorry for late. Please feel free about the warning message in logging. The issue was caused by cloudinit or your images, you can get help form the technical support of HCS.

ShiChangkuo commented 4 years ago

Also you can raise another issue and give more details about the error message: "Volume f4a1a63b-3cc6-4976-a41f-8d5f84cc3b54 could not be found."

thanks!

spokofono commented 4 years ago

Hi ShiChangkuo,

Thanks for your response. You are right, I found a problem in cloud-init. Quick solution - use config_drive: true within a instance

Reason: Without config_drive: true an instance is trying to get config from network. Builtin version of cloudinit in newest centos has bug:

failed run of stage init                                                                 
------------------------------------------------------------                             
Traceback (most recent call last):                                                       
  File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 652, in status_wrapper                                                                                       
    ret = functor(name, args)                                                            
  File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 362, in main_init  
    init.apply_network_config(bring_up=bool(mode != sources.DSMODE_LOCAL))               
  File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 649, in apply_network_config                                                                                   
    netcfg, src = self._find_networking_config()                                         
  File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 636, in _find_networking_config                                                                                
    if self.datasource and hasattr(self.datasource, 'network_config'):      
  File "/usr/lib/python3.6/site-packages/cloudinit/sources/DataSourceOpenStack.py", line 115, in network_config                                                                    
    self.network_json, known_macs=None)     
  File "/usr/lib/python3.6/site-packages/cloudinit/sources/helpers/openstack.py", line 645, in convert_net_json                                                                    
    'Unknown network_data link type: %s' % link['type'])                                 
ValueError: Unknown network_data link type: cascading

The newest centos has builtin version 18.5, but ubuntu has 20.1 and there is this bug solved, so cloudinit run without errors.

The reason why I was able to start instance manually through console is that huawei's openstack use configdrive module, not network. So the finding with cdrom was actually solution.

So there are two options:

Use config_drive: true openstack will generate cd drive and mount it into instance. Easy.
Update cloud-init in centos Start manually instance, get in, update cloud-init, make an image from instance and then use this new image. Harder.

Sorry for bothering you.

Thank you.

huaweicloud / terraform-provider-huaweicloudstack