Can't upload ISO - secondary storage not found

[tetra12]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Storage, UI

CLOUDSTACK VERSION

4.16.0-Snapshot

CONFIGURATION

KVM Advanced Zone Local Storage NO SSL/TLS

OS / ENVIRONMENT

Ubuntu 21.04

SUMMARY

Issue similar to #4734

Can't upload an ISO template - getting Request Failed (530) There is no secondary storage VM for downloading template to image store adv-secondary-1

Meanwhile, I do have a secondary storage:

(localcloud) 🐱 > list imagestores 
{
  "count": 1,
  "imagestore": [
    {
      "id": "bb112cc8-49ba-4b21-a558-ef4587b5ddec",
      "name": "adv-secondary-1",
      "protocol": "nfs",
      "providername": "NFS",
      "readonly": false,
      "scope": "ZONE",
      "url": "nfs://192.168.100.2/export/secondary",
      "zoneid": "a4c6793b-3f50-48ce-a2fb-665b96231c18",
      "zonename": "adv-zone-1"
    }
  ]
}

Secondary storage was mounted by hand inside SSVM and is currently accesimble from SSVM as per ssvm.log

However, when I try to upload ISO from Local, I'm getting the following: Request Failed (530) There's no secondary storage VM for downloading template to image stora adv-secondary-1

I also tried to upload an ISO from a local web server ( on the same host, where ACS management and agent are running). The ISO won't be uploaded despite the message "ISO was registered successfully" was displayed

STEPS TO REPRODUCE

1. Clone ACS master
2. Build deb packages for Ubuntu 21.04
3. Install ACS
4. Update the Global Cofiguration:
 - System.vm.use.local.storage -> true
 - Ca.plugin.root.auth.strictness -> false
5. Setup Advanced Zone without Security Groups
6. Launch Zone
7. Log into SSVM and run ssvm-check.sh
8. Mount a secondary storage by hand if needed ( I did in my case )
9. Run `cmk list imagestores` on a laptop and 
10. Go to Images -> Isos -> Upload ISO from Local
11. Specify the file
12. Click Add and get an error

EXPECTED RESULTS

Upload an ISO from Local

ACTUAL RESULTS

Upload ISO from Local fails with `Request Failed (530) There is no secondary storage VM for downloading template to image store`

[tetra12]

UPD: Inside the SSVM, I found the following:

2021-08-25 19:20:57,397 INFO  [utils.nio.NioClient] (main:null) Connecting to 185.185.x.y:8250
2021-08-25 19:21:00,457 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host

Here's what I have:

root@s-8-VM:~# curl -vv telnet://185.185.x.y:8250
*   Trying 185.185.x.y:8250...
* connect to 185.185.x.y port 8250 failed: No route to host
* Failed to connect to 185.185.x.y port 8250: No route to host
* Closing connection 0
curl: (7) Failed to connect to 185.185.x.y port 8250: No route to host
root@s-8-VM:~# 
root@s-8-VM:~# curl -vv telnet://192.168.100.2:8250
*   Trying 192.168.100.2:8250...
* Connected to 192.168.100.2 (192.168.100.2) port 8250 (#0)

185.185.x.y is node addr on cloudbr0 and 192.168.100.2 is the host addr on cloudbr1. How to setup correct bridges for SSVM?

Here's the dump from my SQL:

mysql> select * from physical_network_traffic_types\G
*************************** 1. row ***************************
                     id: 4                                                            
                   uuid: 41f2dfcf-824e-4a1f-93f3-d0acb3b4269a
    physical_network_id: 201                                                          
           traffic_type: Management
xenserver_network_label: NULL   
      kvm_network_label: cloudbr0
   vmware_network_label: NULL    
simulator_network_label: NULL
      ovm_network_label: NULL
                   vlan: NULL
      lxc_network_label: cloudbr0
   hyperv_network_label: NULL    
*************************** 2. row ***************************
                     id: 5
                   uuid: f3927334-3754-4eca-a325-b63b60450ec5
    physical_network_id: 201
           traffic_type: Guest
xenserver_network_label: NULL
      kvm_network_label: cloudbr0
   vmware_network_label: NULL
simulator_network_label: NULL
      ovm_network_label: NULL
                   vlan: NULL
      lxc_network_label: cloudbr0
   hyperv_network_label: NULL
*************************** 3. row ***************************
*************************** 3. row ***************************
                     id: 6
                   uuid: 61053c18-e666-427f-b8d8-e2a22acc8128
    physical_network_id: 201
           traffic_type: Public
xenserver_network_label: NULL
      kvm_network_label: cloudbr1
   vmware_network_label: NULL
simulator_network_label: NULL
      ovm_network_label: NULL
                   vlan: NULL
      lxc_network_label: cloudbr0
   hyperv_network_label: NULL
*************************** 4. row ***************************
                     id: 7
                   uuid: d5826cc1-d5c1-48f9-bc4f-b7fd1057223f
    physical_network_id: 201
           traffic_type: Storage
xenserver_network_label: NULL
      kvm_network_label: cloudbr0
   vmware_network_label: NULL
simulator_network_label: NULL
      ovm_network_label: NULL
                   vlan: NULL
      lxc_network_label: cloudbr0
   hyperv_network_label: NULL
4 rows in set (0,00 sec)

[DaanHoogland]

@tetra12 the combination

Local Storage
NO SSL/TLS

means you can't upload from local. At least it did mean that. I don't think this has been implemented. see https://github.com/apache/cloudstack/issues/4734#issuecomment-808771319

[GabrielBrascher]

@tetra12 can you please check the global settings values of the parameter management.network.cidr? It looks like it might be assigned to the wrong network, switching the bridges (cloudbr0 <> cloudbr1).

[tetra12]

Hello @GabrielBrascher According to your advice, I made the following adjustments:

Host setup

cloudbr0 - 185.185.X.Y cloudbr1 - 192.168.100.2

In the Physical Network Setup:

Public, Guest -> cloudbr0 Management, Storage -> cloudbr1

In the Global Settings:

management.network.cidr = 192.168.100.0/24

Still no luck. The REAL problem is that I'm getting an exception inside the SSVM since it connects to the wrong host.

Inside the SSVM /var/log/cloud.log I'm getting:

2021-08-30 20:51:27,940 INFO  [utils.nio.NioClient] (main:null) Connecting to 185.185.X.Y:8250
2021-08-30 20:51:31,002 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
        at java.base/sun.nio.ch.Net.connect0(Native Method)

And indeed curl -vv telnet://185.185.X.Y times out, BUT curl -vv telnet://192.168.100.2 work.

I found that SSVM gets its IP from the cmdline file:

root@s-1-VM:~# cat /var/cache/cloud/cmd_line.json
{
"type": "cmdline",
"cmd_line": {
        "template": "domP",
        "type": "secstorage",
        "host": "185.185.X.Y",  <<<----- WRONG ADDRESS from cloudbr0, Need 192.168.100.2 from cloudbr1
        "port": "8250",
        "name": "s-1-VM",
        "zone": "1",
        "pod": "1",
        "guid": "s-1-VM",
        "workers": "5",
        "resource": "org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource",
        "instance": "SecStorage",
        "sslcopy": "false",
        "role": "templateProcessor",
        "mtu": "1500",
        "eth2ip": "185.X.Y.116",
        "eth2mask": "255.255.255.192",
        "gateway": "185.X.Y.65",
        "public.network.device": "eth2",
        "eth0ip": "169.254.123.48",
        "eth0mask": "255.255.0.0",
        "eth1ip": "192.168.100.22",
        "eth1mask": "255.255.255.0",
        "mgmtcidr": "192.168.100.2/24",
        "localgw": "192.168.100.1",
        "private.network.device": "eth1",
        "internaldns1": "193.106.93.93",
        "internaldns2": "8.8.4.4",
        "dns1": "8.8.8.8",
        "dns2": "8.8.4.4",
        "nfsVersion": "null"
        }
}

@DaanHoogland SO indeed, host = 185.185.X.Y, which doesn't work for me. This address is taken from the getControlp() in libvirt, so that's the feature by design.

However, I need to write host address as 192.168.100.2 instead of 185.185.x.y to avoid the exception. How do I do that ? How do I update/change what gets written to the /var/cache/cloud/cmdline inside the SSVM ?

[tetra12]

Fixed after properly configuring SSVM

[unplugged216]

Fixed after properly configuring SSVM

How did you correct it?

[unplugged216]

Since no actual resolve or method to achieve it was provide by OP, here is the documentation used to resolve the issue.

https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM%2C+templates%2C+Secondary+storage+troubleshooting

The ultimate issue is a bit more extensive. Following the above alone did not solve my problem. I need to modify the following files to reflect proper IPs before the agent made communication with the MS.

/var/cache/cloud/cmd_line.json /var/cache/cloud/cmdline

Then restart the service. Whole vm restart not required. service cloud restart