DaanHoogland
commented
1 year ago @bkrajendra , can you mount the storages on your hypervisor?
Closed bkrajendra closed 1 year ago
DaanHoogland
commented
1 year ago @bkrajendra , can you mount the storages on your hypervisor?
bkrajendra
commented
1 year ago @DaanHoogland Yes, as I said I created a test mount on the hypervisor host and I am able to do it. Even primary storage is visible in the storage pool, but not sure why secondary storage is not coming on the storage pool list.
bkrajendra
commented
1 year ago secondary storage shows in the mount on the management server.
10.1.27.14:/store0/export/secondary on /var/lib/cloudstack/mnt/95534533553.4e443153 type nfs4 (rw,relatime,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.1.27.14,local_lock=none,addr=10.1.27.14)my nfs server is on the management server.
still I am getting InsufficientServerCapacityException: Unable to create a deployment for VM instance
Its destroying and creating VMs... currently showing [v-2221-VM]
I have tried with true and false values to Enable local storage for user VMs
What's wrong here?
bkrajendra
commented
1 year ago list capacity shows:
🐱 > list capacity
{
"capacitytotal": 0,
"capacityused": 0,
"name": "SECONDARY_STORAGE",
"percentused": "0",
"type": 6,
"zoneid": "9c1bc99b-a6f4-45d2-9b7e-029b260e042f",
"zonename": "Pune"
}
so you can mount the export on the local machine i.e. the management server, and on the hypervisor host. but the secondary storage VM does not come up. the VM with a name like s-###-VM (not v-###-VM). I suppose you did but just for completeness, did you check your network configuration, any routers between the two machines. Can you show the export line for the NFS share? You did seed the secondary storage with the built-in templates and they ended up on the secstor?
bkrajendra
commented
1 year ago @DaanHoogland There is a Router between two machines. Management server+NFS is on a different host inside VM. but in the same subnet. Export line of nfs share:
/store0/export/secondary *(rw,async,no_root_squash,no_subtree_check)
/store0/export/primary *(rw,async,no_root_squash,no_subtree_check)I used UI to download the template. I registered it through UI and it got downloaded after I create secondary storage. I can see the template folder created inside the secondary store export.
Issue is SystemVMs are not getting started. It comes on UI, shows starting and then gets removed, and again new gets created. Getting error continuously on console ```InsufficientServerCapacityException````
DaanHoogland
commented
1 year ago @bkrajendra , unfortunately InsufficientServerCapacityException is a very generic error.
Are there any more log messages for job-2796or ctx-b66fed52 before what you shared? I can not tell the issue from what i see. But i'm sure it is some environmental thing, we run dozens of automated installs of main and 4.17 for test puroses.
bkrajendra
commented
1 year ago @DaanHoogland I deleted systemvm template and tried to seed them manually on the management server with:
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt -m /store0/export/secondary -u http://download.cloudstack.org/systemvm/4.17/systemvmtemplate-4.17.0-kvm.qcow2.bz2 -h kvm -F While doing this I discovered there is no space left in the management server / (root). hence it was not able to download it.
For a workaround, I created a soft link on another drive (/store0 = 200GB) and worked. Now my SystemVMs are up. Its secondary storage VM shows in an alert state. Still, secondary storage shows 0/0. Could this be due to the Management Server root drive being full?
bkrajendra
commented
1 year ago find the diagnostic logs on systemVMs: console proxy VM
root@v-2995-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
================================================
First DNS server is 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=118 time=7.841 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=5.882 ms
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 5.882/6.861/7.841/0.980 ms
Good: Can ping DNS server
================================================
Good: DNS resolves cloudstack.apache.org
================================================
ERROR: NFS is not currently mounted
Try manually mounting from inside the VM
NFS server is 10.1.27.235
PING 10.1.27.235 (10.1.27.235): 56 data bytes
64 bytes from 10.1.27.235: icmp_seq=0 ttl=64 time=0.045 ms
64 bytes from 10.1.27.235: icmp_seq=1 ttl=64 time=0.026 ms
--- 10.1.27.235 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.026/0.035/0.045/0.000 ms
Good: Can ping nfs server
================================================
Management server is 10.1.27.14. Checking connectivity.
Good: Can connect to management server 10.1.27.14 port 8250
================================================
Good: Java process is running
================================================
Tests Complete. Look for ERROR or WARNING above.Secondary storage VM
root@s-2994-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
================================================
First DNS server is 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=118 time=9.404 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=5.752 ms
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 5.752/7.578/9.404/1.826 ms
Good: Can ping DNS server
================================================
Good: DNS resolves cloudstack.apache.org
================================================
nfs is currently mounted
Mount point is /mnt/SecStorage/660f6cc7-82c6-336f-91f9-b2ce91928a22
Good: Can write to mount point
================================================
Management server is 10.1.27.14. Checking connectivity.
Good: Can connect to management server 10.1.27.14 port 8250
================================================
Good: Java process is running
================================================
Tests Complete. Look for ERROR or WARNING above.restarting the secondary storage VM removed the alert and now I can see the secondary storage properly represented in Dashboard. I am testing ISO registration/download. It's working fine and the download has started. I hope eventually I'll be able to start my VM instance today.
This is the 4th time I am deploying CloudStack. Every time I learned new things and faced different issues. Thank you, team.
DaanHoogland
commented
1 year ago @bkrajendra I'm closing this. please reopen or create a new issue if you feel the need
ISSUE TYPE
COMPONENT NAME
CLOUDSTACK VERSION
CONFIGURATION
OS / ENVIRONMENT
SUMMARY
System VM are not able to start. I have added NFS as secondary storage. I have also added NFS on primary and its working fine But secondary storage is not detected on the dashboard. NFS is set up on the management server. I have tested the NFS mount manually on the host and it works fine.
STEPS TO REPRODUCE
EXPECTED RESULTS
ACTUAL RESULTS
Logs - management server:
Agent Logs:
Storage pools: