creating system VMs is taking too long

[khoudali]

hi, i've followed your guide to deploy Cloudstack in a single server (All-in-one deployment) and everything went well, except during the "creating system VMs" is taking too much time ( I left it all night ) with no result, all steps went well except that one. If you could help me please i'm stuck at this point since a week ago. thank you in advance.

[thehyperadvisor]

I've seen this happen if there are not enough memory resources on the kvm hosts to create the vm instances. How much memory do you have for your kvm host?

Sent from mobile device.

On May 21, 2015, at 6:05 AM, khoudali notifications@github.com wrote:

hi, i've followed your guide to deploy Cloudstack in a single server (All-in-one deployment) and everything went well, except during the "creating system VMs" is taking too much time ( I left it all night ) with no result, all steps went well except that one. If you could help me please i'm stuck at this point since a week ago. thank you in advance.

— Reply to this email directly or view it on GitHub.

[khoudali]

i have a core i7 (Generation 2) with 16GB of RAM. I have an other issue, when I create NFS shares for primary and secondary storage (respectively /var/nfs/primary and /var/nfs/secondary), and I mount them respectively in /mnt/primary and /mnt/secondary, the Cloudstack UI refuse to use the mounted path of the primary storage (/mnt/primary) during the installation process and asks me to change that path, and when I change it to /var/nfs/primary the process continues and hangs on "creating system VMs". I'm new in Cloudstack and I don't know in with log file I have to look for debugging. P.S: when I look in the /mnt directory I found that cloudstack mounted /var/nfs/primary in an other directory which name is an uuid of something and creates in /mnt others folder with uuids as name. Thanks

[thehyperadvisor]

Can you verify that you can manually mount and access the nfs mount point on the management server and kvm hosts please. Also can you send me a copy of the question and answers you took so I can check?

Sent from mobile device.

On May 22, 2015, at 6:12 AM, khoudali notifications@github.com wrote:

i have a core i7 (Generation 2) with 16GB of RAM. I have an other issue, when I create NFS shares for primary and secondary storage (respectively /var/nfs/primary and /var/nfs/secondary), and I mount them respectively in /mnt/primary and /mnt/secondary, the Cloudstack UI refuse to use the mounted path of the primary storage (/mnt/primary) during the installation process and asks me to change that path, and when I change it to /var/nfs/primary the process continues and hangs on "creating system VMs". I'm new in Cloudstack and I don't know in with log file I have to look for debugging. P.S: when I look in the /mnt directory I found that cloudstack mounted /var/nfs/primary in an other directory which name is an uuid of something and creates in /mnt others folder with uuids as name. Thanks

— Reply to this email directly or view it on GitHub.

[khoudali]

hi, when i checked the management server log in /var/log/cloudstack/management, i found that i have this error: "ERROR [ c.c.v.VmWorkJobDispatcher ] {Work-Job-Executor-70:ctx-ba47d8d7 job-1/job-2245 ctx-e36a487f} Invocation Exception, caused by: : com.cloud.exception.InsufficientServerCapacityException Unable to create a deployment for [SecondaryStorageVm|s-449-VM]Scoop=interface com.cloud.dc.datacenter; id=1."

I have enough storage capacity for my NFS shares (above 700GB) where they are mounted. I've set the owner/ownerGroup to nfsnobody (I'm on Centos 6.6) and set permission to 755. I don't know from where this error comes .

[thehyperadvisor]

Can you send me what options your inputing to cldstk-deploy? This will help me understand your configuration. Are you trying to do a single server installation or multi server setup?

Also run ‘free -m’ and ‘df -h’.

I can only assume by the error that you do not have enough memory or storage to support the creation of the vm instance.

Antone Heyward @thehyperadvisor http://thehyperadvisor.com

On May 23, 2015, at 7:09 AM, khoudali notifications@github.com wrote:

com.cloud.exception.InsufficientServerCapacityException Unable to create a deployment for [SecondaryStorageVm|s-449-VM]Scoop=interface com.cloud.dc.datacenter; id=1."

[khoudali]

as I said earlier, I have a 16GB of RAM and over 600GB available space in / and the primary/secondary storages are mounted in /. The OS is freshly installed with almost 5GB of used space in /. I was trying to install a single All-in-one server with the basic inputs as you mentioned in your guide. Here some additionnal info: partitioning scheme: LVM logical volume 1: swap, 16GB logical volume 2: /home, 100GB logical volume 3: /, 720GB standard partition: /boot, 500MB Processor: intel core i7 2630QM Memory: 16GB Internal HDD: 1TB

cat /etc/exports:

     /primary       *(rw,async,no_root_squatsh,no_subtree_check)
     /secondary   *(rw,async,no_root_squatsh,no_subtree_check)

Mount shares permissions and owners info: Permissions: 755

Owner: nfsnobody

hostname -f

cldstkkvm.cloudpriv.edu

---

To restart the installation process I follow these steps: 1- uninstalling all cloudstack packages : rpm -qa | grep cloudstack | xargs yum -y erase 2- deleting everything related to cloudstack that the "locate" command returns 3-uninstalling mysql-server: yum erase mysql-server 4-deleting /var/lib/mysql 5-restoring network state by removing all bridges created by cloudstack and restoring eth0 to its initial config file 6-umounting all nfs shares and deleting their content and the content of /mnt 7-rebooting the system 8-restart cldstk-deploy I've tried to install cloudstack with nfs shares both mounted, both unmounted without any success. When the installer (cldstk-deploy) asks me about the nfs share path, I choose /secondary instead of the mount point (/mnt/secondary). I hope this can help you understand more. The attached image contains some of the cloudstack management log messages

[thehyperadvisor]

I just ran the process in my lab and it all worked fine. This has to be related to the NFS setup you have. I understand your sending the system spec but its more helpful to see what the OS sees. Can you run “df -h” then copy that to me just to verify space? You may also want to check the NFS settings. Below is what I use:

vi /etc/sysconfig/nfs -RQUOTAD_PORT=875 -LOCKD_TCPPORT=32803 -LOCKD_UDPPORT=32769 -MOUNTD_PORT=892

-STATD_PORT=662 -STATD_OUTGOING_PORT=2020

And per one of your previous email, you do not have to pre mount the NFS directories cloudstack will do it. Hopefully this helps but i don’t see this as an issue with cldstk-deploy or cloudstack.

Antone Heyward @thehyperadvisor http://thehyperadvisor.com

On May 23, 2015, at 10:41 AM, khoudali notifications@github.com wrote:

as I said earlier, I have a 16GB of RAM and over 600GB available space in / and the primary/secondary storages are mounted in /. The OS is freshly installed with almost 5GB of used space in /. I was trying to install a single All-in-one server with the basic inputs as you mentioned in your guide. Here some additionnal info: partitioning scheme: LVM logical volume 1: swap, 16GB logical volume 2: /home, 100GB logical volume 3: /, 720GB standard partition: /boot, 500MB Processor: intel core i7 2630QM Memory: 16GB Internal HDD: 1TB

cat /etc/exports:

 /primary       *(rw,async,no_root_squatsh,no_subtree_check)
 /secondary   *(rw,async,no_root_squatsh,no_subtree_check)

Mount shares permissions and owners info: Permissions: 755

Owner: nfsnobody

hostname -f

cldstkkvm.cloudpriv.edu To restart the installation process I follow these steps: 1- uninstalling all cloudstack packages : rpm -qa | grep cloudstack | xargs yum -y erase 2- deleting everything related to cloudstack that the "locate" command returns 3-uninstalling mysql-server: yum erase mysql-server 4-deleting /var/lib/mysql 5-restoring network state by removing all bridges created by cloudstack and restoring eth0 to its initial config file 6-umounting all nfs shares and deleting their content and the content of /mnt 7-rebooting the system 8-restart cldstk-deploy I've tried to install cloudstack with nfs shares both mounted, both unmounted without any success. When the installer (cldstk-deploy) asks me about the nfs share path, I choose /secondary instead of the mount point (/mnt/secondary). I hope this can help you understand more. The attached image contains some of the cloudstack management log messages https://cloud.githubusercontent.com/assets/12543166/7784543/64857868-0161-11e5-8af6-90e4d9e7a4ed.jpg — Reply to this email directly or view it on GitHub https://github.com/thehyperadvisor/cldstk-deploy/issues/5#issuecomment-104903856.

[khoudali]

I think too that this problem is related to the NFS setup even if I followed the apache cloudstack guide to configure the NFS server parameters (I already configured /etc/sysconfig/nfs as you just said, added iptables rules to allow nfs connections). I'm not at home to send you what you asked for but I will in the evening. Meanwhile can you send what are the NFS configuration and parameters (client and server side) that cloudstack needs to get working, and if it's necessary or not to have nfs shares mounted before starting the install process.

[khoudali]

I have finally resolved my problem, I've simply restarted libvirtd service and the setup finished. I don't know the reason why but it worked just fine. thank you for your help.

[thehyperadvisor]

Glad to hear you got it working.

Sent from mobile device

On May 26, 2015, at 7:39 AM, khoudali notifications@github.com wrote:

I have finally resolved my problem, I've simply restarted libvirtd service and the setup finished. I don't know the reason why but it worked just fine. thank you for your help.

— Reply to this email directly or view it on GitHub.

[thehyperadvisor]

Issue was resolved by simply restarted libvirtd service.