OC deployment fails as controller has only 4GB RAM by default

[fhubik]

Baremetal machine with 64GB RAM, Centos 7, tripleo-quickstart ([minimal] virthost) with defaults, following guide https://github.com/redhat-openstack/tripleo-quickstart: ... $ undercloud-install.sh $ undercloud-post-install.sh $ overcloud-deploy.sh # FAILED ... Mar 10 15:45:04 overcloud-controller-0 os-collect-config[1951]: No hosts in Heat, nothing written. Mar 10 15:45:04 overcloud-controller-0 os-collect-config[1951]: dib-run-parts Thu Mar 10 15:45:04 UTC 2016 51-hosts completed Mar 10 15:45:04 overcloud-controller-0 os-collect-config[1951]: dib-run-parts Thu Mar 10 15:45:04 UTC 2016 Running /usr/libexec/os-refresh-config/configure.d/55-heat-config Mar 10 15:45:04 overcloud-controller-0 os-collect-config[1951]: dib-run-parts Thu Mar 10 15:45:04 UTC 2016 55-heat-config completed Mar 10 15:45:04 overcloud-controller-0 os-collect-config[1951]: ----------------------- PROFILING ----------------------- Mar 10 15:45:04 overcloud-controller-0 os-collect-config[1951]: Target: configure.d Mar 10 15:45:04 overcloud-controller-0 os-collect-config[1951]: Script Seconds Mar 10 15:45:04 overcloud-controller-0 os-collect-config[1951]: --------------------------------------- ---------- Mar 10 15:45:04 overcloud-controller-0 os-collect-config[1951]: 10-sysctl-apply-config 0.209 Mar 10 15:45:04 overcloud-controller-0 os-collect-config[1951]: 20-os-apply-config 0.183 ...skipping... 2016-03-10 15:55:32 [0]: CREATE_IN_PROGRESS state changed [0/9923] 2016-03-10 15:55:32 [0]: SIGNAL_COMPLETE Unknown 2016-03-10 15:55:33 [ControllerDeployment]: SIGNAL_COMPLETE Unknown 2016-03-10 15:55:33 [0]: SIGNAL_COMPLETE Unknown 2016-03-10 15:55:33 [ControllerDeployment]: SIGNAL_COMPLETE Unknown 2016-03-10 15:55:34 [0]: SIGNAL_COMPLETE Unknown 2016-03-10 15:55:34 [NetworkDeployment]: SIGNAL_COMPLETE Unknown 2016-03-10 15:55:35 [0]: SIGNAL_COMPLETE Unknown 2016-03-10 15:56:49 [0]: SIGNAL_IN_PROGRESS Signal: deployment failed (6) 2016-03-10 15:56:49 [0]: CREATE_FAILED Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 2016-03-10 15:56:50 [overcloud-ControllerNodesPostDeployment-3mu4j4fs7ybn-ControllerOvercloudServicesDeployment_Step6-jruwoxkxdm32]: CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deploym ent to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 2016-03-10 15:56:51 [0]: SIGNAL_COMPLETE Unknown 2016-03-10 15:56:51 [ControllerOvercloudServicesDeployment_Step6]: CREATE_FAILED Error: resources.ControllerOvercloudServicesDeployment_Step6.resources[0]: Deployment to server failed: deploystatus code: Deployment exited with non-zero status code: 6 2016-03-10 15:56:51 [0]: SIGNAL_COMPLETE Unknown 2016-03-10 15:56:52 [ControllerNodesPostDeployment]: CREATE_FAILED Error: resources.ControllerNodesPostDeployment.resources.ControllerOvercloudServicesDeployment_Step6.resources[0]: Deployment to ser ver failed: deploy_status_code: Deployment exited with non-zero status code: 6 2016-03-10 15:56:52 [overcloud-ControllerNodesPostDeployment-3mu4j4fs7ybn]: CREATE_FAILED Resource CREATE failed: Error: resources.ControllerOvercloudServicesDeployment_Step6.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6 2016-03-10 15:56:52 [0]: SIGNAL_COMPLETE Unknown 2016-03-10 15:56:52 [0]: SIGNAL_COMPLETE Unknown 2016-03-10 15:56:53 [overcloud]: CREATE_FAILED Resource CREATE failed: Error: resources.ControllerNodesPostDeployment.resources.ControllerOvercloudServicesDeployment_Step6.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6 2016-03-10 15:56:53 [0]: SIGNAL_COMPLETE Unknown 2016-03-10 15:56:53 [ControllerDeployment]: SIGNAL_COMPLETE Unknown 2016-03-10 15:56:54 [NetworkDeployment]: SIGNAL_COMPLETE Unknown 2016-03-10 15:56:54 [0]: SIGNAL_COMPLETE Unknown Stack overcloud CREATE_FAILED Deployment failed: Heat Stack create failed.

$ heat resource-list overcloud | grep -i failed | ControllerNodesPostDeployment | a08b472c-ffe0-40c7-a029-073a4b5df79e | OS::TripleO::ControllerPostDeployment | CREATE_FAILED | 2016-03-10T15:41:24 | heat resource-show overcloud ControllerNodesPostDeployment +------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | attributes | {} | | creation_time | 2016-03-10T15:41:24 | | description | | | links | http://192.0.2.1:8004/v1/b592cbcada2946c8881feecd8b9b30a5/stacks/overcloud/51bb82b1-07d7-4222-8739-fec5effaafce/resources/ControllerNodesPostDeployment (self) | | | http://192.0.2.1:8004/v1/b592cbcada2946c8881feecd8b9b30a5/stacks/overcloud/51bb82b1-07d7-4222-8739-fec5effaafce (stack) | | | http://192.0.2.1:8004/v1/b592cbcada2946c8881feecd8b9b30a5/stacks/overcloud-ControllerNodesPostDeployment-3mu4j4fs7ybn/a08b472c-ffe0-40c7-a029-073a4b5df79e (nested) | | logical_resource_id | ControllerNodesPostDeployment | | physical_resource_id | a08b472c-ffe0-40c7-a029-073a4b5df79e | | required_by | BlockStorageNodesPostDeployment | | | CephStorageNodesPostDeployment | | resource_name | ControllerNodesPostDeployment | | resource_status | CREATE_FAILED | | resource_status_reason | Error: resources.ControllerNodesPostDeployment.resources.ControllerOvercloudServicesDeployment_Step6.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6 | | resource_type | OS::TripleO::ControllerPostDeployment | | updated_time | 2016-03-10T15:41:24 | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

$ ssh heat-admin@ $ sudo journalctl -u os-collect-config # and near the end: Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Warning: Scope(Class[Ceilometer::Api]): The keystone_identity_uri parameter is deprecated. Please use identity_uri instead. Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Error: /Stage[main]/Apache::Service/Service[httpd]: Could not evaluate: Cannot allocate memory - fork(2) Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Warning: /Stage[main]/Keystone/Anchor[keystone_started]: Skipping because of failed dependencies Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Error: Could not prefetch keystone_tenant provider 'openstack': Cannot allocate memory - fork(2) Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Warning: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]: Skipping because of failed dependencies Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Warning: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]: Skipping because of failed dependencies Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Error: Could not prefetch keystone_role provider 'openstack': Cannot allocate memory - fork(2) Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Warning: /Stage[main]/Keystone::Roles::Admin/Keystone_role[admin]: Skipping because of failed dependencies Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Error: Could not prefetch keystone_user provider 'openstack': Cannot allocate memory - fork(2) Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Warning: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Skipping because of failed dependencies Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Warning: /Stage[main]/Keystone::Roles::Admin/Keystone_user_role[admin@admin]: Skipping because of failed dependencies Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Error: Could not prefetch keystone_service provider 'openstack': Cannot allocate memory - fork(2) Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Warning: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]: Skipping Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Error: Could not prefetch keystone_endpoint provider 'openstack': Cannot allocate memory - fork(2) Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Warning: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_endpoint[regionOne/keystone::identity] Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Error: /Stage[main]/Heat::Api_cfn/Service[heat-api-cfn]: Could not evaluate: Cannot allocate memory - fork(2) Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Error: /Stage[main]/Heat::Engine/Service[heat-engine]: Could not evaluate: Cannot allocate memory - fork(2) Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Error: /Stage[main]/Heat::Api/Service[heat-api]: Could not evaluate: Cannot allocate memory - fork(2) Mar 10 15:56:48 overcloud-controller-0 os-collect-config[1951]: Error: /Stage[main]/Heat::Api_cloudwatch/Service[heat-api-cloudwatch]: Could not evaluate: Cannot allocate memory - fork(2)

It looks like there is not enough memory for minimal virthost OC controller set up.

[trown]

Proposed fix:

https://review.gerrithub.io/265810

[fhubik]

I was able to deploy overcloud successfully (rdo-m default minimal install) with "-e controll_memory=8192" added to quickstart script as a ansible extra variable. Command now look like:

$ cat quickstart.sh ... command=`ansible-playbook -$VERBOSITY $OPT_WORKDIR/tripleo-quickstart/playbooks/quickstart.yml \ -e url=$OPT_UNDERCLOUD_URL \ -e local_working_dir=$OPT_WORKDIR \ -e virthost=$VIRTHOST \ -t $OPT_TAGS \ -e control_memory=8192

[trown]

This is solved via https://github.com/redhat-openstack/tripleo-quickstart/commit/2f0c16f2aeabcef70dfb12e280c1766485d8b08d