loss of conectivity after neutron deployment

Hi Im trying to setup an openstack envrionement using puppet. I tried both allinone and multinode deployments. The debug files point to loss of conectivity on neutron deployment, specifically upon the creation of brex bridge. If i delete this bridge, i restore connectivity. I only lose connectivity on the external network (which has the internet, and therefore, the install process of horizon and heat do not go through). This is my allinonde.yaml http://paste.openstack.org/show/204899/

this is the route -n of a failed deployment machine(i manually altered the default gateway prior to deployment) 0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 p3p1 10.155.216.0 0.0.0.0 255.255.252.0 U 0 0 0 em1 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 p3p1 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0

this is from other machine (unaltered) Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.155.216.1 0.0.0.0 UG 0 0 0 eth0 10.155.216.0 0.0.0.0 255.255.252.0 U 0 0 0 eth0 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0

EDIT-i forgot to mention, my puppet is master is communicating on 10.155.216.0

Can you clarify what is losing connectivity? Is it a nova instance, or the network node itself? What operating system are you running on?

Im running ubuntu 14.04. The network node itself will lose the connection on the public interface which is connected to the internet and will fail the deployment. Also It can not contact puppet if puppet is on the public side. Puppet has to be on the management side. BTW, I've been reinstalling nearly everyday and since friday you guys broke the openstack module. Installing it on a fresh install (and i tried 4 times) will break the console classification module every single time after puppet installs the openstack module.

THanks

On Mon, Apr 27, 2015 at 11:08 AM, Colleen Murphy notifications@github.com wrote:

Can you clarify what is losing connectivity? Is it a nova instance, or the network node itself? What operating system are you running on?

— Reply to this email directly or view it on GitHub https://github.com/puppetlabs/puppetlabs-openstack/issues/176#issuecomment-96762684 .

I'm sorry you're experiencing this issue, I'm sure it's very frustrating.

I've been reinstalling nearly everyday and since friday you guys broke the openstack module.

We haven't made a forge release in a while so I assume you must be installing the openstack module from github. Could you help us debug this issue by trying to deploy from earlier commits, so that we can try to pinpoint the change that broke the module?

Installing it on a fresh install (and i tried 4 times) will break the console classification module every single time after puppet installs the openstack module.

What symptoms are you experiencing with the console classification module? This is on Puppet Enterprise, correct?

Im using the default command to install the openstack module. >sudo puppet module install puppetlabs-openstack. Isnt this the correct one?

Yes, its on puppet enterprise. I can commit nodes, install ntp (as per example on your documentation). And before friday I was able to install the openstack module and deploy it on the nodes. After friday, as soon as I installed the module, the classification tab presented me "an error occured check the logs". As i said before, this is always on fresh installs, installed the same way, without any OS updates, with the same network configuration and everytihing.

attached image -> http://imgur.com/HQexB6i the logs give something about java: root@dopey:/var/log/pe-console-services# cat console-services.log 2015-04-27 20:05:37,370 INFO [p.c.class-updater] Requesting environment list from "https://dopey:8140/v2.0/environments" 2015-04-27 20:05:37,588 INFO [p.c.class-updater] 200 response received for request for environments from "https://dopey:8140/v2.0/environments" 2015-04-27 20:05:37,588 INFO [p.c.class-updater] Requesting classes in production from "https://dopey.fog.poc:8140/production/resource_types/" 2015-04-27 20:05:39,820 INFO [p.c.class-updater] 200 response received for request for classes in production from " https://dopey:8140/production/resource_types/". 2015-04-27 20:05:39,888 INFO [p.c.class-updater] Synchronized 104 classes from the Puppet Master in 2 seconds 2015-04-27 20:08:24,835 WARN [p.r.h.middleware] GET /node_groups/classifier-api/v1/environments java.net.ConnectException: Connection refused (Unknown Source) sun.nio.ch.SocketChannelImpl.checkConnect SocketChannelImpl.java:739 sun.nio.ch.SocketChannelImpl.finishConnect DefaultConnectingIOReactor.java:173 org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent DefaultConnectingIOReactor.java:147 org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents AbstractMultiworkerIOReactor.java:348 org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute PoolingNHttpClientConnectionManager.java:189 org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute CloseableHttpAsyncClientBase.java:67 org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.doExecute CloseableHttpAsyncClientBase.java:38 org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.access$000 CloseableHttpAsyncClientBase.java:57 org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run Thread.java:745 java.lang.Thread.run

2015-04-27 20:08:24,844 WARN [p.r.h.middleware] GET /node_groups/classifier-api/v1/groups java.net.ConnectException: Connection refused (Unknown Source) sun.nio.ch.SocketChannelImpl.checkConnect SocketChannelImpl.java:739 sun.nio.ch.SocketChannelImpl.finishConnect DefaultConnectingIOReactor.java:173 org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent DefaultConnectingIOReactor.java:147 org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents AbstractMultiworkerIOReactor.java:348 org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute PoolingNHttpClientConnectionManager.java:189 org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute CloseableHttpAsyncClientBase.java:67 org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.doExecute CloseableHttpAsyncClientBase.java:38 org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.access$000 CloseableHttpAsyncClientBase.java:57 org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run Thread.java:745 java.lang.Thread.run

2015-04-27 20:08:25,318 WARN [p.r.h.middleware] GET /node_groups/classifier-api/v1/classes java.net.ConnectException: Connection refused (Unknown Source) sun.nio.ch.SocketChannelImpl.checkConnect SocketChannelImpl.java:739 sun.nio.ch.SocketChannelImpl.finishConnect DefaultConnectingIOReactor.java:173 org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent DefaultConnectingIOReactor.java:147 org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents AbstractMultiworkerIOReactor.java:348 org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute PoolingNHttpClientConnectionManager.java:189 org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute CloseableHttpAsyncClientBase.java:67 org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.doExecute CloseableHttpAsyncClientBase.java:38 org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.access$000 CloseableHttpAsyncClientBase.java:57 org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run Thread.java:745 java.lang.Thread.run

On Mon, Apr 27, 2015 at 3:48 PM, Colleen Murphy notifications@github.com wrote:

I'm sorry you're experiencing this issue, I'm sure it's very frustrating.

I've been reinstalling nearly everyday and since friday you guys broke the openstack module.

We haven't made a forge release in a while so I assume you must be installing the openstack module from github. Could you help us debug this issue by trying to deploy from earlier commits, so that we can try to pinpoint the change that broke the module?

Installing it on a fresh install (and i tried 4 times) will break the console classification module every single time after puppet installs the openstack module.

What symptoms are you experiencing with the console classification module? This is on Puppet Enterprise, correct?

— Reply to this email directly or view it on GitHub https://github.com/puppetlabs/puppetlabs-openstack/issues/176#issuecomment-96842763 .

We have not made a release to this module for a while, so there is no way that this could have suddenly caused your environment to break on Friday. Simply installing the module with puppet module install puppetlabs-openstack does not make any configuration changes, so it cannot have caused your network outage. Please open a support ticket for your Puppet Enterprise installation.

the network outage happened even before friday. I just casually mentioned friday. And yes, you guys broke it. Because i did not changed anything on my environment and as i stated previously, everytime I installed the puppet enterprise and its nodes, the were always on fresh AKA FORMATTED installs.

On Tue, Apr 28, 2015 at 9:18 AM, Colleen Murphy notifications@github.com wrote:

Closed #176 https://github.com/puppetlabs/puppetlabs-openstack/issues/176.

— Reply to this email directly or view it on GitHub https://github.com/puppetlabs/puppetlabs-openstack/issues/176#event-292153191 .

Are you applying the allinone role or the network role to the puppetmaster itself?

I tried both. Same problem

On Tue, Apr 28, 2015 at 10:04 AM, Colleen Murphy notifications@github.com wrote:

Are you applying the allinone role or the network role to the puppetmaster itself?

— Reply to this email directly or view it on GitHub https://github.com/puppetlabs/puppetlabs-openstack/issues/176#issuecomment-97138493 .

You should not apply any openstack configuration to the puppetmaster. You should only make sure ntp is running on the puppetmaster. Please try again using different nodes for your openstack deployment.

Sorry, my bad, i didn not read your comment properly. I have 6machines, one is puppet master and Im NOT deploying anything there.

On Tue, Apr 28, 2015 at 10:30 AM, Colleen Murphy notifications@github.com wrote:

You should not apply any openstack configuration to the puppetmaster. You should only make sure ntp is running on the puppetmaster. Please try again using different nodes for your openstack deployment.

— Reply to this email directly or view it on GitHub https://github.com/puppetlabs/puppetlabs-openstack/issues/176#issuecomment-97144372 .

Okay. With regard to the issue with your puppet console, I've spoken to one of our support specialists, who recommends following these instructions to tune your puppetmaster by reducing the number of JRuby processes and resetting the memory allocated to the console. Our PE engineers recommend checking /etc/puppetlabs/console-services/conf.d/console.conf and /etc/puppetlabs/console-services/conf.d/webserver.conf on the puppet master for anything that has changed. Specifically, they think that setting an incorrect hostname for classifier-server in the console.conf file or an incorrect hostname for the api server in webserver.conf may cause the error you're seeing. If you are unable to resolve the issue with the puppet console and you have a standard support license, I recommend opening a support ticket. The support team and I are confident that installing a module but applying no additional configuration from the module to the puppet master cannot cause this issue, so it appears to be a coincidence.

With regard to the networking issue on the network node, we can still try to debug that here. Can you provide the following information:

You said "i manually altered the default gateway prior to deployment", can you verify that you are able to reach the external network after manually altering the gateway but before applying any puppet configuration?
the output of the following commands on the network node both before and after applying the puppet configuration:
- iptables -S
- ifconfig
- ovs-vsctl
the output of puppet agent -t for applying configuration to the unconfigured network node (make sure the controller is configured first)
the output of puppet module list from the puppet master
Is there any additional configuration being applied to the network node?

Hi I tried to alter the hostname to specific IP address and the problem still persists. Im going to make a format and a full install again just for you. Do you believe a BIND server might be interfering with the puppet master? Might be the only thing i added along side it last week.

Regarding those outputs, I will give them to you if I'm able to deploy again.

BTW nothing correlated with my problems, but I thought you would like to know that everything you perform a reboot on the master puppet, the web interface refuses connections and I have to issue the following command: sudo service pe-puppetserver restart; sudo service pe-console-services restart

On Tue, Apr 28, 2015 at 3:25 PM, Colleen Murphy notifications@github.com wrote:

Okay. With regard to the issue with your puppet console, I've spoken to one of our support specialists, who recommends following these instructions http://docs.puppetlabs.com/pe/3.7/config_puppetserver.html to tune your puppetmaster by reducing the number of JRuby processes and resetting the memory allocated to the console. Our PE engineers recommend checking /etc/puppetlabs/console-services/conf.d/console.conf and /etc/puppetlabs/console-services/conf.d/webserver.conf on the puppet master for anything that has changed. Specifically, they think that setting an incorrect hostname for classifier-server in the console.conf file or an incorrect hostname for the api server in webserver.conf may cause the error you're seeing. If you are unable to resolve the issue with the puppet console and you have a standard support license, I recommend opening a support ticket. The support team and I are confident that installing a module but applying no addit ional configuration from the module to the puppet master cannot cause this issue, so it appears to be a coincidence.

With regard to the networking issue on the network node, we can still try to debug that here. Can you provide the following information:

-

You said "i manually altered the default gateway prior to deployment", can you verify that you are able to reach the external network after manually altering the gateway but before applying any puppet configuration?

the output of the following commands on the network node both before and after applying the puppet configuration:

iptables -S

ifconfig

ovs-vsctl

the output of puppet agent -t for applying configuration to the unconfigured network node (make sure the controller is configured first)

the output of puppet module list from the puppet master

Is there any additional configuration being applied to the network node?

— Reply to this email directly or view it on GitHub https://github.com/puppetlabs/puppetlabs-openstack/issues/176#issuecomment-97244550 .

A bad DNS configuration in bind could cause issues with your puppetmaster. I think opening a support ticket would be the most effective way to resolve the issue with your puppetmaster.

DNS is working good. Still i manually bypassed the DNS with /etc/hosts file and still gave me problems

On Tue, Apr 28, 2015 at 4:19 PM, Colleen Murphy notifications@github.com wrote:

A bad DNS configuration in bind could cause issues with your puppetmaster. I think opening a support ticket would be the most effective way to resolve the issue with your puppetmaster.

— Reply to this email directly or view it on GitHub https://github.com/puppetlabs/puppetlabs-openstack/issues/176#issuecomment-97261037 .

New install, new server, still same error.

On Tue, Apr 28, 2015 at 4:21 PM, Pedro Coelho Silv wrote:

DNS is working good. Still i manually bypassed the DNS with /etc/hosts file and still gave me problems

On Tue, Apr 28, 2015 at 4:19 PM, Colleen Murphy notifications@github.com wrote:

A bad DNS configuration in bind could cause issues with your puppetmaster. I think opening a support ticket would be the most effective way to resolve the issue with your puppetmaster.

— Reply to this email directly or view it on GitHub https://github.com/puppetlabs/puppetlabs-openstack/issues/176#issuecomment-97261037 .

Hi, I have also a lost connection with my configuration when I try to install a network node. Like mentionned above ovs-vsctl del-br brex resolve the connection lost problem. Clear iptables doesn't change anything. I can't ping anything external or internal. My node IP : 10.0.249.149 You can see my error log of puppet agent -t here :

Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install neutron-fwaas' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package neutron-fwaas
Error: /Stage[main]/Neutron::Services::Fwaas/Package[neutron-fwaas]/ensure: change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install neutron-fwaas' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package neutron-fwaas
Warning: /Stage[main]/Neutron::Services::Fwaas/Neutron_fwaas_service_config[fwaas/driver]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Services::Fwaas/Neutron_fwaas_service_config[fwaas/enabled]: Skipping because of failed dependencies
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install neutron-plugin-openvswitch-agent' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  neutron-plugin-openvswitch-agent
0 upgraded, 1 newly installed, 0 to remove and 84 not upgraded.
Need to get 3872 B of archives.
After this operation, 76.8 kB of additional disk space will be used.
Err http://ubuntu-cloud.archive.canonical.com/ubuntu/ trusty-updates/juno/main neutron-plugin-openvswitch-agent all 1:2014.2.3-0ubuntu1~cloud0
  Could not resolve 'ubuntu-cloud.archive.canonical.com'
E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/n/neutron/neutron-plugin-openvswitch-agent_2014.2.3-0ubuntu1~cloud0_all.deb  Could not resolve 'ubuntu-cloud.archive.canonical.com'

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Error: /Stage[main]/Neutron::Agents::Ml2::Ovs/Package[neutron-ovs-agent]/ensure: change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install neutron-plugin-openvswitch-agent' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  neutron-plugin-openvswitch-agent
0 upgraded, 1 newly installed, 0 to remove and 84 not upgraded.
Need to get 3872 B of archives.
After this operation, 76.8 kB of additional disk space will be used.
Err http://ubuntu-cloud.archive.canonical.com/ubuntu/ trusty-updates/juno/main neutron-plugin-openvswitch-agent all 1:2014.2.3-0ubuntu1~cloud0
  Could not resolve 'ubuntu-cloud.archive.canonical.com'
E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/n/neutron/neutron-plugin-openvswitch-agent_2014.2.3-0ubuntu1~cloud0_all.deb  Could not resolve 'ubuntu-cloud.archive.canonical.com'

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/local_ip]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/enable_distributed_routing]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/tunnel_bridge]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/enable_tunneling]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/integration_bridge]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[securitygroup/firewall_driver]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/arp_responder]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/polling_interval]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/l2_population]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/tunnel_types]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Service[neutron-ovs-agent-service]: Skipping because of failed dependencies
Error: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]: Failed to call refresh: keystone-manage db_sync returned 1 instead of one of [0]
Error: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]: keystone-manage db_sync returned 1 instead of one of [0]
Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_service[neutron]: Could not evaluate: undefined method `collect' for nil:NilClass
Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: undefined method `collect' for nil:NilClass
Warning: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_endpoint[openstack/neutron]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user_role[neutron@services]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Server/Service[neutron-server]: Skipping because of failed dependencies
Error: Could not send report: No route to host - connect(2)

Do you want a full trace ? My config file :

---
classes:
# - openstack::role::controller
# - openstack::role::storage
 - openstack::role::network
# - openstack::role::compute
# - openstack::role::allinone
 - openstack

openstack::region: 'openstack'

######## Networks
openstack::network::api: '10.0.249.0/16'
openstack::network::external: '10.0.249.0/16'

openstack::network::management: '10.0.249.0/16'
openstack::network::data: '10.0.249.0/16'

######## Fixed IPs (controllers)

openstack::controller::address::api: '10.0.249.119'
openstack::controller::address::management: '10.0.249.119'
openstack::storage::address::api: '10.0.1.222'
openstack::storage::address::management: '10.0.1.222'

######## Database

openstack::mysql::root_password: 'spam-gak'
openstack::mysql::service_password: 'fuva-wax'
openstack::mysql::allowed_hosts: ['localhost', '127.0.0.1', '10.0.249.%', '10.0.1.%']

openstack::mysql::keystone::user: 'keystone'
openstack::mysql::keystone::pass: 'fuva-wax'

openstack::mysql::cinder::user: 'cinder'
openstack::mysql::cinder::pass: 'fuva-wax'

openstack::mysql::glance::user: 'glance'
openstack::mysql::glance::pass: 'fuva-wax'
openstack::glance::api_servers: ['10.0.1.222:9292']

openstack::mysql::nova::user: 'nova'
openstack::mysql::nova::pass: 'fuva-wax'

openstack::mysql::neutron::user: 'neutron'
openstack::mysql::neutron::pass: 'fuva-wax'

openstack::mysql::heat::user: 'heat'
openstack::mysql::heat::pass: 'fuva-wax'

######## RabbitMQ

openstack::rabbitmq::user: 'openstack'
openstack::rabbitmq::password: 'pose-vix'
openstack::rabbitmq::hosts: ['10.0.249.119:5672']

######## Keystone

openstack::keystone::admin_token: 'sosp-kyl'
openstack::keystone::admin_email: 'chris.hoge@puppetlabs.com'
openstack::keystone::admin_password: 'fyby-tet'

openstack::keystone::tenants:
    "test":
        description: "Test tenant"
    "test2":
        description: "Test tenant"

openstack::keystone::users:
    "test":
        password: "abc123"
        tenant: "test"
        email: "test@example.com"
        admin: true
    "demo":
        password: "abc123"
        tenant: "test"
        email: "demo@example.com"
        admin: false
    "demo2":
        password: "abc123"
        tenant: "test2"
        email: "demo2@example.com"
        admin: false

######## Glance

openstack::images:
  Cirros:
    container_format: 'bare'
    disk_format: 'qcow2'
    source: 'http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img'

openstack::glance::password: 'na-mu-va'

######## Cinder

openstack::cinder::password: 'zi-co-se'
openstack::cinder::volume_size: '8G'

######## Swift

openstack::swift::password: 'dexc-flo'
openstack::swift::hash_suffix: 'pop-bang'

######## Nova

openstack::nova::libvirt_type: 'kvm'
openstack::nova::password: 'quuk-paj'

######## Neutron

openstack::neutron::password: 'whi-rtuz'
openstack::neutron::shared_secret: 'by-sa-bo'
openstack::neutron::core_plugin: 'ml2'
openstack::neutron::service_plugins: ['router', 'firewall', 'lbaas', 'vpnaas', 'metering']

######## Ceilometer
openstack::ceilometer::address::management: '10.0.249.119'
openstack::ceilometer::mongo::username: 'mongo'
openstack::ceilometer::mongo::password: 'mongosecretkey123'
openstack::ceilometer::password: 'whi-truz'
openstack::ceilometer::meteringsecret: 'ceilometersecretkey'

######## Heat
openstack::heat::password: 'zap-bang'
openstack::heat::encryption_key: 'heatsecretkey123'

######## Horizon

openstack::horizon::secret_key: 'whu-ghuk'
openstack::horizon::allowed_hosts: ['localhost', '127.0.0.1', '*']

######## Tempest

openstack::tempest::configure_images    : true
openstack::tempest::image_name          : 'Cirros'
openstack::tempest::image_name_alt      : 'Cirros'
openstack::tempest::username            : 'demo'
openstack::tempest::username_alt        : 'demo2'
openstack::tempest::username_admin      : 'test'
openstack::tempest::configure_network   : true
openstack::tempest::public_network_name : 'public'
openstack::tempest::cinder_available    : true
openstack::tempest::glance_available    : true
openstack::tempest::horizon_available   : true
openstack::tempest::nova_available      : true
openstack::tempest::neutron_available   : true
openstack::tempest::heat_available      : false
openstack::tempest::swift_available     : false

######## Log levels
openstack::verbose: 'True'
openstack::debug: 'True'

List of module installed with r10k:

/etc/puppet/modules
├── duritong-sysctl (v0.0.8)  invalid
├── erlang (???)
├── memcached (???)
├── nanliu-staging (v1.0.2)
├── puppetlabs-apache (v1.2.0)
├── puppetlabs-apt (v1.4.1)  invalid
├── puppetlabs-concat (v1.1.2)
├── puppetlabs-firewall (v1.6.0)
├── puppetlabs-inifile (v1.1.4)
├── puppetlabs-mongodb (v0.10.0)
├── puppetlabs-mysql (v3.3.0)
├── puppetlabs-ntp (v3.0.4)
├── puppetlabs-openstack (v5.0.2)
├── puppetlabs-postgresql (v4.0.0)  invalid
├── puppetlabs-puppetdb (v4.0.0)
├── puppetlabs-rabbitmq (v5.2.1)  invalid
├── puppetlabs-stdlib (v4.3.2)
├── qpid (???)
├── rsync (???)
├── ssh (???)
├── stackforge-ceilometer (v5.0.0)
├── stackforge-cinder (v5.0.0)
├── stackforge-glance (v5.0.0)
├── stackforge-heat (v5.0.0)
├── stackforge-horizon (v5.0.0)
├── stackforge-keystone (v5.0.0)
├── stackforge-neutron (v5.0.0)
├── stackforge-nova (v5.0.0)
├── stackforge-openstack_extras (v5.0.0)
├── stackforge-openstacklib (v5.0.0)
├── stackforge-swift (v5.0.0)
├── stackforge-tempest (v5.0.0)
├── stackforge-vswitch (v1.0.0)
├── stahnma-epel (v1.0.2)
├── vcsrepo (???)
└── xinetd (???)

My neutron node run Ubuntu 14.04 and have only one interface (eth0). Regards.

To fix the connectivity issue stop and uninstall firewalld and NetworkManager packages/servers during kickstart your server. Seems like OpenStack juno neutron module doesn't like with NetworkManage .

I put following in my kickstart file under %post.

systemctl mask firewalld systemctl stop firewalld systemctl mask NetworkManager.service systemctl stop NetworkManager.service yum remove -y firewalld yum remove -y NetworkManager

Make sure iptables-services is installed . The minimum installation doesn't add this package.

@sakibsyl: thanks for your input. Since @bunnis and @Oyabi are both running Ubuntu, I don't think firewalld or NetworkManager are related to this particular issue, but your instructions might be able to help someone else.

@Oyabi this module is expected to run on a machine with at least two interfaces and up to five interfaces. The reason this is failing for you is because your default route goes through your only interface. When you add the brex bridge to it, network traffic will stop working. You can read about this here. I would recommend running this on a machine with at least two interfaces, one to be used as a management interface. If this is not possible, you can fix the issue manually with this guide.

@bunnis I believe your issue is similar to @Oyabi's but you haven't provided the information I requested earlier. If your puppet master is repaired, could you provide that information, especially the output of ifconfig and ovs-vsctl, as well as more information on your manual alterations to the default gateway?

Hi @cmurphy Sorry I havent payed much attention to this as I have been busy with other projects. I stopped using your all-in-one solution as I couldnt control all the steps. Right now I deployed everything using stackforge modules. I'm still having some issues with neutron but all the other services are up and running. I believe both puppet-all in one and stackforge-neutron dont create/add ports to the bridges br-ext br-int br-tun and Im trying to configure that right now. Ill be checking the guides you posted. Some info from my network node : (Although I've manually added br-ex to my internet port i still cant ping)

root@fing:/home/fing# ifconfig
br-ex     Link encap:Ethernet  HWaddr 00:22:19:d6:82:cd
          inet addr:10.155.216.205  Bcast:10.155.219.255  Mask:255.255.252.0
          inet6 addr: fe80::442b:c4ff:fec9:71c4/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:985225 errors:0 dropped:6348 overruns:0 frame:0
          TX packets:684 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:164436815 (164.4 MB)  TX bytes:59348 (59.3 KB)

br-int    Link encap:Ethernet  HWaddr 02:f5:f9:f9:af:41
          inet6 addr: fe80::c69:49ff:feb8:978b/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:17 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1400 (1.4 KB)  TX bytes:648 (648.0 B)

br-tun    Link encap:Ethernet  HWaddr 5a:84:d7:76:8e:43
          inet6 addr: fe80::34f0:4ff:fea3:fd9/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)

eth0      Link encap:Ethernet  HWaddr 00:22:19:d6:82:cc
          inet addr:192.168.0.7  Bcast:192.168.1.255  Mask:255.255.254.0
          inet6 addr: fe80::222:19ff:fed6:82cc/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:236992 errors:0 dropped:0 overruns:0 frame:0
          TX packets:255333 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:97384574 (97.3 MB)  TX bytes:71563381 (71.5 MB)
          Interrupt:16

eth1      Link encap:Ethernet  HWaddr 00:22:19:d6:82:cd
          inet6 addr: fe80::222:19ff:fed6:82cd/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:1901449 errors:593 dropped:4 overruns:0 frame:5
          TX packets:1319 errors:0 dropped:0 overruns:0 carrier:0
          collisions:1 txqueuelen:1000
          RX bytes:586724803 (586.7 MB)  TX bytes:134972 (134.9 KB)
          Interrupt:17

eth2      Link encap:Ethernet  HWaddr 00:15:17:b9:6d:76
          inet addr:192.168.10.7  Bcast:192.168.10.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:16 Memory:dfc80000-dfca0000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:19 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2272 (2.2 KB)  TX bytes:2272 (2.2 KB)

root@fing:/home/fing# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    0      0        0 eth0
10.155.216.0    0.0.0.0         255.255.252.0   U     0      0        0 br-ex
192.168.0.0     0.0.0.0         255.255.254.0   U     0      0        0 eth0
192.168.10.0    0.0.0.0         255.255.255.0   U     0      0        0 eth2
root@fing:/home/fing#           options: {peer=patch-int}
options:: command not found
root@fing:/home/fing#     ovs_version: "2.0.2"
ovs_version:: command not found
root@fing:/home/fing# ovs-vsctl show
fce3de98-b573-48a8-a060-b89d02abe907
    Bridge br-tun
        fail_mode: secure
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port "qg-3bf41e31-77"
            Interface "qg-3bf41e31-77"
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-int
        fail_mode: secure
        Port "tapb27fc70a-a1"
            tag: 4095
            Interface "tapb27fc70a-a1"
                type: internal
        Port "qr-a1ffa3ae-e1"
            tag: 4095
            Interface "qr-a1ffa3ae-e1"
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    ovs_version: "2.0.2"
root@fing:/home/fing# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
        address 192.168.0.7
        netmask 255.255.254.0
        network 192.168.0.0
        broadcast 192.168.1.255
        gateway 192.168.0.1
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 192.168.0.1

#up route add -net 192.168.0.0 netmask 255.255.254.0 gw 192.168.0.1 eth0

auto eth1
iface eth1 inet manual
        up ifconfig $IFACE 0.0.0.0 up
        up ip link set $IFACE promisc on
        down ip link set $IFACE promisc off
        down ifconfig $IFACE down

auto br-ex
iface br-ex inet static
       address 10.155.216.205
       netmask 255.255.252.0
       #network 192.168.0.0
       #broadcast 192.168.1.255
       gateway 10.155.216.1
       # dns-* options are implemented by the resolvconf package, i$
       dns-nameservers 8.8.8.8
 #      up route add -net 10.155.216.0 netmask 255.255.252.0 gw 10.155.216.1 br-ex

auto eth2
iface eth2 inet static
address 192.168.10.7
netmask 255.255.255.0

@bunnis,

I stopped using your all-in-one solution as I couldnt control all the steps. Right now I deployed everything using stackforge modules.

I think that is actually a great approach, as this module is not flexible enough for many deployment scenarios. At this point, if you're still having trouble with neutron or OpenVSwitch I recommend getting help from the puppet-openstack team or seeking help with OpenStack itself rather than with the puppet modules.

May I close this issue for now? If you find this module needs to be modified we can reopen this issue or you can open a pull request.

puppetlabs-toy-chest / puppetlabs-openstack

loss of conectivity after neutron deployment #176

You said "i manually altered the default gateway prior to deployment", can you verify that you are able to reach the external network after manually altering the gateway but before applying any puppet configuration?

the output of puppet agent -t for applying configuration to the unconfigured network node (make sure the controller is configured first)

the output of puppet module list from the puppet master