interface_driver in l3_agent.ini should be specified for OVN Neutron driver

[TsutomuNakamura]

Interface driver should be specified in l3_agent.ini like below.

• /etc/neutron/l3_agent.ini

  [DEFAULT]
  # ...
  interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

Neutron will output errors if interface_driver was not specified when we use OVN Neutron driver.

[TsutomuNakamura]

Compute nodes are recovered but a controller node is still hanging up.

Commands below is an memo to test.

Test Case1

• site.yml

• import_playbook: commons.yml
• import_playbook: controllers.yml

  - import_playbook: computes.yml

  - import_playbook: controllers_discover_hosts.yml

  - import_playbook: controllers_create_example_instances.yml

Run the command below.

docker run --rm \
    --add-host dev-controller01:<IP> \
    --add-host dev-compute01:<IP> \
    --add-host dev-compute02:<IP> \
    --volume ${PWD}:/opt/ansible \
    --volume <src_private_key>:<dst_private_key> \
    -ti tsutomu/ansible-runner \
    --user <user> -i production -l dev-controller01:dev-compute01:dev-compute02

Check whether controller node is up or not.
-> The controller was NOT hanging up.

Check which instructions will cause hung up controller. Continue testing...

• site.yml

• import_playbook: commons.yml

  - import_playbook: controllers.yml

• import_playbook: computes.yml

  - import_playbook: controllers_discover_hosts.yml

  - import_playbook: controllers_create_example_instances.yml

(Run the command ...)

The controller was NOT hanging up so far.

• site.yml

  
  #- import_playbook: commons.yml
  #- import_playbook: controllers.yml
  #- import_playbook: computes.yml

• import_playbook: controllers_discover_hosts.yml

  - import_playbook: controllers_create_example_instances.yml

(Run the command ...)

The controller was NOT hanging up so far.

• site.yml

  
  #- import_playbook: commons.yml
  #- import_playbook: controllers.yml
  #- import_playbook: computes.yml
  #- import_playbook: controllers_discover_hosts.yml

• import_playbook: controllers_create_example_instances.yml

  
  **The controller was hanging up.**

• htop on the controller node

  neutron-server: api worker (/usr/bin/python3 /usr/bin/neutron-server --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --log-file=/var/log/neutron/neutron-server.log)

controllers_create_example_instances.yml will make a controller node hanging up.

[TsutomuNakamura]

Cause of the issue might be in a role test_env.

Test01

• site.yml

  
  #- import_playbook: commons.yml
  #- import_playbook: controllers.yml
  #- import_playbook: computes.yml
  #- import_playbook: controllers_discover_hosts.yml

• import_playbook: controllers_create_example_instances.yml

• roles/test_env/tasks/main.yml

• name: Determin which network driver will be used. Linux Bridge or OVN are only supported so far. set_fact: network_plugin: "{% if 'network' not in group_vars.openstack or 'plugin' not in group_vars.openstack.network or 'Linux Bridge' == group_vars.openstack.network.plugin %}Linux Bridge{% else %}{{ group_vars.openstack.network.plugin }}{% endif %}"

Pre-bridge-configuration for OVN

• name: Include configure_bridge_for_ovn/main.yml include: configure_bridge_for_ovn/main.yml when: "'OVN' == network_plugin"

- name: Block create test env on controller node

block:

... comment out

when: "'controllers' in group_names"

Run Ansible.

docker run --rm \ --add-host dev-controller01: \ --add-host dev-compute01: \ --add-host dev-compute02: \ --volume ${PWD}:/opt/ansible \ --volume : \ -ti tsutomu/ansible-runner \ --user -i production -l dev-controller01:dev-compute01:dev-compute02

It did not cause any issues.

## Test 02
Changed roles/test_env/tasks/main.yml only from the above.

......

• name: Block create test env on controller node block:

  • name: Include create_example_network/main.yml to create test environment include: create_example_network/main.yml

  • name: Include create_example_subnet/main.yml to create test environment include: create_example_subnet/main.yml

  • name: Include create_example_router/main.yml to create test environment include: create_example_router/main.yml

  • name: Include create_flavor/main.yml to create test environment include: create_example_flavor/main.yml

......

when: "'controllers' in group_names"

It makes controller hanging up.

## Test 03

* roles/test_env/tasks/main.yml

• name: Block create test env on controller node block:

  • name: Include create_example_network/main.yml to create test environment include: create_example_network/main.yml

  • name: Include create_example_subnet/main.yml to create test environment include: create_example_subnet/main.yml

......

when: "'controllers' in group_names"

It did NOT make controller hanging up.

## Test 04

.......

• name: Block create test env on controller node block:

  • name: Include create_example_network/main.yml to create test environment include: create_example_network/main.yml

  • name: Include create_example_subnet/main.yml to create test environment include: create_example_subnet/main.yml

  • name: Include create_example_router/main.yml to create test environment include: create_example_router/main.yml

......

when: "'controllers' in group_names"

It Makes controller hanging up.
There is a cause in `roles/test_env/tasks/create_example_router/do_create_router.yml`.

[TsutomuNakamura]

Test 05

• roles/test_env_tasks/create_example_router/do_create_router.yml

• name: Block of creating router block:

  • name: Create a router ansible.builtin.command: argv: ["openstack", "router", "create", "{{ router.name }}"] register: result

  • name: Print the result of creating router ansible.builtin.debug: msg: "{{ result.stdout.split('\n') }}"

  • name: Set a gateway to the router ansible.builtin.command: argv: ["openstack", "router", "set", "--external-gateway", "{{ router.external_gateway }}", "{{ router.name }}"] register: result

  • name: Print the result of setting a gateway ansible.builtin.debug: msg: "{{ result.stdout.split('\n') }}"

  ......

  when: result_of_checking_router.rc == 0

It makes hanging up.

- name: Block of creating router
  block:
    - name: Create a router
      ansible.builtin.command:
        argv: ["openstack", "router", "create", "{{ router.name }}"]
      register: result

    - name: Print the result of creating router
      ansible.builtin.debug:
        msg: "{{ result.stdout.split('\n') }}"

    # ......
  when: result_of_checking_router.rc == 0

It did NOT make hanging up.

- name: Block of creating router
  block:
    - name: Create a router
      ansible.builtin.command:
        argv: ["openstack", "router", "create", "{{ router.name }}"]
      register: result

    - name: Print the result of creating router
      ansible.builtin.debug:
        msg: "{{ result.stdout.split('\n') }}"

    - name: Set a gateway to the router
      ansible.builtin.command:
        argv: ["openstack", "router", "set", "--external-gateway", "{{ router.external_gateway }}", "{{ router.name }}"]
      register: result

    # ......
  when: result_of_checking_router.rc == 0

It makes hanging up.

openstack router set --external-gateway ...

Makes hanging up.

[TsutomuNakamura]

Restarting controller node after ...

    - name: Set a gateway to the router
      ansible.builtin.command:
        argv: ["openstack", "router", "set", "--external-gateway", "{{ router.external_gateway }}", "{{ router.name }}"]
      register: result

will solve hanging up.