sonic-net / sonic-mgmt

Configuration management examples for SONiC
Other
195 stars 707 forks source link

[Dual-Tor][Active-Active] Failed to add dualtor-aa topology due to missing ";" in the grpc proto files #8723

Closed congh-nvidia closed 1 year ago

congh-nvidia commented 1 year ago

Description

Failed to add dualtor-aa topology via testbed-cli.sh, this PR https://github.com/sonic-net/sonic-mgmt/pull/8017 removed all the semicolons(";") in the proto files, and looks like this is the cause of the failure.

Steps to reproduce the issue:

  1. Run this command to add the dualtor-aa topology: ./testbed-cli.sh -t testbed.yaml -k ceos add-topo sonic-dual-tor-tigon-dualtor-aa vault -e ptf_imagetag=latest -vvvvv

Describe the results you received:

TASK [vm_set : Generate gRPC dependencies from the protocol buffer definitions] ************************************************
task path: /root/mars/workspace/sonic-mgmt/ansible/roles/vm_set/tasks/control_nic_simulator.yml:61
Monday 26 June 2023  12:46:53 +0300 (0:00:01.979)       0:05:13.814 *********** 
Using module file /usr/local/lib/python2.7/dist-packages/ansible/modules/commands/command.py
Pipelining is enabled.
<r-r640-10> ESTABLISH SSH CONNECTION FOR USER: svc-nbu-sws-sonic
<r-r640-10> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=120s)(-o)(UserKnownHostsFile=/dev/null)(-o)(StrictHostKeyChecking=no)
<r-r640-10> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<r-r640-10> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="svc-nbu-sws-sonic")
<r-r640-10> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=30)
<r-r640-10> SSH: PlayContext set ssh_common_args: ()
<r-r640-10> SSH: PlayContext set ssh_extra_args: ()
<r-r640-10> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/root/.ansible/cp/c941c902d7)
<r-r640-10> SSH: EXEC sshpass -d10 ssh -vvv -o ControlMaster=auto -o ControlPersist=120s -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o StrictHostKeyChecking=no -o 'User="svc-nbu-sws-sonic"' -o ConnectTimeout=30 -o ControlPath=/root/.ansible/cp/c941c902d7 r-r640-10 '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
<r-r640-10> (1, '\n{"changed": true, "end": "2023-06-26 12:46:53.554683", "stdout": "", "cmd": "python3 -m grpc.tools.protoc -I. --python_out=. --grpc_python_out=. nic_simulator_grpc_service.proto", "failed": true, "delta": "0:00:00.180635", "stderr": "nic_simulator_grpc_service.proto:3:1: Expected \\";\\".", "rc": 1, "invocation": {"module_args": {"creates": null, "executable": null, "_uses_shell": true, "strip_empty_ends": true, "_raw_params": "python3 -m grpc.tools.protoc -I. --python_out=. --grpc_python_out=. nic_simulator_grpc_service.proto", "removes": null, "argv": null, "warn": true, "chdir": "/images/veos-vm/nic_simulator", "stdin_add_newline": true, "stdin": null}}, "start": "2023-06-26 12:46:53.374048", "msg": "non-zero return code"}\n', 'OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f  31 Mar 2020\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 25359\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n')
<r-r640-10> Failed to connect to the host via ssh: OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 25359
debug3: mux_client_request_session: session request sent
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 1
failed: [R-R640-10] (item=nic_simulator_grpc_service.proto) => {
    "ansible_loop_var": "item", 
    "changed": true, 
    "cmd": "python3 -m grpc.tools.protoc -I. --python_out=. --grpc_python_out=. nic_simulator_grpc_service.proto", 
    "delta": "0:00:00.180635", 
    "end": "2023-06-26 12:46:53.554683", 
    "invocation": {
        "module_args": {
            "_raw_params": "python3 -m grpc.tools.protoc -I. --python_out=. --grpc_python_out=. nic_simulator_grpc_service.proto", 
            "_uses_shell": true, 
            "argv": null, 
            "chdir": "/images/veos-vm/nic_simulator", 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "stdin": null, 
            "stdin_add_newline": true, 
            "strip_empty_ends": true, 
            "warn": true
        }
    }, 
    "item": "nic_simulator_grpc_service.proto", 
    "msg": "non-zero return code", 
    "rc": 1, 
    "start": "2023-06-26 12:46:53.374048", 
    "stderr": "nic_simulator_grpc_service.proto:3:1: Expected \";\".", 
    "stderr_lines": [
        "nic_simulator_grpc_service.proto:3:1: Expected \";\"."
    ], 
    "stdout": "", 
    "stdout_lines": []
}
Using module file /usr/local/lib/python2.7/dist-packages/ansible/modules/commands/command.py
Pipelining is enabled.
<r-r640-10> ESTABLISH SSH CONNECTION FOR USER: svc-nbu-sws-sonic
<r-r640-10> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=120s)(-o)(UserKnownHostsFile=/dev/null)(-o)(StrictHostKeyChecking=no)
<r-r640-10> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<r-r640-10> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="svc-nbu-sws-sonic")
<r-r640-10> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=30)
<r-r640-10> SSH: PlayContext set ssh_common_args: ()
<r-r640-10> SSH: PlayContext set ssh_extra_args: ()
<r-r640-10> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/root/.ansible/cp/c941c902d7)
<r-r640-10> SSH: EXEC sshpass -d10 ssh -vvv -o ControlMaster=auto -o ControlPersist=120s -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o StrictHostKeyChecking=no -o 'User="svc-nbu-sws-sonic"' -o ConnectTimeout=30 -o ControlPath=/root/.ansible/cp/c941c902d7 r-r640-10 '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
<r-r640-10> (1, '\n{"changed": true, "end": "2023-06-26 12:46:53.828315", "stdout": "", "cmd": "python3 -m grpc.tools.protoc -I. --python_out=. --grpc_python_out=. nic_simulator_grpc_mgmt_service.proto", "failed": true, "delta": "0:00:00.175599", "stderr": "[libprotobuf WARNING third_party/protobuf/src/google/protobuf/compiler/parser.cc:649] No syntax specified for the proto file: nic_simulator_grpc_mgmt_service.proto. Please use \'syntax = \\"proto2\\";\' or \'syntax = \\"proto3\\";\' to specify a syntax version. (Defaulted to proto2 syntax.)\\nnic_simulator_grpc_mgmt_service.proto:2:1: Expected \\";\\".", "rc": 1, "invocation": {"module_args": {"creates": null, "executable": null, "_uses_shell": true, "strip_empty_ends": true, "_raw_params": "python3 -m grpc.tools.protoc -I. --python_out=. --grpc_python_out=. nic_simulator_grpc_mgmt_service.proto", "removes": null, "argv": null, "warn": true, "chdir": "/images/veos-vm/nic_simulator", "stdin_add_newline": true, "stdin": null}}, "start": "2023-06-26 12:46:53.652716", "msg": "non-zero return code"}\n', 'OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f  31 Mar 2020\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 25359\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n')
<r-r640-10> Failed to connect to the host via ssh: OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 25359
debug3: mux_client_request_session: session request sent
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 1
failed: [R-R640-10] (item=nic_simulator_grpc_mgmt_service.proto) => {
    "ansible_loop_var": "item", 
    "changed": true, 
    "cmd": "python3 -m grpc.tools.protoc -I. --python_out=. --grpc_python_out=. nic_simulator_grpc_mgmt_service.proto", 
    "delta": "0:00:00.175599", 
    "end": "2023-06-26 12:46:53.828315", 
    "invocation": {
        "module_args": {
            "_raw_params": "python3 -m grpc.tools.protoc -I. --python_out=. --grpc_python_out=. nic_simulator_grpc_mgmt_service.proto", 
            "_uses_shell": true, 
            "argv": null, 
            "chdir": "/images/veos-vm/nic_simulator", 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "stdin": null, 
            "stdin_add_newline": true, 
            "strip_empty_ends": true, 
            "warn": true
        }
    }, 
    "item": "nic_simulator_grpc_mgmt_service.proto", 
    "msg": "non-zero return code", 
    "rc": 1, 
    "start": "2023-06-26 12:46:53.652716", 
    "stderr": "[libprotobuf WARNING third_party/protobuf/src/google/protobuf/compiler/parser.cc:649] No syntax specified for the proto file: nic_simulator_grpc_mgmt_service.proto. Please use 'syntax = \"proto2\";' or 'syntax = \"proto3\";' to specify a syntax version. (Defaulted to proto2 syntax.)\nnic_simulator_grpc_mgmt_service.proto:2:1: Expected \";\".", 
    "stderr_lines": [
        "[libprotobuf WARNING third_party/protobuf/src/google/protobuf/compiler/parser.cc:649] No syntax specified for the proto file: nic_simulator_grpc_mgmt_service.proto. Please use 'syntax = \"proto2\";' or 'syntax = \"proto3\";' to specify a syntax version. (Defaulted to proto2 syntax.)", 
        "nic_simulator_grpc_mgmt_service.proto:2:1: Expected \";\"."
    ], 
    "stdout": "", 
    "stdout_lines": []
}

PLAY RECAP *********************************************************************************************************************
R-R640-10                  : ok=272  changed=29   unreachable=0    failed=1    skipped=280  rescued=0    ignored=0   

Describe the results you expected: The dualtor-aa topology should be successfully deployed.

Additional information you deem important:

**Output of `show version`:**

```
(paste your output here)
```

**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```
congh-nvidia commented 1 year ago

Fixed by https://github.com/sonic-net/sonic-mgmt/pull/8761