Remove greenlet from asr1kl3 agent, use python threads instead

[notandy]

This patch removes greenlet from asr1kl3 agent by stripping eventlet (and thus monkey_patching), and replacing following (eventlet) based modules with python-thread based:

neutron.Manager -> embedded into main class L3ASRAgentWithStateReport oslo-service.looping_call -> futurist.periodics oslo-service.service -> python Thread with graceful_exit signal handler grenlet.greenpool -> futurist.ThreadPoolExecutor neutron.rpc -> oslo_messaging low level api (to switch it to threading)

Why?

Removing eventlet and replacing it with python.threading has the huge benefit that external libraries as ncclient doesn't need to be patched or prepared for eventlet. For a single-process and single worker-queue based agent python threading performance should be more than sufficient.

Fixes

ncclient (upstream version, no more double socket exceptions) graceful shutdown (all threads are gracefully stopped)

Reports

Also, there is now a regular thread report that can be manually triggered via kill -SIGUSR2 <pid>:

Currently running 8 periodic workers:
+----------------------------------------------------------------------------------------------------+--------+-------------+-----------+------+----------+-----------+----------------+-----------------+-------------------------+
|                                                Name                                                | Active | Periodicity |  Runs in  | Runs | Failures | Successes | Stop Requested | Average elapsed | Average elapsed waiting |
+----------------------------------------------------------------------------------------------------+--------+-------------+-----------+------+----------+-----------+----------------+-----------------+-------------------------+
|     asr1k_neutron_l3.plugins.l3.agents.asr1k_l3_agent.L3ASRAgent.periodic_requeue_routers_task     | False  |      60     |  30.6859s |  1   |    0     |     1     |     False      |     0.0002s     |         0.0007s         |
| asr1k_neutron_l3.plugins.l3.agents.asr1k_l3_agent.L3ASRAgent.periodic_refresh_address_scope_config | False  |      5      |  0.6895s  |  5   |    0     |     5     |     False      |     0.0621s     |         0.0002s         |
|        asr1k_neutron_l3.plugins.l3.agents.asr1k_l3_agent.L3ASRAgent.periodic_worker_report         | False  |     600     | 570.6783s |  0   |    0     |     0     |     False      |        .        |            .            |
|     asr1k_neutron_l3.plugins.l3.agents.asr1k_l3_agent.L3ASRAgentWithStateReport._report_state      | False  |     30.0    |  0.6798s  |  0   |    0     |     0     |     False      |        .        |            .            |
|      asr1k_neutron_l3.plugins.l3.agents.asr1k_l3_agent.L3ASRAgent._periodic_sync_routers_task      | False  |     120     |  91.3323s |  0   |    0     |     0     |     False      |        .        |            .            |
|        asr1k_neutron_l3.plugins.l3.agents.asr1k_l3_agent.L3ASRAgent._periodic_scavenge_task        | False  |     120     |  91.3325s |  0   |    0     |     0     |     False      |        .        |            .            |
|         asr1k_neutron_l3.plugins.l3.agents.asr1k_l3_agent.L3ASRAgent._check_devices_alive          | False  |     60.0    |  31.3325s |  0   |    0     |     0     |     False      |        .        |            .            |
|             asr1k_neutron_l3.plugins.l3.agents.asr1k_l3_agent.L3ASRAgent._clean_device             | False  |     600     | 571.7885s |  0   |    0     |     0     |     False      |        .        |            .            |
+----------------------------------------------------------------------------------------------------+--------+-------------+-----------+------+----------+-----------+----------------+-----------------+-------------------------+
Currently running 11 threads: ['MainThread', 'pydevd.Writer', 'pydevd.Reader', 'pydevd.CommandThread', 'pydevd.CheckAliveThread', 'Thread-7', 'ASR1k Worker Thread', 'Thread-8', 'Rabbit-heartbeat', 'Thread-10', 'Thread-11']

[joker-at-work]

Will we install gdb with python extension by default, so we can get some kind of backdoor or is there a console-thread started, too?

[notandy]

Will we install gdb with python extension by default, so we can get some kind of backdoor or is there a console-thread started, too?

Good question, I don't think python provides something similar by default, maybe there is a similar library out there? I will look into installing gdb-python via loci pipeline, which also make sense for all non-eventlet openstack projects.

[notandy]

https://pypi.org/project/manhole/ looks promising

[notandy]

Just evaluated backdoor in qa-de-2:

> hammer k backdoor neutron-asr1k-agent-01-5b68cb88fc-qsm65
Select a container:

  ▶ neutron-asr1k
    neutron-asr1k-ml2
/var/lib/neutron/eventlet_backdoor-1.socket
Calling  /usr/bin/socat -d -d TCP-LISTEN:33442,bind=localhost unix-connect:/var/lib/neutron/eventlet_backdoor-1.socket
Executing rlwrap --histsize 2147483647 socat stdio TCP:localhost:54985

######### ProcessID=1, ThreadID=139641907050240 #########
... (manhole prints a stacktrace of all threads on startup)

Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(ManholeConsole)
>>>