[Service Bus & Event Hubs] Improve livetest test stability

[yunhaoling]

• finding if there's any tests repeatedly failing in our livetest
• whether it's an issue in SDK or VM agents' problem
• if not issue in SDK, improve the test

[swathipil]

check devops and check if something can be combined with stress test onboarding

[swathipil]

SB test flaky: test_queue_send_twice: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1321365&view=logs&j=217ad987-cc73-5dcd-abbd-eb23972de726&t=f10366f4-42a0-5bf3-0a2a-6be221a21989&l=1873

[swathipil]

update:

• have been checking the nightly test runs, and there are no consistently flaky tests other than one above. Just some other ones only failing once from what I've seen:

EH: tests/livetest/synctests/test_consumer_client.py::test_receive_batch_no_max_wait_time

SB: FAILED tests/test_subscriptions.py::ServiceBusSubscriptionTests::test_subscription_by_servicebus_client_receive_batch_with_deadletter: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1326700&view=logs&j=fa284728-64c9-5de0-4442-32743d14c85e&t=b93d123d-eb8b-5272-3a5e-e2a5347ca4ca&l=2272 FAILED tests/async_tests/test_queues_async.py::ServiceBusQueueAsyncTests::test_async_queue_by_servicebus_client_browse_messages_with_receiver FAILED tests\test_queues.py::ServiceBusQueueTests::test_queue_by_queue_client_conn_str_receive_handler_with_autolockrenew - https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1326700&view=logs&j=217ad987-cc73-5dcd-abbd-eb23972de726&t=f10366f4-42a0-5bf3-0a2a-6be221a21989&l=2126 FAILED tests/test_queues.py::ServiceBusQueueTests::test_queue_send_dict_messages_scheduled FAILED tests/test_queues.py::ServiceBusQueueTests::test_queue_operation_negative

[swathipil]

check 12/28: FAILED tests/async_tests/test_queues_async.py::ServiceBusQueueAsyncTests::test_async_queue_by_queue_client_conn_str_receive_handler_peeklock

12/31: FAILED tests/test_sessions.py::ServiceBusSessionTests::test_session_by_servicebus_client_session_pool FAILED tests/test_sessions.py::ServiceBusSessionTests::test_session_by_servicebus_client_session_pool FAILED tests/test_sessions.py::ServiceBusSessionTests::test_session_connection_failure_is_idempotent FAILED tests/async_tests/test_sessions_async.py::ServiceBusAsyncSessionTests::test_async_session_connection_failure_is_idempotent FAILED tests/async_tests/test_queues_async.py::ServiceBusQueueAsyncTests::test_async_queue_receiver_respects_max_wait_time_overrides FAILED tests/async_tests/test_sessions_async.py::ServiceBusAsyncSessionTests::test_async_session_by_servicebus_client_session_pool FAILED tests/async_tests/test_sessions_async.py::ServiceBusAsyncSessionTests::test_async_session_by_session_client_conn_str_receive_handler_with_no_session FAILED tests/async_tests/test_sessions_async.py::ServiceBusAsyncSessionTests::test_async_session_cancel_scheduled_messages FAILED tests/async_tests/test_sessions_async.py::ServiceBusAsyncSessionTests::test_async_session_cancel_scheduled_messages FAILED tests/test_sessions.py::ServiceBusSessionTests::test_session_by_servicebus_client_session_pool FAILED tests/async_tests/test_queues_async.py::ServiceBusQueueAsyncTests::test_async_queue_receiver_respects_max_wait_time_overrides FAILED tests/async_tests/test_sessions_async.py::ServiceBusAsyncSessionTests::test_async_session_by_servicebus_client_session_pool FAILED tests/async_tests/test_sessions_async.py::ServiceBusAsyncSessionTests::test_async_session_by_session_client_conn_str_receive_handler_with_no_session FAILED tests/async_tests/test_subscriptions_async.py::ServiceBusSubscriptionAsyncTests::test_topic_by_servicebus_client_receive_batch_with_deadletter FAILED tests/test_queues.py::ServiceBusQueueTests::test_queue_receive_keep_conn_alive FAILED tests/test_sessions.py::ServiceBusSessionTests::test_session_by_servicebus_client_session_pool FAILED tests/test_sessions.py::ServiceBusSessionTests::test_session_connection_failure_is_idempotent FAILED tests/async_tests/test_queues_async.py::ServiceBusQueueAsyncTests::test_queue_receive_keep_conn_alive_async FAILED tests/mgmt_tests/test_mgmt_namespaces.py::ServiceBusManagementClientNamespaceTests::test_mgmt_namespace_get_properties

FIX: async/sync test_session_by_servicebus_client_session_pool: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1336668&view=logs&j=a5c73adf-21c4-51b0-3477-575974909b75&t=920caeb1-8790-523a-08a8-f5f4e4c5560b&l=1787

[swathipil]

A lot of these are failing b/c of this error: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1343200&view=logs&j=fe7804ca-eaac-501a-67e8-e2604a50cb52&t=581043c6-4578-52e9-e224-2083bb8809be&l=3251 and this: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1343200&view=logs&j=fe7804ca-eaac-501a-67e8-e2604a50cb52&t=581043c6-4578-52e9-e224-2083bb8809be&l=2979 And this one with Auth Timeout: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1343200&view=logs&j=fe7804ca-eaac-501a-67e8-e2604a50cb52&t=581043c6-4578-52e9-e224-2083bb8809be&l=1947

[swathipil]

receive_handler_with_auto_lock_renew: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1349066&view=logs&j=36c1d9f8-0c10-5405-9a7f-ac15f4c32de9&t=654dbbb3-fbc2-5bfe-7f7d-126a91b09b22&l=2191

[swathipil]

iter_with_defer_not_found: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1355297&view=logs&j=fa284728-64c9-5de0-4442-32743d14c85e&t=b93d123d-eb8b-5272-3a5e-e2a5347ca4ca&l=2009 test_message_lock_renew: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1358691&view=logs&j=217ad987-cc73-5dcd-abbd-eb23972de726&t=f10366f4-42a0-5bf3-0a2a-6be221a21989&l=4644

[swathipil]

The error in test_session_by_servicebus_client_session_pool with Cannot open log for source 'Microsoft.ServiceBus' seems to be an error with the service. Similar issue was filed here : https://github.com/Azure/azure-sdk-for-net/issues/27067

For now, ignoring this error in the test.

EDIT: It looks like this error is happening in every test where .get_queue_receiver(...session_id=NEXT_AVAILABLE_SESSION) and seems to be after this call results in an OperationTimeoutError. For ex: https://github.com/Azure/azure-sdk-for-python/blob/a1deaf3ed7ea91c96377ad8e895facc624454126/sdk/servicebus/azure-servicebus/tests/test_sessions.py#L259

[swathipil]

nightly runs have been green for a week so closing

[swathipil]

more failures 3/7: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1413650&view=results

[yunhaoling]

success criteria: 90% pass rate for the past two weeks

[swathipil]

most flaky EH tests:

• sync and async test_buffered_producer.py:

  • test_basic_send_single_events_round_robin[--False]

  • test_long_wait_small_buffer:

    with producer:
        for i in range(100):
            producer.send_event(EventData("test"))
    
    time.sleep(60)
    
    assert not on_error.err
    assert sum([len(sent_events[key]) for key in sent_events]) == 100
    >       assert sum([len(received_events[key]) for key in received_events]) == 100
    E       assert 50 == 100
    E         +50
    E         -100

  • test_basic_send_batch_events_round_robin[--False]

        if not flush_after_sending and not close_after_sending:
            # ensure it's buffered sending
            for pid in partitions:
                assert len(sent_events[pid]) < each_partition_cnt
            assert sum([len(sent_events[pid]) for pid in partitions]) < total_events_cnt
            # give some time for producer to complete sending and consumer to complete receiving
        else:
            if flush_after_sending:
                producer.flush()
            if close_after_sending:
                producer.close()
            # ensure all events are sent
            assert sum([len(sent_events[pid]) for pid in partitions]) == total_events_cnt
    
        time.sleep(10)
    >           assert len(sent_events) == len(received_events) == partitions_cnt
    E           assert 2 == 1
    E             +2
    E             -1

• sync and async test_send.py:

  • test_send_with_partition_key[***]:

    
    source = "amqps://{}/{}/ConsumerGroups/{}/Partitions/{}".format(
                    live_eventhub['hostname'],
                    live_eventhub['event_hub'],
                    live_eventhub['consumer_group'],
                    index)
                partition = uamqp.ReceiveClient(source, auth=sas_auth, debug=***, timeout=0, prefetch=500)
                reconnect_receivers.append(partition)
                retry_total += 1
        if retry_total == 3:
            raise OperationTimeoutError(f"Exhausted retries for receiving from {live_eventhub['hostname']}.")
    
    for r in reconnect_receivers:
        r.close()

  assert single_cnt == 60

E assert 30 == 60 E +30 E -60

In general, only receiving half of expected events when using a ConsumerClient to receive, instead of receivers in the conftest fixture.