Try to seek for race conditions occurring in Actions only

bertllll commented 3 years ago

I've experienced them more than little times in: Node/masq/communication/communication_manager at more places in there.

dnwiebe commented 3 years ago

We can move this out of Backlog when we get some specific details about particular tests that fail sometimes but not always in Actions. When you experience that, post a description in a comment here and attach the downloaded console log from Actions.

bertllll commented 3 years ago

I got this despite this area of our code could not be affected by the worked card (GH-381). Happened after a break from seeing any strangely failing tests. When a new Rust version came.

thread 'main' panicked at 'ClientListenerThread could not be restarted: Timeout', D:\a\Node\Node\masq\src\communications\connection_manager.rs:455:23 stack backtrace: 0: 0x7ff74c24187e - std::backtrace_rs::backtrace::dbghelp::trace test communications::connection_manager::tests::handles_outgoing_fire_and_forget_messages_to_dead_server ... FAILED at /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b\/library\std\src....\backtrace\src\backtrace\dbghelp.rs:98 1: 0x7ff74c24187e - std::backtrace_rs::backtrace::trace_unsynchronized at /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b\/library\std\src....\backtrace\src\backtrace\mod.rs:66 2: 0x7ff74c24187e - std::sys_common::backtrace::_print_fmt at /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b\/library\std\src\sys_common\backtrace.rs:67

communications::connection_manager::tests::handles_outgoing_fire_and_forget_messages_to_dead_server

bertllll commented 3 years ago

This problem occurred at the same day as the previous but I didn't have time to report it. It showed itself once and in the next run Actions did not complain though I did no changes in the code.

log multinode test

multinode_integration_tests/tests/connection_termination_test

dnwiebe commented 3 years ago

One of the multinode tests failed: provided_and_consumed_services_are_recorded_in_databases. logs_473.zip

The same test...Bert. 2_Build node on ubuntu-latest.txt

Again 2_Build node on ubuntu-latest.txt

bertllll commented 3 years ago

Windows ...one of node unit tests. No relation to the card being coped (GH-373).

logs_509.zip

bertllll commented 3 years ago

configurator_standard ... setting the gas price

logs_519.zip

bertllll commented 3 years ago

windows: tls_through_node_test (integration); I know this test because that's the one which permanently fails on my Win machine.

logs_530.zip

bertllll commented 3 years ago

MacOS: Node: unit_tests: initialize_with_clandestine_port_produces_expected_clandestine_discriminator_factories_vector

logs_599.zip

bertllll commented 3 years ago

This is unfortunate. By all evidences, GitHub stopped to support zipped packages and these ours which are attached to the comments here are inaccessible by now. I'd be happy if I was wrong.

Now it accept zipped folders again...so...I don't know. But we lost the old ones.

bertllll commented 3 years ago

handle_startup_and_shutdown_integration

This test was renovated for this card, MASQ-Project/MASQ-Node-issues#386, however, usually it passes, this time it didn't. Maybe RC.

3_Build node on windows-latest.txt ... 3_Build node on windows-latest.txt

And another one: 22.6.21 3_Build node on windows-latest.txt

Hmmm...this is growing up fast 24.6.2021 3_Build node on windows-latest.txt 26.6.2021 twice 3_Build node on windows-latest.txt 3_Build node on windows-latest.txt

In MASQ-Project/MASQ-Node-issues#451, I was trying to fix this by adding a piece of logic to the code. It may happen that this error will evaporate then.

Nope, without a relief, the frequency of this failure may even have increased.

3_Build node on windows-latest.txt

27th of September I tried to prevent this issue more reliably with a modification in fallback - connection manager. Card MASQ-Project/MASQ-Node-issues#305.

4th October Not fixed yet

bertllll commented 3 years ago

handle_redirect_order_disappoints_waiting_conversations_with_resend_or_graceful

Maybe newly created RC after I did a lot of modifications (GH-386) in the UI where some could affect the connection_manager in his work, well, maybe not.

3_Build node on windows-latest.txt

bertllll commented 3 years ago

communications::connection_manager::tests::handles_listener_fallback_from_node

Another test that I don't remember failing. Again MASQ-Project/MASQ-Node-issues#386.

3_Build node on windows-latest.txt

bertllll commented 3 years ago

provided_and_consumed_services_are_recorded_in_databases

Multinode-integration-test...I remember seeing this before. Yup. Look up there.

Again, this was during MASQ-Project/MASQ-Node-issues#386.

2_Build node on ubuntu-latest.txt

7.7.22 2_Build.node.on.ubuntu-latest.txt

bertllll commented 3 years ago

connection_manager::tests::handles_fire_and_forget_outgoing_message

I thought that this had been seen multiple times, but I'm not sure now.

2_Build node on ubuntu-latest.txt

This time on MacOs 1_Build node on macOS-latest.txt

Again... 2_Build node on macOS-latest.txt

bertllll commented 3 years ago

test_utils::mock_websockets_server::tests::conversational_and_broadcast_messages_can_work_together_testing_corner_cases

mac_os

1_Build node on macOS-latest.txt

bertllll commented 3 years ago

I'm adding this after a while passed from when it happened; I found it my Downloads folder.

blockchain::blockchain_interface::tests::blockchain_interface_non_clandestine_retrieves_transactions 2_Build node on ubuntu-latest.txt

1st of Sep 2_Build node on ubuntu-latest.txt

23rd of Sep 2_Build node on ubuntu-latest.txt

bertllll commented 3 years ago

similarly, I found this:

websocket_supervisor::tests::send_msg_fails_to_look_up_client_to_send_to 3_Build node on windows-latest.txt

bertllll commented 3 years ago

Seems like we've got a new one. blockchain::blockchain_interface::tests::blockchain_interface_non_clandestine_retrieve_transactions_returns_an_error_if_a_response_with_data_that_is_too_long_is_returned 2_Build node on ubuntu-latest.txt

bertllll commented 3 years ago

Too many of blockchain_interface lately, feeling suspicious. blockchain::blockchain_interface::tests::blockchain_interface_non_clandestine_retrieve_transactions_ignores_transaction_logs_that_have_no_block_number 2_Build node on ubuntu-latest.txt

From earlier: 2_Build node on ubuntu-latest.txt

A very old one.. 2_Build node on ubuntu-latest.txt

2_Build node on ubuntu-latest.txt

bertllll commented 3 years ago

neighborhood::gossip_acceptor::tests::standard_gossip_containing_unfamiliar_node_addrs_leads_to_them_being_ignored 2_Build node on ubuntu-latest.txt

bertllll commented 3 years ago

Multinode... provided_and_consumed_services_are_recorded_in_databases 2_Build node on ubuntu-latest.txt

Don't know why yet, but now a time of a lot of failures in multinode tests has come.

Another: provided_and_consumed_services_are_recorded_in_databases 2_Build node on ubuntu-latest.txt

2_Build node on ubuntu-latest.txt

Seems like the manifestation of the error can be slightly different at different times.

bertllll commented 3 years ago

multinode downed_nodes_not_offered_in_passes_or_introductions 2_Build node on ubuntu-latest.txt

2_Build node on ubuntu-latest.txt

21/12/2022 logs_1396.zip

bertllll commented 3 years ago

blockchain::blockchain_interface::tests::blockchain_interface_non_clandestine_can_retrieve_eth_balance_of_a_wallet 2_Build node on ubuntu-latest.txt

FinsaasGH commented 3 years ago

multinode multiple_stream_zero_hop_test 2_Build node on ubuntu-latest.txt

bertllll commented 3 years ago

pass_args_or_print_messages_work_under_fine_lock_for_eof

I guess this is a rare one. I suggest just to increase the multiplier in the assertion like from 3 to 4 or something. I know there is a couple of tests like this and this was the latest I added, maybe the chosen multiplier doesn't fit as well as for the other tests, I think its value could be a bit test specific.

1_Build node on macOS-latest.txt

bertllll commented 2 years ago

tls_through_node_integration

This one is really rare in Actions I think, on the other hand we come to see it quite often on dev machines. Speaking mainly for myself (maybe occurred almost all the time) but I also saw it at Utkarsh.

3_Build node on windows-latest.txt

dnwiebe commented 2 years ago

The multinode test tls_end_to_end_routing_test hung for most of five hours. We don't know why, but here's the log.

logs_830.zip 2_Build node on ubuntu-latest.txt