fluent / fluentd

Fluentd: Unified Logging Layer (project under CNCF)
https://www.fluentd.org
Apache License 2.0
12.89k stars 1.34k forks source link

CI: Fails with "Fluent::Plugin::ForwardOutput::NoNodesAvailable: no nodes are available" #4684

Open Watson1978 opened 2 hours ago

Watson1978 commented 2 hours ago

Describe the bug

Sometimes, CI fails with "Fluent::Plugin::ForwardOutput::NoNodesAvailable: no nodes are available" on macOS platform.

1. Error: test: a node supporting responses(ForwardOutputTest): Fluent::Plugin::ForwardOutput::NoNodesAvailable: no nodes are available
/Users/runner/work/fluentd/fluentd/lib/fluent/plugin/out_forward/load_balancer.rb:55:in `select_healthy_node' /Users/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/service_discovery/manager.rb:108:in` select_service'
/Users/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/service_discovery.rb:82:in `service_discovery_select_service' /Users/runner/work/fluentd/fluentd/lib/fluent/plugin/out_forward.rb:375:in` try_write'
/Users/runner/work/fluentd/fluentd/test/plugin/test_out_forward.rb:57:in `try_write' /Users/runner/work/fluentd/fluentd/lib/fluent/plugin/output.rb:1216:in` try_flush'
/Users/runner/work/fluentd/fluentd/lib/fluent/plugin/out_forward.rb:353:in `try_flush' /Users/runner/work/fluentd/fluentd/lib/fluent/plugin/output.rb:1538:in` flush_thread_run'
/Users/runner/work/fluentd/fluentd/lib/fluent/plugin/output.rb:510:in `block (2 levels) in start' /Users/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/thread.rb:78:in` block in thread_create'

To Reproduce

Run CI

Expected behavior

Succeeded

Your Environment

- Fluentd version:
- Package version:
- Operating system:
- Kernel version:

Your Configuration

none

Your Error Log

See above

Additional context

No response

Watson1978 commented 2 hours ago

Looks like the nodes which LoadBalancer has are unavailable when invoke after_shutdown at https://github.com/fluent/fluentd/blob/f493f25d0d33da8e16bab59d7856105788cdc7a5/test/plugin/test_out_forward.rb#L20

#<Thread:0x00000001378ecf50@flush_thread_0 /Users/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/thread.rb:70 run> terminated with exception (report_on_exception is true):
/Users/runner/work/fluentd/fluentd/lib/fluent/plugin/out_forward/load_balancer.rb:57:in `select_healthy_node': no nodes are available (Fluent::Plugin::ForwardOutput::NoNodesAvailable)
    from /Users/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/service_discovery/manager.rb:108:in `select_service'
    from /Users/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/service_discovery.rb:82:in `service_discovery_select_service'
    from /Users/runner/work/fluentd/fluentd/lib/fluent/plugin/out_forward.rb:375:in `try_write'
    from /Users/runner/work/fluentd/fluentd/test/plugin/test_out_forward.rb:57:in `try_write'
    from /Users/runner/work/fluentd/fluentd/lib/fluent/plugin/output.rb:1216:in `try_flush'
    from /Users/runner/work/fluentd/fluentd/lib/fluent/plugin/out_forward.rb:353:in `try_flush'
    from /Users/runner/work/fluentd/fluentd/lib/fluent/plugin/output.rb:1538:in `flush_thread_run'
    from /Users/runner/work/fluentd/fluentd/lib/fluent/plugin/output.rb:510:in `block (2 levels) in start'
    from /Users/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
EE, [2024-10-27T04:01:58.832977 #6300] ERROR -- : unexpected error while after_shutdown, Fluent::Plugin::ForwardOutput::NoNodesAvailable:no nodes are available
E, [2024-10-27T04:01:58.833250 #6300] ERROR -- :    /Users/runner/work/fluentd/fluentd/lib/fluent/plugin/out_forward/load_balancer.rb:57:in `select_healthy_node'
E, [2024-10-27T04:01:58.833282 #6300] ERROR -- :    /Users/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/service_discovery/manager.rb:108:in `select_service'
E, [2024-10-27T04:01:58.833304 #6300] ERROR -- :    /Users/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/service_discovery.rb:82:in `service_discovery_select_service'
E, [2024-10-27T04:01:58.833326 #6300] ERROR -- :    /Users/runner/work/fluentd/fluentd/lib/fluent/plugin/out_forward.rb:375:in `try_write'
E, [2024-10-27T04:01:58.833346 #6300] ERROR -- :    /Users/runner/work/fluentd/fluentd/test/plugin/test_out_forward.rb:57:in `try_write'
E, [2024-10-27T04:01:58.833365 #6300] ERROR -- :    /Users/runner/work/fluentd/fluentd/lib/fluent/plugin/output.rb:1216:in `try_flush'
E, [2024-10-27T04:01:58.833385 #6300] ERROR -- :    /Users/runner/work/fluentd/fluentd/lib/fluent/plugin/out_forward.rb:353:in `try_flush'
E, [2024-10-27T04:01:58.833403 #6300] ERROR -- :    /Users/runner/work/fluentd/fluentd/lib/fluent/plugin/output.rb:1538:in `flush_thread_run'
E, [2024-10-27T04:01:58.833421 #6300] ERROR -- :    /Users/runner/work/fluentd/fluentd/lib/fluent/plugin/output.rb:510:in `block (2 levels) in start'
E, [2024-10-27T04:01:58.833439 #6300] ERROR -- :    /Users/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Watson1978 commented 2 hours ago

The some tests has flush_at_shutdown false. https://github.com/fluent/fluentd/blob/f493f25d0d33da8e16bab59d7856105788cdc7a5/test/plugin/test_out_forward.rb#L733

This test also seems to be unnecessary to be true. So it may be better to set it to false to make the test more stable.