Ignoring nodes with non-default cluster name?

rfoltyns / log4j2-elasticsearch

Log4j2 Elasticsearch Appender plugins

Apache License 2.0

174 stars 46 forks source link

Ignoring nodes with non-default cluster name? #42

Closed wierdoki closed 4 years ago

wierdoki commented 4 years ago

Description I have a "None of the configured nodes are available" thrown at org.appenders.log4j2.elasticsearch.bulkprocessor.BulkProcessorObjectFactory line 90. After digging for a while, I realized that in BulkProcessorObjectFactory.createClient(), after client.addTransportAddress(..), client still has zero node but one filterred node, and during putTemplate(), "this.ensureNodesAreAvailable(nodes)"throw this exception.

rfoltyns commented 4 years ago

Could you post your configuration, ES version and exact bulkprocessor module that you used? Otherwise I can only guess here and I see two possibilities:

TransportClient rejects the node you configured for some reason (maybe the cluster name you mentioned in above)
BulkProcessor uses ES TransportClient and it's based on TCP, so maybe your config points at HTTP port (9200 by default) instead of TCP (9300 by default)?

wierdoki commented 4 years ago

My es cluster version is 6.1.4, and the bulkprocessor I'm using is log4j2-elasticsearch6-bulkprocessor, and it's 9300.

rfoltyns commented 4 years ago

Could you post the ES config and log4j2 config please? I'd like to reproduce this one.

However, it look like it might be a limitation within the TransportClient which would be out of scope of this project.

Would you consider switching to jest or hc module? Both are backwards-compatible with ES 6.x.

wierdoki commented 4 years ago

believe me, my config is quite ordinary and u can reproduce this by changing your es client cluster name. And I think u can fix this by like allowing us to pass cluster.name to InsecureTransportClientProvider.createClient or SecureClientProvider.createClient. For example, InsecureTransportClientProvider only createClient by "return new PreBuiltTransportClient(Settings.Builder.EMPTY_SETTINGS, Collections.EMPTY_LIST); “ in es6-bulkprocessor-1.4.1 for now.

rfoltyns commented 4 years ago

That makes sense now. I'll add support for it in 1.4.2.

rfoltyns commented 4 years ago

1.4.2 is released. You can now define all client settings in all bulkprocessor modules with:

<ClientSettings>
    <ClientSetting name="setting.name" value="${sys/env/map/customLookup:value}" />
</ClientSettings>

Thank you for this feature request! 👍 I've completely missed it. If you have any other suggestions, feel free to open more issues or PRs