rfoltyns / log4j2-elasticsearch

Log4j2 Elasticsearch Appender plugins
Apache License 2.0
174 stars 46 forks source link

Ignoring nodes with non-default cluster name? #42

Closed wierdoki closed 4 years ago

wierdoki commented 4 years ago

Description I have a "None of the configured nodes are available" thrown at org.appenders.log4j2.elasticsearch.bulkprocessor.BulkProcessorObjectFactory line 90. After digging for a while, I realized that in BulkProcessorObjectFactory.createClient(), after client.addTransportAddress(..), client still has zero node but one filterred node, and during putTemplate(), "this.ensureNodesAreAvailable(nodes)"throw this exception.

rfoltyns commented 4 years ago

Could you post your configuration, ES version and exact bulkprocessor module that you used? Otherwise I can only guess here and I see two possibilities:

wierdoki commented 4 years ago

My es cluster version is 6.1.4, and the bulkprocessor I'm using is log4j2-elasticsearch6-bulkprocessor, and it's 9300.

rfoltyns commented 4 years ago

Could you post the ES config and log4j2 config please? I'd like to reproduce this one.

However, it look like it might be a limitation within the TransportClient which would be out of scope of this project.

Would you consider switching to jest or hc module? Both are backwards-compatible with ES 6.x.

wierdoki commented 4 years ago

believe me, my config is quite ordinary and u can reproduce this by changing your es client cluster name. And I think u can fix this by like allowing us to pass cluster.name to InsecureTransportClientProvider.createClient or SecureClientProvider.createClient. For example, InsecureTransportClientProvider only createClient by "return new PreBuiltTransportClient(Settings.Builder.EMPTY_SETTINGS, Collections.EMPTY_LIST); “ in es6-bulkprocessor-1.4.1 for now.

rfoltyns commented 4 years ago

That makes sense now. I'll add support for it in 1.4.2.

rfoltyns commented 4 years ago

1.4.2 is released. You can now define all client settings in all bulkprocessor modules with:

<ClientSettings>
    <ClientSetting name="setting.name" value="${sys/env/map/customLookup:value}" />
</ClientSettings>

Thank you for this feature request! 👍 I've completely missed it. If you have any other suggestions, feel free to open more issues or PRs