TL;DR: It would be nice to be able to pass headers to Elasticsearch
Description
When trying out the changes from #92 I provisioned an Elasticsearch instance with auth enabled and it did not work as expected. Since there are no way to add custom headers I tried to use the URL basic authentication instead but it did not work.
Values I tried:
https://elastic.company.internal:9200 - "failed: [security_exception] missing authentication credentials for REST request"
Stacktrace
```txt
ERROR [2024-04-23 13:24:42,129] com.scienceminer.glutton.indexing.ElasticSearchIndexer: Create index failed
! java.net.UnknownHostException: elastic:password@elastic-1.company.internal: invalid IPv6 address literal
! at java.base/java.net.InetAddress.invalidIPv6LiteralException(InetAddress.java:1391)
! at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1361)
! at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1307)
! at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
! at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalAddressResolver.resolveRemoteAddress(PoolingNHttpClientConnectionManager.java:666)
! at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalAddressResolver.resolveRemoteAddress(PoolingNHttpClientConnectionManager.java:637)
! at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:474)
! at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280)
! at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295)
! at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:381)
! at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:130)
! at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.execute(CloseableHttpAsyncClientBase.java:116)
! at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:138)
! at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
! ... 12 common frames omitted
! Causing: java.io.IOException: elastic:password@elastic-1.company.internal: invalid IPv6 address literal
! at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:939)
! at org.elasticsearch.client.RestClient.performRequest(RestClient.java:304)
! at org.elasticsearch.client.RestClient.performRequest(RestClient.java:292)
! at co.elastic.clients.transport.rest_client.RestClientHttpClient.performRequest(RestClientHttpClient.java:91)
! at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:144)
! at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.create(ElasticsearchIndicesClient.java:266)
! at com.scienceminer.glutton.indexing.ElasticSearchIndexer.createIndex(ElasticSearchIndexer.java:123)
! at com.scienceminer.glutton.indexing.ElasticSearchIndexer.setupIndex(ElasticSearchIndexer.java:338)
! at com.scienceminer.glutton.command.LoadCrossrefCommand.run(LoadCrossrefCommand.java:94)
! at com.scienceminer.glutton.command.LoadCrossrefCommand.run(LoadCrossrefCommand.java:50)
! at io.dropwizard.core.cli.ConfiguredCommand.run(ConfiguredCommand.java:98)
! at io.dropwizard.core.cli.Cli.run(Cli.java:78)
! at io.dropwizard.core.Application.run(Application.java:94)
! at com.scienceminer.glutton.web.LookupServiceApplication.main(LookupServiceApplication.java:200)
```
Proposed solution
Add possibility to add custom headers via the Elastic field in the config or add fields specific for authentication. Would be extra nice if credentials could be loaded from env variables.
TL;DR: It would be nice to be able to pass headers to Elasticsearch
Description
When trying out the changes from #92 I provisioned an Elasticsearch instance with auth enabled and it did not work as expected. Since there are no way to add custom headers I tried to use the URL basic authentication instead but it did not work.
Values I tried:
https://elastic.company.internal:9200- "failed: [security_exception] missing authentication credentials for REST request"https://username:password@elastic.company.internal:9200- "invalid IPv6 address literal"username:password@elastic.company.internal:9200- "invalid IPv6 address literal"Stacktrace
```txt ERROR [2024-04-23 13:24:42,129] com.scienceminer.glutton.indexing.ElasticSearchIndexer: Create index failed ! java.net.UnknownHostException: elastic:password@elastic-1.company.internal: invalid IPv6 address literal ! at java.base/java.net.InetAddress.invalidIPv6LiteralException(InetAddress.java:1391) ! at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1361) ! at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1307) ! at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45) ! at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalAddressResolver.resolveRemoteAddress(PoolingNHttpClientConnectionManager.java:666) ! at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalAddressResolver.resolveRemoteAddress(PoolingNHttpClientConnectionManager.java:637) ! at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:474) ! at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280) ! at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295) ! at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:381) ! at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:130) ! at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.execute(CloseableHttpAsyncClientBase.java:116) ! at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:138) ! at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300) ! ... 12 common frames omitted ! Causing: java.io.IOException: elastic:password@elastic-1.company.internal: invalid IPv6 address literal ! at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:939) ! at org.elasticsearch.client.RestClient.performRequest(RestClient.java:304) ! at org.elasticsearch.client.RestClient.performRequest(RestClient.java:292) ! at co.elastic.clients.transport.rest_client.RestClientHttpClient.performRequest(RestClientHttpClient.java:91) ! at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:144) ! at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.create(ElasticsearchIndicesClient.java:266) ! at com.scienceminer.glutton.indexing.ElasticSearchIndexer.createIndex(ElasticSearchIndexer.java:123) ! at com.scienceminer.glutton.indexing.ElasticSearchIndexer.setupIndex(ElasticSearchIndexer.java:338) ! at com.scienceminer.glutton.command.LoadCrossrefCommand.run(LoadCrossrefCommand.java:94) ! at com.scienceminer.glutton.command.LoadCrossrefCommand.run(LoadCrossrefCommand.java:50) ! at io.dropwizard.core.cli.ConfiguredCommand.run(ConfiguredCommand.java:98) ! at io.dropwizard.core.cli.Cli.run(Cli.java:78) ! at io.dropwizard.core.Application.run(Application.java:94) ! at com.scienceminer.glutton.web.LookupServiceApplication.main(LookupServiceApplication.java:200) ```Proposed solution
Add possibility to add custom headers via the Elastic field in the config or add fields specific for authentication. Would be extra nice if credentials could be loaded from env variables.