search

dadoonet / fscrawler

Elasticsearch File System Crawler (FS Crawler)
https://fscrawler.readthedocs.io/
Apache License 2.0
1.34k stars 297 forks source link

Can't connect to my local elasticsearch #1456

Open yossiarieli532 opened 2 years ago

yossiarieli532 commented 2 years ago

Describe the bug

Followed the described procedure of importing my custom ssl certificate. Received a successful import message. But still receiving the same authentication error:

08:29:02,448 WARN [f.p.e.c.f.c.ElasticsearchClient] Failed to create elasticsearch client on Elasticsearch{nodes=[https://127.0.0.1:9220], index='try1', indexFolder='try1_folder', bulkSize=100, flushInterval=5s, byteSize=10mb, username='import', pipeline='null', pathPrefix='null', sslVerification='false'}. Message: Can not execute GET https://127.0.0.1:9220/ : PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.

Job Settings

---
name: "try1"
fs:
  url: "/tmp/es"
  update_rate: "0"
  excludes:
  - "*/~*"
  json_support: false
  filename_as_id: false
  add_filesize: true
  remove_deleted: true
  add_as_inner_object: false
  store_source: false
  index_content: true
  attributes_support: false
  raw_metadata: false
  xml_support: false
  index_folders: true
  lang_detect: false
  continue_on_error: false
  ocr:
    language: "eng"
    enabled: true
    pdf_strategy: "ocr_and_text"
  follow_symlinks: false
elasticsearch:
  nodes:
  - url: "https://127.0.0.1:9220"
  ssl_verification: false
  bulk_size: 100
  flush_interval: "5s"
  byte_size: "10mb"
  username: "USERNAME"
  password: "PASSWORD"

Logs

08:29:02,041 INFO  [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
08:29:02,041 INFO  [f.p.e.c.f.FsCrawlerImpl] FS crawler started in watch mode. It will run unless you stop it with CTRL+C.
08:29:02,448 WARN  [f.p.e.c.f.c.ElasticsearchClient] Failed to create elasticsearch client on Elasticsearch{nodes=[https://127.0.0.1:9220], index='try1', indexFolder='try1_folder', bulkSize=100, flushInterval=5s, byteSize=10mb, username='logstash_import', pipeline='null', pathPrefix='null', sslVerification='false'}. Message: Can not execute GET https://127.0.0.1:9220/ : PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.
08:29:02,448 FATAL [f.p.e.c.f.c.FsCrawlerCli] We can not start Elasticsearch Client. Exiting.
fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClientException: Can not execute GET https://127.0.0.1:9220/ : PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.httpCall(ElasticsearchClient.java:756) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.httpGet(ElasticsearchClient.java:709) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.getVersion(ElasticsearchClient.java:168) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.start(ElasticsearchClient.java:131) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.service.FsCrawlerManagementServiceElasticsearchImpl.start(FsCrawlerManagementServiceElasticsearchImpl.java:65) ~[fscrawler-core-2.10-SNAPSHOT.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.FsCrawlerImpl.start(FsCrawlerImpl.java:116) ~[fscrawler-core-2.10-SNAPSHOT.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.startEsClient(FsCrawlerCli.java:322) [fscrawler-cli-2.10-SNAPSHOT.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.main(FsCrawlerCli.java:298) [fscrawler-cli-2.10-SNAPSHOT.jar:?]
Caused by: jakarta.ws.rs.ProcessingException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:269) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:675) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-3.0.4.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-3.0.4.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-3.0.4.jar:?]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:674) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:422) ~[jersey-client-3.0.4.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.httpCall(ElasticsearchClient.java:730) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
    ... 7 more
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
    at sun.security.ssl.TransportContext.fatal(TransportContext.java:353) ~[?:?]
    at sun.security.ssl.TransportContext.fatal(TransportContext.java:296) ~[?:?]
    at sun.security.ssl.TransportContext.fatal(TransportContext.java:291) ~[?:?]
    at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357) ~[?:?]
    at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) ~[?:?]
    at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
    at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
    at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) ~[?:?]
    at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421) ~[?:?]
    at sun.security.ssl.TransportContext.dispatch(TransportContext.java:183) ~[?:?]
    at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
    at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506) ~[?:?]
    at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1416) ~[?:?]
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456) ~[?:?]
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427) ~[?:?]
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:572) ~[?:?]
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197) ~[?:?]
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592) ~[?:?]
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520) ~[?:?]
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527) ~[?:?]
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334) ~[?:?]
    at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:378) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:267) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:675) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-3.0.4.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-3.0.4.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-3.0.4.jar:?]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:674) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:422) ~[jersey-client-3.0.4.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.httpCall(ElasticsearchClient.java:730) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
    ... 7 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) ~[?:?]
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
    at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313) ~[?:?]
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222) ~[?:?]
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129) ~[?:?]
    at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1341) ~[?:?]
    at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) ~[?:?]
    at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
    at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
    at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) ~[?:?]
    at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421) ~[?:?]
    at sun.security.ssl.TransportContext.dispatch(TransportContext.java:183) ~[?:?]
    at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
    at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506) ~[?:?]
    at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1416) ~[?:?]
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456) ~[?:?]
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427) ~[?:?]
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:572) ~[?:?]
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197) ~[?:?]
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592) ~[?:?]
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520) ~[?:?]
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527) ~[?:?]
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334) ~[?:?]
    at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:378) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:267) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:675) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-3.0.4.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-3.0.4.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-3.0.4.jar:?]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:674) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:422) ~[jersey-client-3.0.4.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.httpCall(ElasticsearchClient.java:730) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
    ... 7 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:?]
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:?]
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) ~[?:?]
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ~[?:?]
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
    at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313) ~[?:?]
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222) ~[?:?]
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129) ~[?:?]
    at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1341) ~[?:?]
    at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) ~[?:?]
    at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
    at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
    at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) ~[?:?]
    at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421) ~[?:?]
    at sun.security.ssl.TransportContext.dispatch(TransportContext.java:183) ~[?:?]
    at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
    at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506) ~[?:?]
    at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1416) ~[?:?]
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456) ~[?:?]
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427) ~[?:?]
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:572) ~[?:?]
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197) ~[?:?]
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592) ~[?:?]
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520) ~[?:?]
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527) ~[?:?]
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334) ~[?:?]
    at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:378) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:267) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:675) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-3.0.4.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-3.0.4.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-3.0.4.jar:?]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:674) ~[jersey-client-3.0.4.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:422) ~[jersey-client-3.0.4.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.httpCall(ElasticsearchClient.java:730) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
    ... 7 more
08:29:02,454 INFO  [f.p.e.c.f.FsCrawlerImpl] FS crawler [try1] stopped
08:29:02,455 INFO  [f.p.e.c.f.FsCrawlerImpl] FS crawler [try1] stopped

Versions:

dadoonet commented 2 years ago

Could you share exactly what you did? As you mentioned a custom certificate, I guess it's in Elasticsearch, right? Or Is that related to https://fscrawler.readthedocs.io/en/latest/admin/fs/elasticsearch.html#ssl-configuration?

Is there a way for me to reproduce that issue?