dadoonet / fscrawler

Elasticsearch File System Crawler (FS Crawler)
https://fscrawler.readthedocs.io/
Apache License 2.0
1.34k stars 297 forks source link

Caused by: jakarta.ws.rs.NotAllowedException: HTTP 405 Method Not Allowed #1908

Closed 19157681683 closed 1 month ago

19157681683 commented 1 month ago

Describe the bug

When I use docker to start elasticsearch/kibana/fscrawler, elasticsearch and kibana start normally, but fscrawler reports an exception

Job Settings

name: "job_name"
fs:
  url: "/home/zucc/桌面/project/c2_superComputer_li_education_rag/rag/elasticsearch/test_file"
  update_rate: "15m"
  excludes:
  - "*/~*"
  json_support: false
  filename_as_id: false
  add_filesize: true
  remove_deleted: true
  add_as_inner_object: false
  store_source: false
  index_content: true
  attributes_support: false
  raw_metadata: false
  xml_support: false
  index_folders: true
  lang_detect: false
  continue_on_error: false
  ocr:
    language: "eng"
    enabled: false
    pdf_strategy: "ocr_and_text"
  follow_symlinks: false
elasticsearch:
  nodes:
  - url: "http://172.17.0.1:9200"
  bulk_size: 100
  flush_interval: "5s"
  byte_size: "10mb"
  ssl_verification: false
  push_templates: true

Logs

11:33:34,689 INFO  [f.p.e.c.f.c.BootstrapChecks] Memory [Free/Total=Percent]: HEAP [622.8mb/9.7gb=6.21%], RAM [26.1gb/39.1gb=66.68%], Swap [1.9gb/1.9gb=100.0%].
11:33:34,861 INFO  [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
11:33:34,862 INFO  [f.p.e.c.f.FsCrawlerImpl] FS crawler started in watch mode. It will run unless you stop it with CTRL+C.
11:33:34,932 WARN  [f.p.e.c.f.c.ElasticsearchClient] We are not doing SSL verification. It's not recommended for production.
SLF4J(W): No SLF4J providers were found.
SLF4J(W): Defaulting to no-operation (NOP) logger implementation
SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
SLF4J(W): Class path contains SLF4J bindings targeting slf4j-api versions 1.7.x or earlier.
SLF4J(W): Ignoring binding found at [jar:file:/usr/share/fscrawler/lib/log4j-slf4j-impl-2.23.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J(W): See https://www.slf4j.org/codes.html#ignoredBindings for an explanation.
11:33:35,239 INFO  [f.p.e.c.f.c.ElasticsearchClient] Elasticsearch Client connected to a node running version 7.6.0
11:33:35,243 WARN  [f.p.e.c.f.c.ElasticsearchClient] We are not doing SSL verification. It's not recommended for production.
11:33:35,276 INFO  [f.p.e.c.f.c.ElasticsearchClient] Elasticsearch Client connected to a node running version 7.6.0
11:33:35,286 FATAL [f.p.e.c.f.c.FsCrawlerCli] We can not start Elasticsearch Client. Exiting.
fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClientException: Error while creating component template fscrawler_alias
    at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.pushComponentTemplate(ElasticsearchClient.java:318) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.loadAndPushComponentTemplate(ElasticsearchClient.java:536) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.createIndexAndComponentTemplates(ElasticsearchClient.java:510) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.service.FsCrawlerDocumentServiceElasticsearchImpl.createSchema(FsCrawlerDocumentServiceElasticsearchImpl.java:71) ~[fscrawler-core-2.10-SNAPSHOT.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.FsCrawlerImpl.start(FsCrawlerImpl.java:110) ~[fscrawler-core-2.10-SNAPSHOT.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.startEsClient(FsCrawlerCli.java:430) [fscrawler-cli-2.10-SNAPSHOT.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.runner(FsCrawlerCli.java:406) [fscrawler-cli-2.10-SNAPSHOT.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.main(FsCrawlerCli.java:137) [fscrawler-cli-2.10-SNAPSHOT.jar:?]
Caused by: jakarta.ws.rs.NotAllowedException: HTTP 405 Method Not Allowed
    at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:968) ~[jersey-client-3.1.7.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:770) ~[jersey-client-3.1.7.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:687) ~[jersey-client-3.1.7.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:709) ~[jersey-client-3.1.7.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:703) ~[jersey-client-3.1.7.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-3.1.7.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-3.1.7.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-3.1.7.jar:?]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:391) ~[jersey-common-3.1.7.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:703) ~[jersey-client-3.1.7.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-3.1.7.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:450) ~[jersey-client-3.1.7.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.httpCall(ElasticsearchClient.java:926) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.httpPut(ElasticsearchClient.java:908) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
    at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.pushComponentTemplate(ElasticsearchClient.java:316) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
    ... 7 more
11:33:35,293 INFO  [f.p.e.c.f.FsCrawlerImpl] FS crawler [job_name] stopped
11:33:35,294 INFO  [f.p.e.c.f.FsCrawlerImpl] FS crawler [job_name] stopped
dadoonet commented 1 month ago

It might be the Elasticsearch version "7.6.0". Could you try with at least 7.17.23?

19157681683 commented 1 month ago

ok, I try again

19157681683 commented 1 month ago

solved,Thank you very much. dadoonet