Unable to import csv using Kibana -> ML via Kibana connected with Elasticsearch

[byteshiva]

Kibana version: 7.9.2

Elasticsearch version: 7.9.2

Server OS version: Ubunu 20.4

$uname -a 
Linux  5.4.0-49-generic #53-Ubuntu SMP Fri Sep 18 09:54:57 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Browser version: Brave Version 1.14.84 Chromium: 85.0.4183.121 (Official Build) (64-bit)

Browser OS version: Version 1.14.84 Chromium: 85.0.4183.121 (Official Build) (64-bit)

Original install method (e.g. download page, yum, from source, etc.):

• https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
• https://www.elastic.co/guide/en/kibana/current/docker.html

Describe the bug: Unable to push csv via kibana portal.

Steps to reproduce: Click machine learning and import csv file.

Simple Data csv import

cat ~/airlinesample.csv 
time,airline,responsetime
2014-06-23 00:00:00Z,AAL,132.2046
2014-06-23 00:00:00Z,JZA,990.4628
2014-06-23 00:00:00Z,JBU,877.5927
2014-06-23 00:00:00Z,KLM,1355.4812

Expected behavior:

Screenshots (if relevant):

Errors in browser console (if relevant):

core.entry.js:34 POST http://localhost:5601/api/ml/file_data_visualizer/import 500 (Internal Server Error)
core.entry.js:84 Detected an unhandled Promise rejection.
Error: Internal Server Error
core.entry.js:28 Uncaught (in promise) Error: Internal Server Error
    at Fetch._callee3$ (core.entry.js:34)
    at l (kbn-ui-shared-deps.js:368)
    at Generator._invoke (kbn-ui-shared-deps.js:368)
    at Generator.forEach.e.<computed> [as next] (kbn-ui-shared-deps.js:368)
    at fetch_asyncGeneratorStep (core.entry.js:34)
    at _next (core.entry.js:34)


HTTP response

{statusCode: 500, error: "Internal Server Error",…}
error: "Internal Server Error"
message: "index [.kibana_1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];: [cluster_block_exception] index [.kibana_1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"
statusCode: 500

Provide logs and/or server output (if relevant):

Kibana logs

{"type":"error","@timestamp":"2020-09-26T10:52:32Z","tags":[],"pid":6,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:132:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:86:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:81:17)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:162:34)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/api/ml/file_data_visualizer/import","path":"/api/ml/file_data_visualizer/import","href":"/api/ml/file_data_visualizer/import"},"message":"Internal Server Error"}
{"type":"response","@timestamp":"2020-09-26T10:52:32Z","tags":["access:ml:canFindFileStructure"],"pid":6,"method":"post","statusCode":500,"req":{"url":"/api/ml/file_data_visualizer/import","method":"post","headers":{"host":"localhost:5601","connection":"keep-alive","content-length":"1342","kbn-version":"7.9.2","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","kbn-system-request":"true","content-type":"application/json","accept":"*/*","origin":"http://localhost:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"http://localhost:5601/app/ml","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"172.17.0.1","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","referer":"http://localhost:5601/app/ml"},"res":{"statusCode":500,"responseTime":55,"contentLength":9},"message":"POST /api/ml/file_data_visualizer/import 500 55ms - 9.0B"}
{"type":"log","@timestamp":"2020-09-26T10:52:59Z","tags":["error","plugins","taskManager","taskManager"],"pid":6,"message":"Failed to poll for work: Error: Request Timeout after 30000ms"}
{"type":"response","@timestamp":"2020-09-26T10:53:00Z","tags":[],"pid":6,"method":"post","statusCode":200,"req":{"url":"/api/ui_metric/report","method":"post","headers":{"host":"localhost:5601","connection":"keep-alive","content-length":"337","kbn-version":"7.9.2","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://localhost:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"http://localhost:5601/app/ml","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"172.17.0.1","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","referer":"http://localhost:5601/app/ml"},"res":{"statusCode":200,"responseTime":7,"contentLength":9},"message":"POST /api/ui_metric/report 200 7ms - 9.0B"}
{"type":"response","@timestamp":"2020-09-26T10:53:08Z","tags":[],"pid":6,"method":"post","statusCode":200,"req":{"url":"/api/ui_metric/report","method":"post","headers":{"host":"localhost:5601","connection":"keep-alive","content-length":"337","kbn-version":"7.9.2","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://localhost:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"http://localhost:5601/app/ml","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"172.17.0.1","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","referer":"http://localhost:5601/app/ml"},"res":{"statusCode":200,"responseTime":7,"contentLength":9},"message":"POST /api/ui_metric/report 200 7ms - 9.0B"}
{"type":"response","@timestamp":"2020-09-26T10:53:19Z","tags":[],"pid":6,"method":"post","statusCode":200,"req":{"url":"/api/ui_metric/report","method":"post","headers":{"host":"localhost:5601","connection":"keep-alive","content-length":"337","kbn-version":"7.9.2","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://localhost:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"http://localhost:5601/app/ml","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"172.17.0.1","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","referer":"http://localhost:5601/app/ml"},"res":{"statusCode":200,"responseTime":10,"contentLength":9},"message":"POST /api/ui_metric/report 200 10ms - 9.0B"}
{"type":"log","@timestamp":"2020-09-26T10:53:29Z","tags":["error","plugins","taskManager","taskManager"],"pid":6,"message":"Failed to poll for work: Error: Request Timeout after 30000ms"}
{"type":"response","@timestamp":"2020-09-26T10:53:39Z","tags":[],"pid":6,"method":"post","statusCode":200,"req":{"url":"/api/ui_metric/report","method":"post","headers":{"host":"localhost:5601","connection":"keep-alive","content-length":"337","kbn-version":"7.9.2","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://localhost:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"http://localhost:5601/app/ml","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"172.17.0.1","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","referer":"http://localhost:5601/app/ml"},"res":{"statusCode":200,"responseTime":12,"contentLength":9},"message":"POST /api/ui_metric/report 200 12ms - 9.0B"}
{"type":"log","@timestamp":"2020-09-26T10:53:59Z","tags":["error","plugins","taskManager","taskManager"],"pid":6,"message":"Failed to poll for work: Error: Request Timeout after 30000ms"}
{"type":"response","@timestamp":"2020-09-26T10:54:19Z","tags":[],"pid":6,"method":"post","statusCode":200,"req":{"url":"/api/ui_metric/report","method":"post","headers":{"host":"localhost:5601","connection":"keep-alive","content-length":"103","kbn-version":"7.9.2","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://localhost:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"http://localhost:5601/app/ml","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"172.17.0.1","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","referer":"http://localhost:5601/app/ml"},"res":{"statusCode":200,"responseTime":13,"contentLength":9},"message":"POST /api/ui_metric/report 200 13ms - 9.0B"}
{"type":"log","@timestamp":"2020-09-26T10:54:29Z","tags":["error","plugins","taskManager","taskManager"],"pid":6,"message":"Failed to poll for work: Error: Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2020-09-26T10:54:59Z","tags":["error","plugins","taskManager","taskManager"],"pid":6,"message":"Failed to poll for work: Error: Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2020-09-26T10:55:29Z","tags":["error","plugins","taskManager","taskManager"],"pid":6,"message":"Failed to poll for work: Error: Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2020-09-26T10:55:59Z","tags":["error","plugins","taskManager","taskManager"],"pid":6,"message":"Failed to poll for work: Error: Request Timeout after 30000ms"}
{"type":"response","@timestamp":"2020-09-26T10:56:18Z","tags":[],"pid":6,"method":"post","statusCode":200,"req":{"url":"/api/ui_metric/report","method":"post","headers":{"host":"localhost:5601","connection":"keep-alive","content-length":"115","kbn-version":"7.9.2","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://localhost:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"http://localhost:5601/app/ml","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"172.17.0.1","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","referer":"http://localhost:5601/app/ml"},"res":{"statusCode":200,"responseTime":7,"contentLength":9},"message":"POST /api/ui_metric/report 200 7ms - 9.0B"}
{"type":"log","@timestamp":"2020-09-26T10:56:29Z","tags":["error","plugins","taskManager","taskManager"],"pid":6,"message":"Failed to poll for work: Error: Request Timeout after 30000ms"}

Kibana Logs

{"type":"error","@timestamp":"2020-09-26T11:10:28Z","tags":[],"pid":6,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:132:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:86:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:81:17)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:162:34)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/api/ml/file_data_visualizer/import","path":"/api/ml/file_data_visualizer/import","href":"/api/ml/file_data_visualizer/import"},"message":"Internal Server Error"}

Any additional context:

ElasticSearch Logs

{"type": "server", "timestamp": "2020-09-26T11:15:18,764Z", "level": "WARN", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "docker-cluster", "node.name": "5911ac36931f", "message": "flood stage disk watermark [95%] exceeded on [6PKzGpxKTn6vnJBfpfvTNw][5911ac36931f][/usr/share/elasticsearch/data/nodes/0] free: 17.9gb[4.2%], all indices on this node will be marked read-only", "cluster.uuid": "PTTa4AlnQguohxSgeUnXvg", "node.id": "6PKzGpxKTn6vnJBfpfvTNw"  }

[byteshiva]

Searched Net, found the below article. https://techoverflow.net/2019/04/17/how-to-disable-elasticsearch-disk-quota-watermark/

Maybe its because of my disk usage. I'll clear up some space and try the query again.

curl -XGET "http://localhost:9200/_cat/allocation?v&pretty"
shards disk.indices disk.used disk.avail disk.total disk.percent host       ip         node
     6       10.4mb   407.7gb     17.7gb    425.5gb           95 172.17.0.2 172.17.0.2 6834ab86412e

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[azasypkin]

Maybe its because of my disk usage. I'll clear up some space and try the query again.

Yeah, sounds like it, please report back if you manage to resolve your issue so that we can close or investigate it further.

[droberts195]

I agree the fundamental problem here is lack of disk space causing all ES indices to become read-only.

The one thing we should try to improve is the error reporting. It would be better if the underlying ES exception was displayed instead of "Internal Server Error".

[byteshiva]

Cool! it worked for me. After clearing the HDD space, I'm able to upload CSV data via Kibana->ML

$curl -XGET "http://localhost:9200/_cat/allocation?v&pretty"
shards disk.indices disk.used disk.avail disk.total disk.percent host       ip         node
     8       11.1mb     376gb     49.4gb    425.5gb           88 172.11.0.3 172.11.0.3 6834ab86412e
     2                                                                                 UNASSIGNED

Some sample working CSV uploads

[peteharverson]

We'll do some investigation to see if the error handling can be improved on the UI side to display the underlying ES exception here.