Closed liyichao closed 9 years ago
same error after upgrading from rc28 to rc31 any data could be inserted
May be related to #2493 ( I havent enabled graphite)
Same error using Graphite input with inability to create/query databases after upgrade from rc30 to rc31.
A netstat on server seem connection still open
see same problem as #2310.
This makes influxdb totally unusable, and it makes me confused why it is only added to milestone 0.9.1
This seems to be a problem unrelated to the Graphite input plugin. We're working on fixes for the CQ errors and this should be coming soon. Moved to 0.9.0
Hi, I'm encoutering this issue myself making it impossible to use continuous queries (InfluxDB is not that useful for my use case without them..). I'd like to know if fixing this issue is a priority for you guys ? Thanks for your time
It's a priority. On the 0.9.0 milestone
Nice to hear that. Looking forward to be able to use these CQs so that I can finally finish the project I'm working on. Keep up the good work and thanks for the quick reply !
I am facing what seems to be the same problem. I think that #2666 may be related, besides the others already mentioned here.
Some notes on my configuration:
What I think is happening is that all http endpoints are, for some reason, unresponsive. I think that what is happening is that something is "locking" the database access.
With this 'lock' hold for sometime, and clients trying to reach the endpoints, the connections that are being opened and being kept without a response may reach the open files limit and everything really stops working. At least that's what happened in my case.
My clients (the ones that are writing and reading) are using the InfluxDbClient
. I think they would also get a "use of closed connection" if they had timeouts as is the case for the CQ broker when he tries to send the data/process_continuous_queries
endpoint. I think the error is a timeout based on this https://github.com/golang/go/issues/9405. What I see in the client is that they give no error but don't get any response. Neither for writes nor for reads, until the file limit is reached when everything starts erroring.
I tried finding the problem but couldn't get anywhere. The issues pointed in the beginning are saying that a downgrade may solve the problem but I don't think so (as some commenters also have pointed). This is probably related to a race/deadlock condition, that is hard to identify.
A simple process restart has solved the problem for me and everything is working since then.
Experts from my log is shown below.
Reading and writing in a slow pace:
[http] 2015/05/26 03:20:55 127.0.0.1 - - [26/May/2015:03:20:55 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 5db832e7-036f-11e5-9033-000000000000 3.8687ms
[http] 2015/05/26 03:21:08 127.0.0.1 - - [26/May/2015:03:21:08 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 65675c0f-036f-11e5-9041-000000000000 19.589242ms
[http] 2015/05/26 03:21:08 127.0.0.1 - - [26/May/2015:03:21:08 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 656a8aa5-036f-11e5-9043-000000000000 9.176669ms
[http] 2015/05/26 03:21:08 127.0.0.1 - - [26/May/2015:03:21:08 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27tf2%27+and+model%3D%27cabe%C3%A7a%27+and+po%3D%27LQ00131015%27+and+number%3D%27101121%27 HTTP/1.1 200 560 - tsuchiyama 6568eabc-036f-11e5-9042-000000000000 50.706557ms
[http] 2015/05/26 03:21:08 127.0.0.1 - - [26/May/2015:03:21:08 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27tf2%27+and+model%3D%27centro%27+and+po%3D%27LQ00131015%27+and+number%3D%27101121%27 HTTP/1.1 200 560 - tsuchiyama 65717ca5-036f-11e5-9044-000000000000 31.721039ms
[http] 2015/05/26 03:21:08 127.0.0.1 - - [26/May/2015:03:21:08 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27tf2%27+and+model%3D%27p%C3%A9%27+and+po%3D%27LQ00131015%27+and+number%3D%27101121%27 HTTP/1.1 200 563 - tsuchiyama 657a866f-036f-11e5-9045-000000000000 30.219171ms
[http] 2015/05/26 03:21:08 127.0.0.1 - - [26/May/2015:03:21:08 -0300] GET /query?db=sofia&q=select+mean%28tmp01%29%2Cmean%28stp01%29%2Cmean%28tmp02%29%2Cmean%28stp02%29%2Cmean%28tmp03%29%2Cmean%28stp03%29%2Cmean%28tmp04%29%2Cmean%28stp04%29%2Cmean%28tmp05%29%2Cmean%28stp05%29%2Cmean%28tmp06%29%2Cmean%28stp06%29%2Cmean%28tmp07%29%2Cmean%28stp07%29%2Cmean%28tmp08%29%2Cmean%28stp08%29%2Cmean%28tmp09%29%2Cmean%28stp09%29%2Cmean%28tmp10%29%2Cmean%28stp10%29%2Cmean%28tmp11%29%2Cmean%28stp11%29%2Cmean%28tmp12%29%2Cmean%28stp12%29%2Cmean%28tmp13%29%2Cmean%28stp13%29%2Cmean%28tmp14%29%2Cmean%28stp14%29%2Cmean%28tmp15%29%2Cmean%28stp15%29%2Cmean%28tmp16%29%2Cmean%28stp16%29%2Cmean%28tmp17%29%2Cmean%28stp17%29%2Cmean%28tmp18%29%2Cmean%28stp18%29%2Cmean%28tmp19%29%2Cmean%28stp19%29%2Cmean%28tmp20%29%2Cmean%28stp20%29+from+controldata+where+furnace%3D%27tf2%27+and+time+%3E%3D+%272015-05-26+05%3A41%3A57.823272%27+and+time+%3C%3D+%272015-05-26+06%3A21%3A07.785733%27%3Bselect+mean%28stp01%29%2Cmean%28stp02%29%2Cmean%28stp03%29%2Cmean%28stp04%29%2Cmean%28stp05%29%2Cmean%28stp06%29%2Cmean%28stp07%29%2Cmean%28stp08%29%2Cmean%28stp09%29%2Cmean%28stp10%29%2Cmean%28stp11%29%2Cmean%28stp12%29%2Cmean%28stp13%29%2Cmean%28stp14%29%2Cmean%28stp15%29%2Cmean%28stp16%29%2Cmean%28stp17%29%2Cmean%28stp18%29%2Cmean%28stp19%29%2Cmean%28stp20%29+from+sofiacontroldata+where+furnace%3D%27tf2%27+and+time+%3E%3D+%272015-05-26+05%3A41%3A57.823272%27+and+time+%3C%3D+%272015-05-26+06%3A21%3A07.785733%27 HTTP/1.1 200 217 - sofiacoordinator 658750ff-036f-11e5-9046-000000000000 7.379076ms
[http] 2015/05/26 03:21:13 127.0.0.1 - - [26/May/2015:03:21:13 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 689ed0e5-036f-11e5-904d-000000000000 11.147108ms
[http] 2015/05/26 03:21:13 127.0.0.1 - - [26/May/2015:03:21:13 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 68962e2d-036f-11e5-904c-000000000000 131.194399ms
[http] 2015/05/26 03:21:13 127.0.0.1 - - [26/May/2015:03:21:13 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 68ae7c19-036f-11e5-904e-000000000000 168.97809ms
[http] 2015/05/26 03:21:14 127.0.0.1 - - [26/May/2015:03:21:13 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 68cade1a-036f-11e5-904f-000000000000 167.516261ms
[http] 2015/05/26 03:21:43 127.0.0.1 - - [26/May/2015:03:21:43 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 7a749981-036f-11e5-906e-000000000000 21.315987ms
[http] 2015/05/26 03:21:43 127.0.0.1 - - [26/May/2015:03:21:43 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 7a7d00e9-036f-11e5-906f-000000000000 21.724745ms
[http] 2015/05/26 03:21:43 127.0.0.1 - - [26/May/2015:03:21:43 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 7a83f9ca-036f-11e5-9070-000000000000 25.79228ms
[http] 2015/05/26 03:21:46 127.0.0.1 - - [26/May/2015:03:21:46 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27tf1%27+and+model%3D%27cabe%C3%A7a%27+and+po%3D%27LQ00131015%27+and+number%3D%27402121%27 HTTP/1.1 200 647 - tsuchiyama 7bf32aa3-036f-11e5-9073-000000000000 53.470337ms
[http] 2015/05/26 03:21:46 127.0.0.1 - - [26/May/2015:03:21:46 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27tf1%27+and+model%3D%27centro%27+and+po%3D%27LQ00131015%27+and+number%3D%27402121%27 HTTP/1.1 200 642 - tsuchiyama 7bfbf3a3-036f-11e5-9074-000000000000 132.490316ms
[http] 2015/05/26 03:21:46 127.0.0.1 - - [26/May/2015:03:21:46 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27tf1%27+and+model%3D%27p%C3%A9%27+and+po%3D%27LQ00131015%27+and+number%3D%27402121%27 HTTP/1.1 200 658 - tsuchiyama 7c119ba8-036f-11e5-9076-000000000000 14.689583ms
[http] 2015/05/26 03:21:46 127.0.0.1 - - [26/May/2015:03:21:46 -0300] GET /query?db=sofia&q=select+mean%28tmp01%29%2Cmean%28stp01%29%2Cmean%28tmp02%29%2Cmean%28stp02%29%2Cmean%28tmp03%29%2Cmean%28stp03%29%2Cmean%28tmp04%29%2Cmean%28stp04%29%2Cmean%28tmp05%29%2Cmean%28stp05%29%2Cmean%28tmp06%29%2Cmean%28stp06%29%2Cmean%28tmp07%29%2Cmean%28stp07%29%2Cmean%28tmp08%29%2Cmean%28stp08%29%2Cmean%28tmp09%29%2Cmean%28stp09%29%2Cmean%28tmp10%29%2Cmean%28stp10%29%2Cmean%28tmp11%29%2Cmean%28stp11%29%2Cmean%28tmp12%29%2Cmean%28stp12%29%2Cmean%28tmp13%29%2Cmean%28stp13%29%2Cmean%28tmp14%29%2Cmean%28stp14%29%2Cmean%28tmp15%29%2Cmean%28stp15%29%2Cmean%28tmp16%29%2Cmean%28stp16%29%2Cmean%28tmp17%29%2Cmean%28stp17%29%2Cmean%28tmp18%29%2Cmean%28stp18%29%2Cmean%28tmp19%29%2Cmean%28stp19%29%2Cmean%28tmp20%29%2Cmean%28stp20%29+from+controldata+where+furnace%3D%27tf1%27+and+time+%3E%3D+%272015-05-26+05%3A42%3A15.111147%27+and+time+%3C%3D+%272015-05-26+06%3A21%3A45.476457%27%3Bselect+mean%28stp01%29%2Cmean%28stp02%29%2Cmean%28stp03%29%2Cmean%28stp04%29%2Cmean%28stp05%29%2Cmean%28stp06%29%2Cmean%28stp07%29%2Cmean%28stp08%29%2Cmean%28stp09%29%2Cmean%28stp10%29%2Cmean%28stp11%29%2Cmean%28stp12%29%2Cmean%28stp13%29%2Cmean%28stp14%29%2Cmean%28stp15%29%2Cmean%28stp16%29%2Cmean%28stp17%29%2Cmean%28stp18%29%2Cmean%28stp19%29%2Cmean%28stp20%29+from+sofiacontroldata+where+furnace%3D%27tf1%27+and+time+%3E%3D+%272015-05-26+05%3A42%3A15.111147%27+and+time+%3C%3D+%272015-05-26+06%3A21%3A45.476457%27 HTTP/1.1 200 304 - sofiacoordinator 7c1a7140-036f-11e5-9077-000000000000 3.194096ms
[http] 2015/05/26 03:21:55 127.0.0.1 - - [26/May/2015:03:21:55 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 81d9d31e-036f-11e5-9081-000000000000 7.359322ms
[http] 2015/05/26 03:22:00 127.0.0.1 - - [26/May/2015:03:22:00 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27hf%27+and+model%3D%27cabe%C3%A7a%27+and+po%3D%27LQR0070410%27+and+number%3D%27201125%27 HTTP/1.1 200 612 - tsuchiyama 8470cba6-036f-11e5-9087-000000000000 34.719337ms
[http] 2015/05/26 03:22:00 127.0.0.1 - - [26/May/2015:03:22:00 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27hf%27+and+model%3D%27centro%27+and+po%3D%27LQR0070410%27+and+number%3D%27201125%27 HTTP/1.1 200 617 - tsuchiyama 84769206-036f-11e5-9088-000000000000 24.034959ms
[http] 2015/05/26 03:22:00 127.0.0.1 - - [26/May/2015:03:22:00 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27hf%27+and+model%3D%27p%C3%A9%27+and+po%3D%27LQR0070410%27+and+number%3D%27201125%27 HTTP/1.1 200 607 - tsuchiyama 847acb7c-036f-11e5-9089-000000000000 23.946391ms
[http] 2015/05/26 03:22:00 127.0.0.1 - - [26/May/2015:03:22:00 -0300] GET /query?db=sofia&q=select+mean%28tmp01%29%2Cmean%28stp01%29%2Cmean%28tmp02%29%2Cmean%28stp02%29%2Cmean%28tmp03%29%2Cmean%28stp03%29%2Cmean%28tmp04%29%2Cmean%28stp04%29%2Cmean%28tmp05%29%2Cmean%28stp05%29%2Cmean%28tmp06%29%2Cmean%28stp06%29%2Cmean%28tmp07%29%2Cmean%28stp07%29%2Cmean%28tmp08%29%2Cmean%28stp08%29%2Cmean%28tmp09%29%2Cmean%28stp09%29%2Cmean%28tmp10%29%2Cmean%28stp10%29%2Cmean%28tmp11%29%2Cmean%28stp11%29%2Cmean%28tmp12%29%2Cmean%28stp12%29%2Cmean%28tmp13%29%2Cmean%28stp13%29%2Cmean%28tmp14%29%2Cmean%28stp14%29%2Cmean%28tmp15%29%2Cmean%28stp15%29%2Cmean%28tmp16%29%2Cmean%28stp16%29%2Cmean%28tmp17%29%2Cmean%28stp17%29%2Cmean%28tmp18%29%2Cmean%28stp18%29%2Cmean%28tmp19%29%2Cmean%28stp19%29%2Cmean%28tmp20%29%2Cmean%28stp20%29%2Cmean%28tmp21%29%2Cmean%28stp21%29%2Cmean%28tmp22%29%2Cmean%28stp22%29%2Cmean%28tmp23%29%2Cmean%28stp23%29%2Cmean%28tmp24%29%2Cmean%28stp24%29%2Cmean%28tmp25%29%2Cmean%28stp25%29%2Cmean%28tmp26%29%2Cmean%28stp26%29%2Cmean%28tmp27%29%2Cmean%28stp27%29%2Cmean%28tmp28%29%2Cmean%28stp28%29%2Cmean%28tmp29%29%2Cmean%28stp29%29%2Cmean%28tmp30%29%2Cmean%28stp30%29+from+controldata+where+furnace%3D%27hf%27+and+time+%3E%3D+%272015-05-26+05%3A44%3A29.480889%27+and+time+%3C%3D+%272015-05-26+06%3A21%3A59.862359%27%3Bselect+mean%28stp01%29%2Cmean%28stp02%29%2Cmean%28stp03%29%2Cmean%28stp04%29%2Cmean%28stp05%29%2Cmean%28stp06%29%2Cmean%28stp07%29%2Cmean%28stp08%29%2Cmean%28stp09%29%2Cmean%28stp10%29%2Cmean%28stp11%29%2Cmean%28stp12%29%2Cmean%28stp13%29%2Cmean%28stp14%29%2Cmean%28stp15%29%2Cmean%28stp16%29%2Cmean%28stp17%29%2Cmean%28stp18%29%2Cmean%28stp19%29%2Cmean%28stp20%29%2Cmean%28stp21%29%2Cmean%28stp22%29%2Cmean%28stp23%29%2Cmean%28stp24%29%2Cmean%28stp25%29%2Cmean%28stp26%29%2Cmean%28stp27%29%2Cmean%28stp28%29%2Cmean%28stp29%29%2Cmean%28stp30%29+from+sofiacontroldata+where+furnace%3D%27hf%27+and+time+%3E%3D+%272015-05-26+05%3A44%3A29.480889%27+and+time+%3C%3D+%272015-05-26+06%3A21%3A59.862359%27 HTTP/1.1 200 459 - sofiacoordinator 8482f7bd-036f-11e5-908a-000000000000 24.761985ms
[http] 2015/05/26 03:22:10 127.0.0.1 - - [26/May/2015:03:22:10 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 8a404a61-036f-11e5-9094-000000000000 9.684853ms
[http] 2015/05/26 03:22:10 127.0.0.1 - - [26/May/2015:03:22:10 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 8a423764-036f-11e5-9095-000000000000 8.821489ms
[http] 2015/05/26 03:22:13 127.0.0.1 - - [26/May/2015:03:22:13 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 8c552445-036f-11e5-909a-000000000000 120.120043ms
[http] 2015/05/26 03:22:13 127.0.0.1 - - [26/May/2015:03:22:13 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 8c6b28eb-036f-11e5-909b-000000000000 18.938366ms
[http] 2015/05/26 03:22:13 127.0.0.1 - - [26/May/2015:03:22:13 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 8c71be5b-036f-11e5-909c-000000000000 231.91294ms
[http] 2015/05/26 03:22:24 127.0.0.1 - - [26/May/2015:03:22:24 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27tf2%27+and+model%3D%27cabe%C3%A7a%27+and+po%3D%27LQ00131015%27+and+number%3D%27303123%27 HTTP/1.1 200 569 - tsuchiyama 930b6338-036f-11e5-90a8-000000000000 51.783054ms
[http] 2015/05/26 03:22:24 127.0.0.1 - - [26/May/2015:03:22:24 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27tf2%27+and+model%3D%27centro%27+and+po%3D%27LQ00131015%27+and+number%3D%27303123%27 HTTP/1.1 200 560 - tsuchiyama 931417ac-036f-11e5-90a9-000000000000 48.635901ms
[http] 2015/05/26 03:22:25 127.0.0.1 - - [26/May/2015:03:22:24 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27tf2%27+and+model%3D%27p%C3%A9%27+and+po%3D%27LQ00131015%27+and+number%3D%27303123%27 HTTP/1.1 200 557 - tsuchiyama 931cab08-036f-11e5-90aa-000000000000 167.409427ms
[http] 2015/05/26 03:22:25 127.0.0.1 - - [26/May/2015:03:22:25 -0300] GET /query?db=sofia&q=select+mean%28tmp01%29%2Cmean%28stp01%29%2Cmean%28tmp02%29%2Cmean%28stp02%29%2Cmean%28tmp03%29%2Cmean%28stp03%29%2Cmean%28tmp04%29%2Cmean%28stp04%29%2Cmean%28tmp05%29%2Cmean%28stp05%29%2Cmean%28tmp06%29%2Cmean%28stp06%29%2Cmean%28tmp07%29%2Cmean%28stp07%29%2Cmean%28tmp08%29%2Cmean%28stp08%29%2Cmean%28tmp09%29%2Cmean%28stp09%29%2Cmean%28tmp10%29%2Cmean%28stp10%29%2Cmean%28tmp11%29%2Cmean%28stp11%29%2Cmean%28tmp12%29%2Cmean%28stp12%29%2Cmean%28tmp13%29%2Cmean%28stp13%29%2Cmean%28tmp14%29%2Cmean%28stp14%29%2Cmean%28tmp15%29%2Cmean%28stp15%29%2Cmean%28tmp16%29%2Cmean%28stp16%29%2Cmean%28tmp17%29%2Cmean%28stp17%29%2Cmean%28tmp18%29%2Cmean%28stp18%29%2Cmean%28tmp19%29%2Cmean%28stp19%29%2Cmean%28tmp20%29%2Cmean%28stp20%29+from+controldata+where+furnace%3D%27tf2%27+and+time+%3E%3D+%272015-05-26+05%3A43%3A14.197471%27+and+time+%3C%3D+%272015-05-26+06%3A22%3A24.3132%27%3Bselect+mean%28stp01%29%2Cmean%28stp02%29%2Cmean%28stp03%29%2Cmean%28stp04%29%2Cmean%28stp05%29%2Cmean%28stp06%29%2Cmean%28stp07%29%2Cmean%28stp08%29%2Cmean%28stp09%29%2Cmean%28stp10%29%2Cmean%28stp11%29%2Cmean%28stp12%29%2Cmean%28stp13%29%2Cmean%28stp14%29%2Cmean%28stp15%29%2Cmean%28stp16%29%2Cmean%28stp17%29%2Cmean%28stp18%29%2Cmean%28stp19%29%2Cmean%28stp20%29+from+sofiacontroldata+where+furnace%3D%27tf2%27+and+time+%3E%3D+%272015-05-26+05%3A43%3A14.197471%27+and+time+%3C%3D+%272015-05-26+06%3A22%3A24.3132%27 HTTP/1.1 200 228 - sofiacoordinator 9341b362-036f-11e5-90ab-000000000000 5.431863ms
[http] 2015/05/26 03:22:41 127.0.0.1 - - [26/May/2015:03:22:41 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 9cb8f2c5-036f-11e5-90bc-000000000000 7.192156ms
[http] 2015/05/26 03:22:43 127.0.0.1 - - [26/May/2015:03:22:43 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 9e36cecd-036f-11e5-90c0-000000000000 26.17291ms
Pre create shard starts (I don't know if related with the problem but in the same second as the last write):
[srvr] 2015/05/26 03:22:43 shard group pre-create check commencing
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 1, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 2, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 3, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 4, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 5, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 6, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 7, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 8, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 9, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 10, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 11, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 12, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 13, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 14, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 16, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 17, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 18, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 19, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 20, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 21, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 22, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 23, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 24, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 25, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 26, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 27, retention policy raw, database sofia
No more response for the broker CQ request and no more loggs of writes or reads from my clients (and I know they are doing it):
[srvr] 2015/05/26 03:22:45 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:22:47 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:22:49 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:22:51 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:22:53 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:22:55 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:22:57 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:22:59 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:02 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:04 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[broker] 2015/05/26 03:23:05 executing truncation check for 28 topic(s)
[srvr] 2015/05/26 03:23:06 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:08 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:08 retention policy enforcement check commencing
[srvr] 2015/05/26 03:23:10 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:12 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:14 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:16 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:18 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:21 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:23 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:25 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:27 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:29 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:31 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:33 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:35 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:37 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:39 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:42 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:44 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:46 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:48 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:50 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:52 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:54 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:56 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:58 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:00 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:03 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:05 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:07 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:09 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:11 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:13 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:15 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:17 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:19 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:21 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:24 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:26 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:28 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:30 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:32 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:34 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:36 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:38 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:40 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:42 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:45 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:47 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:49 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:51 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:53 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
Then after a log of the previous error, the process reach the file limit and nothing else works:
[srvr] 2015/05/26 03:50:27 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:27 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:50:29 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:29 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:50:30 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:30 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 10ms
[srvr] 2015/05/26 03:50:31 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:31 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:50:32 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:32 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 10ms
[srvr] 2015/05/26 03:50:33 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:33 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:50:34 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:35 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:35 message stream error: next segment: read segments: open /var/influxdb/raft/22: too many open files
[srvr] 2015/05/26 03:50:35 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 10ms
[messaging] 2015/05/26 03:50:35 connected to broker: http://127.0.0.1:8086/messaging/messages?index=22562052&streaming=true&topicID=22
[srvr] 2015/05/26 03:50:36 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:50:36 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:36 message stream error: next segment: read segments: open /var/influxdb/raft/7: too many open files
[srvr] 2015/05/26 03:50:36 message stream error: next segment: read segments: open /var/influxdb/raft/0: too many open files
[srvr] 2015/05/26 03:50:36 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:36 message stream error: segment by index: read segments: open /var/influxdb/raft/0: too many open files
[messaging] 2015/05/26 03:50:36 connected to broker: http://127.0.0.1:8086/messaging/messages?index=34232436&streaming=true&topicID=0
[messaging] 2015/05/26 03:50:36 connected to broker: http://127.0.0.1:8086/messaging/messages?index=2994728&streaming=true&topicID=7
[messaging] 2015/05/26 03:50:36 connected to broker: http://127.0.0.1:8086/messaging/messages?index=34232436&streaming=true&topicID=0
[srvr] 2015/05/26 03:50:37 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:37 message stream error: next segment: read segments: open /var/influxdb/raft/17: too many open files
[srvr] 2015/05/26 03:50:37 message stream error: next segment: read segments: open /var/influxdb/raft/9: too many open files
[srvr] 2015/05/26 03:50:37 message stream error: next segment: read segments: open /var/influxdb/raft/9: too many open files
[srvr] 2015/05/26 03:50:37 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:37 message stream error: next segment: read segments: open /var/influxdb/raft/4: too many open files
[messaging] 2015/05/26 03:50:37 connected to broker: http://127.0.0.1:8086/messaging/messages?index=4652311&streaming=true&topicID=9
[srvr] 2015/05/26 03:50:37 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:37 message stream error: next segment: read segments: open /var/influxdb/raft/7: too many open files
[messaging] 2015/05/26 03:50:37 connected to broker: http://127.0.0.1:8086/messaging/messages?index=13403886&streaming=true&topicID=17
[srvr] 2015/05/26 03:50:37 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:37 message stream error: next segment: read segments: open /var/influxdb/raft/9: too many open files
[messaging] 2015/05/26 03:50:37 connected to broker: http://127.0.0.1:8086/messaging/messages?index=4652311&streaming=true&topicID=9
[messaging] 2015/05/26 03:50:37 connected to broker: http://127.0.0.1:8086/messaging/messages?index=2994728&streaming=true&topicID=7
[messaging] 2015/05/26 03:50:37 connected to broker: http://127.0.0.1:8086/messaging/messages?index=1150975&streaming=true&topicID=4
[messaging] 2015/05/26 03:50:37 connected to broker: http://127.0.0.1:8086/messaging/messages?index=4652311&streaming=true&topicID=9
[srvr] 2015/05/26 03:50:38 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:50:38 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:39 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
This should be fixed with RC32 later today. I'm going to close it out for now to keep the v0.9.0 milestone clear, but let me know if this reappears.
Recently, I install 0.9.0rc31 (via deb package), with the default config with graphite input, when I inject data in, the influxdb seems not accepting them:
Then there is a lot:
And it seems I can not write data in and now the server does not respond any command issued from influx client including
show databases
my config is below, if that helps: