search

influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
28.9k stars 3.55k forks source link

[0.9.0rc31] broker cq: no data nodes currently available. #2655

Closed liyichao closed 9 years ago

liyichao commented 9 years ago

Recently, I install 0.9.0rc31 (via deb package), with the default config with graphite input, when I inject data in, the influxdb seems not accepting them:

[srvr] 2015/05/26 16:44:27 using configuration at: /etc/opt/influxdb/influxdb.conf
[srvr] 2015/05/26 16:44:27 influxdb started, version 0.9.0-rc31, commit 8ade485b5865e642b128a7cc6eb20c9da73d1b48
[srvr] 2015/05/26 16:44:27 Cluster server listening on [::]:8086
[srvr] 2015/05/26 16:44:27 broker opened at /data/data/influxdb/raft
[raft] 2015/05/26 16:44:27 stopped[0]: log pending: waiting for initialization or join
[raft] 2015/05/26 16:44:27 stopped[1]: changing term: 0 => 1
[raft] 2015/05/26 16:44:27 leader[1]: log initialize: promoted to 'leader' with cluster ID 4319093539205343698, log ID 1, term 1
[srvr] 2015/05/26 16:44:27 initialized broker: http://localhost:8086
[srvr] 2015/05/26 16:44:27 data node(0) opened at /data/data/influxdb/db
[messaging] 2015/05/26 16:44:28 connected to broker: http://localhost:8086/messaging/messages?index=0&streaming=true&topicID=0
[srvr] 2015/05/26 16:44:28 initialized data node: http://localhost:8086
[srvr] 2015/05/26 16:44:28 authentication enabled: false
[srvr] 2015/05/26 16:44:28 broker enforcing retention policies with check interval of 10m0s
[srvr] 2015/05/26 16:44:28 shard group pre-create with check interval of 45m0s
[srvr] 2015/05/26 16:44:28 snapshot server listening on 0.0.0.0:8086
[srvr] 2015/05/26 16:44:28 admin server listening on :8083
[server] 2015/05/26 16:44:28 retention policy 'default' auto-created for database 'host'
[graphite] 2015/05/26 16:44:28 listening on TCP connection [::]:2013
[server] 2015/05/26 16:44:28 retention policy 'default' auto-created for database '_influxdb'
[srvr] 2015/05/26 16:44:28 started self-monitoring at interval of 1m0s
[srvr] 2015/05/26 16:44:28 API server listening on 0.0.0.0:8086
[srvr] 2015/05/26 16:44:28 broker cq: no data nodes currently available.

Then there is a lot:

[srvr] 2015/05/26 17:17:22 received write for database 'host', retention policy '', with 1 points
[srvr] 2015/05/26 17:17:22 received write for database 'host', retention policy '', with 1 points
[srvr] 2015/05/26 17:17:23 broker cq: error hitting data node: http://localhost:8086: Post http://localhost:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 17:17:24 received write for database 'host', retention policy '', with 1 points

And it seems I can not write data in and now the server does not respond any command issued from influx client including show databases

my config is below, if that helps:

# Welcome to the InfluxDB configuration file.

# If hostname (on the OS) doesn't return a name that can be resolved by the other
# systems in the cluster, you'll have to set the hostname to an IP or something
# that can be resolved here.
# hostname = ""
bind-address = "0.0.0.0"

# The default cluster and API port
port = 8086

# Once every 24 hours InfluxDB will report anonymous data to m.influxdb.com
# The data includes raft id (random 8 bytes), os, arch and version
# We don't track ip addresses of servers reporting. This is only used
# to track the number of instances running and the versions, which
# is very helpful for us.
# Change this option to true to disable reporting.
reporting-disabled = false

# Controls settings for initial start-up. Once a node is successfully started,
# these settings are ignored.  If a node is started with the -join flag,
# these settings are ignored.
[initialization]
join-urls = "" # Comma-delimited URLs, in the form http://host:port, for joining another cluster.

# Control authentication
# If not set authetication is DISABLED. Be sure to explicitly set this flag to
# true if you want authentication.
[authentication]
enabled = false

# Configure the admin server
[admin]
enabled = true
port = 8083

# Configure the HTTP API endpoint. All time-series data and queries uses this endpoint.
[api]
# ssl-port = 8087    # SSL support is enabled if you set a port and cert
# ssl-cert = "/path/to/cert.pem"

# Configure the Graphite plugins.
[[graphite]] # 1 or more of these sections may be present.
enabled = true
protocol = "tcp" # Set to "tcp" or "udp"
address = "0.0.0.0" # If not set, is actually set to bind-address.
port = 2013
name-position = "last"
name-separator = "."
database = "host"  # store graphite data in this database

# Configure the collectd input.
[collectd]
enabled = false
#address = "0.0.0.0" # If not set, is actually set to bind-address.
#port = 25827
#database = "collectd_database"
#typesdb = "types.db"

# Configure the OpenTSDB input.
[opentsdb]
enabled = false
#address = "0.0.0.0" # If not set, is actually set to bind-address.
#port = 4242
#database = "opentsdb_database"

# Configure UDP listener for series data.
[udp]
enabled = false
#bind-address = "0.0.0.0"
#port = 4444

# Broker configuration. Brokers are nodes which participate in distributed
# consensus.
[broker]
enabled = true
# Where the Raft logs are stored. The user running InfluxDB will need read/write access.
dir  = "/data/data/influxdb/raft"
truncation-interval = "10m"
max-topic-size = 52428800
max-segment-size = 10485760

# Raft configuration. Controls the distributed consensus system.
[raft]
apply-interval = "10ms"
election-timeout = "5s"
heartbeat-interval = "100ms"
reconnect-timeout = "10ms"

# Data node configuration. Data nodes are where the time-series data, in the form of
# shards, is stored.
[data]
enabled = true
dir = "/data/data/influxdb/db"

# Auto-create a retention policy when a database is created. Defaults to true.
retention-auto-create = true

# Control whether retention policies are enforced and how long the system waits between
# enforcing those policies.
retention-check-enabled = true
retention-check-period = "10m"

# Configuration for snapshot endpoint.
[snapshot]
enabled = true # Enabled by default if not set.

[logging]
write-tracing = true # If true, enables detailed logging of the write system.
raft-tracing = true # If true, enables detailed logging of Raft consensus.
http-access = true # If true, logs each HTTP access to the system.

# InfluxDB can store statistical and diagnostic information about itself. This is useful for
# monitoring purposes. This feature is disabled by default, but if enabled, these data can be
# queried like any other data.
[monitoring]
enabled = true
write-interval = "1m"          # Period between writing the data.
julienmathevet commented 9 years ago

same error after upgrading from rc28 to rc31 any data could be inserted

May be related to #2493 ( I havent enabled graphite)

spynode commented 9 years ago

Same error using Graphite input with inability to create/query databases after upgrade from rc30 to rc31.

julienmathevet commented 9 years ago

A netstat on server seem connection still open

julienmathevet commented 9 years ago

see same problem as #2310.

liyichao commented 9 years ago

This makes influxdb totally unusable, and it makes me confused why it is only added to milestone 0.9.1

pauldix commented 9 years ago

This seems to be a problem unrelated to the Graphite input plugin. We're working on fixes for the CQ errors and this should be coming soon. Moved to 0.9.0

renan-strauss commented 9 years ago

Hi, I'm encoutering this issue myself making it impossible to use continuous queries (InfluxDB is not that useful for my use case without them..). I'd like to know if fixing this issue is a priority for you guys ? Thanks for your time

pauldix commented 9 years ago

It's a priority. On the 0.9.0 milestone

renan-strauss commented 9 years ago

Nice to hear that. Looking forward to be able to use these CQs so that I can finally finish the project I'm working on. Keep up the good work and thanks for the quick reply !

vladlopes commented 9 years ago

I am facing what seems to be the same problem. I think that #2666 may be related, besides the others already mentioned here.

Some notes on my configuration:

What I think is happening is that all http endpoints are, for some reason, unresponsive. I think that what is happening is that something is "locking" the database access.

With this 'lock' hold for sometime, and clients trying to reach the endpoints, the connections that are being opened and being kept without a response may reach the open files limit and everything really stops working. At least that's what happened in my case.

My clients (the ones that are writing and reading) are using the InfluxDbClient. I think they would also get a "use of closed connection" if they had timeouts as is the case for the CQ broker when he tries to send the data/process_continuous_queries endpoint. I think the error is a timeout based on this https://github.com/golang/go/issues/9405. What I see in the client is that they give no error but don't get any response. Neither for writes nor for reads, until the file limit is reached when everything starts erroring.

I tried finding the problem but couldn't get anywhere. The issues pointed in the beginning are saying that a downgrade may solve the problem but I don't think so (as some commenters also have pointed). This is probably related to a race/deadlock condition, that is hard to identify.

A simple process restart has solved the problem for me and everything is working since then.

Experts from my log is shown below.

Reading and writing in a slow pace:

[http] 2015/05/26 03:20:55 127.0.0.1 - - [26/May/2015:03:20:55 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 5db832e7-036f-11e5-9033-000000000000 3.8687ms
[http] 2015/05/26 03:21:08 127.0.0.1 - - [26/May/2015:03:21:08 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 65675c0f-036f-11e5-9041-000000000000 19.589242ms
[http] 2015/05/26 03:21:08 127.0.0.1 - - [26/May/2015:03:21:08 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 656a8aa5-036f-11e5-9043-000000000000 9.176669ms
[http] 2015/05/26 03:21:08 127.0.0.1 - - [26/May/2015:03:21:08 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27tf2%27+and+model%3D%27cabe%C3%A7a%27+and+po%3D%27LQ00131015%27+and+number%3D%27101121%27 HTTP/1.1 200 560 - tsuchiyama 6568eabc-036f-11e5-9042-000000000000 50.706557ms
[http] 2015/05/26 03:21:08 127.0.0.1 - - [26/May/2015:03:21:08 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27tf2%27+and+model%3D%27centro%27+and+po%3D%27LQ00131015%27+and+number%3D%27101121%27 HTTP/1.1 200 560 - tsuchiyama 65717ca5-036f-11e5-9044-000000000000 31.721039ms
[http] 2015/05/26 03:21:08 127.0.0.1 - - [26/May/2015:03:21:08 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27tf2%27+and+model%3D%27p%C3%A9%27+and+po%3D%27LQ00131015%27+and+number%3D%27101121%27 HTTP/1.1 200 563 - tsuchiyama 657a866f-036f-11e5-9045-000000000000 30.219171ms
[http] 2015/05/26 03:21:08 127.0.0.1 - - [26/May/2015:03:21:08 -0300] GET /query?db=sofia&q=select+mean%28tmp01%29%2Cmean%28stp01%29%2Cmean%28tmp02%29%2Cmean%28stp02%29%2Cmean%28tmp03%29%2Cmean%28stp03%29%2Cmean%28tmp04%29%2Cmean%28stp04%29%2Cmean%28tmp05%29%2Cmean%28stp05%29%2Cmean%28tmp06%29%2Cmean%28stp06%29%2Cmean%28tmp07%29%2Cmean%28stp07%29%2Cmean%28tmp08%29%2Cmean%28stp08%29%2Cmean%28tmp09%29%2Cmean%28stp09%29%2Cmean%28tmp10%29%2Cmean%28stp10%29%2Cmean%28tmp11%29%2Cmean%28stp11%29%2Cmean%28tmp12%29%2Cmean%28stp12%29%2Cmean%28tmp13%29%2Cmean%28stp13%29%2Cmean%28tmp14%29%2Cmean%28stp14%29%2Cmean%28tmp15%29%2Cmean%28stp15%29%2Cmean%28tmp16%29%2Cmean%28stp16%29%2Cmean%28tmp17%29%2Cmean%28stp17%29%2Cmean%28tmp18%29%2Cmean%28stp18%29%2Cmean%28tmp19%29%2Cmean%28stp19%29%2Cmean%28tmp20%29%2Cmean%28stp20%29+from+controldata+where+furnace%3D%27tf2%27+and+time+%3E%3D+%272015-05-26+05%3A41%3A57.823272%27+and+time+%3C%3D+%272015-05-26+06%3A21%3A07.785733%27%3Bselect+mean%28stp01%29%2Cmean%28stp02%29%2Cmean%28stp03%29%2Cmean%28stp04%29%2Cmean%28stp05%29%2Cmean%28stp06%29%2Cmean%28stp07%29%2Cmean%28stp08%29%2Cmean%28stp09%29%2Cmean%28stp10%29%2Cmean%28stp11%29%2Cmean%28stp12%29%2Cmean%28stp13%29%2Cmean%28stp14%29%2Cmean%28stp15%29%2Cmean%28stp16%29%2Cmean%28stp17%29%2Cmean%28stp18%29%2Cmean%28stp19%29%2Cmean%28stp20%29+from+sofiacontroldata+where+furnace%3D%27tf2%27+and+time+%3E%3D+%272015-05-26+05%3A41%3A57.823272%27+and+time+%3C%3D+%272015-05-26+06%3A21%3A07.785733%27 HTTP/1.1 200 217 - sofiacoordinator 658750ff-036f-11e5-9046-000000000000 7.379076ms
[http] 2015/05/26 03:21:13 127.0.0.1 - - [26/May/2015:03:21:13 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 689ed0e5-036f-11e5-904d-000000000000 11.147108ms
[http] 2015/05/26 03:21:13 127.0.0.1 - - [26/May/2015:03:21:13 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 68962e2d-036f-11e5-904c-000000000000 131.194399ms
[http] 2015/05/26 03:21:13 127.0.0.1 - - [26/May/2015:03:21:13 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 68ae7c19-036f-11e5-904e-000000000000 168.97809ms
[http] 2015/05/26 03:21:14 127.0.0.1 - - [26/May/2015:03:21:13 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 68cade1a-036f-11e5-904f-000000000000 167.516261ms
[http] 2015/05/26 03:21:43 127.0.0.1 - - [26/May/2015:03:21:43 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 7a749981-036f-11e5-906e-000000000000 21.315987ms
[http] 2015/05/26 03:21:43 127.0.0.1 - - [26/May/2015:03:21:43 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 7a7d00e9-036f-11e5-906f-000000000000 21.724745ms
[http] 2015/05/26 03:21:43 127.0.0.1 - - [26/May/2015:03:21:43 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 7a83f9ca-036f-11e5-9070-000000000000 25.79228ms
[http] 2015/05/26 03:21:46 127.0.0.1 - - [26/May/2015:03:21:46 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27tf1%27+and+model%3D%27cabe%C3%A7a%27+and+po%3D%27LQ00131015%27+and+number%3D%27402121%27 HTTP/1.1 200 647 - tsuchiyama 7bf32aa3-036f-11e5-9073-000000000000 53.470337ms
[http] 2015/05/26 03:21:46 127.0.0.1 - - [26/May/2015:03:21:46 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27tf1%27+and+model%3D%27centro%27+and+po%3D%27LQ00131015%27+and+number%3D%27402121%27 HTTP/1.1 200 642 - tsuchiyama 7bfbf3a3-036f-11e5-9074-000000000000 132.490316ms
[http] 2015/05/26 03:21:46 127.0.0.1 - - [26/May/2015:03:21:46 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27tf1%27+and+model%3D%27p%C3%A9%27+and+po%3D%27LQ00131015%27+and+number%3D%27402121%27 HTTP/1.1 200 658 - tsuchiyama 7c119ba8-036f-11e5-9076-000000000000 14.689583ms
[http] 2015/05/26 03:21:46 127.0.0.1 - - [26/May/2015:03:21:46 -0300] GET /query?db=sofia&q=select+mean%28tmp01%29%2Cmean%28stp01%29%2Cmean%28tmp02%29%2Cmean%28stp02%29%2Cmean%28tmp03%29%2Cmean%28stp03%29%2Cmean%28tmp04%29%2Cmean%28stp04%29%2Cmean%28tmp05%29%2Cmean%28stp05%29%2Cmean%28tmp06%29%2Cmean%28stp06%29%2Cmean%28tmp07%29%2Cmean%28stp07%29%2Cmean%28tmp08%29%2Cmean%28stp08%29%2Cmean%28tmp09%29%2Cmean%28stp09%29%2Cmean%28tmp10%29%2Cmean%28stp10%29%2Cmean%28tmp11%29%2Cmean%28stp11%29%2Cmean%28tmp12%29%2Cmean%28stp12%29%2Cmean%28tmp13%29%2Cmean%28stp13%29%2Cmean%28tmp14%29%2Cmean%28stp14%29%2Cmean%28tmp15%29%2Cmean%28stp15%29%2Cmean%28tmp16%29%2Cmean%28stp16%29%2Cmean%28tmp17%29%2Cmean%28stp17%29%2Cmean%28tmp18%29%2Cmean%28stp18%29%2Cmean%28tmp19%29%2Cmean%28stp19%29%2Cmean%28tmp20%29%2Cmean%28stp20%29+from+controldata+where+furnace%3D%27tf1%27+and+time+%3E%3D+%272015-05-26+05%3A42%3A15.111147%27+and+time+%3C%3D+%272015-05-26+06%3A21%3A45.476457%27%3Bselect+mean%28stp01%29%2Cmean%28stp02%29%2Cmean%28stp03%29%2Cmean%28stp04%29%2Cmean%28stp05%29%2Cmean%28stp06%29%2Cmean%28stp07%29%2Cmean%28stp08%29%2Cmean%28stp09%29%2Cmean%28stp10%29%2Cmean%28stp11%29%2Cmean%28stp12%29%2Cmean%28stp13%29%2Cmean%28stp14%29%2Cmean%28stp15%29%2Cmean%28stp16%29%2Cmean%28stp17%29%2Cmean%28stp18%29%2Cmean%28stp19%29%2Cmean%28stp20%29+from+sofiacontroldata+where+furnace%3D%27tf1%27+and+time+%3E%3D+%272015-05-26+05%3A42%3A15.111147%27+and+time+%3C%3D+%272015-05-26+06%3A21%3A45.476457%27 HTTP/1.1 200 304 - sofiacoordinator 7c1a7140-036f-11e5-9077-000000000000 3.194096ms
[http] 2015/05/26 03:21:55 127.0.0.1 - - [26/May/2015:03:21:55 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 81d9d31e-036f-11e5-9081-000000000000 7.359322ms
[http] 2015/05/26 03:22:00 127.0.0.1 - - [26/May/2015:03:22:00 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27hf%27+and+model%3D%27cabe%C3%A7a%27+and+po%3D%27LQR0070410%27+and+number%3D%27201125%27 HTTP/1.1 200 612 - tsuchiyama 8470cba6-036f-11e5-9087-000000000000 34.719337ms
[http] 2015/05/26 03:22:00 127.0.0.1 - - [26/May/2015:03:22:00 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27hf%27+and+model%3D%27centro%27+and+po%3D%27LQR0070410%27+and+number%3D%27201125%27 HTTP/1.1 200 617 - tsuchiyama 84769206-036f-11e5-9088-000000000000 24.034959ms
[http] 2015/05/26 03:22:00 127.0.0.1 - - [26/May/2015:03:22:00 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27hf%27+and+model%3D%27p%C3%A9%27+and+po%3D%27LQR0070410%27+and+number%3D%27201125%27 HTTP/1.1 200 607 - tsuchiyama 847acb7c-036f-11e5-9089-000000000000 23.946391ms
[http] 2015/05/26 03:22:00 127.0.0.1 - - [26/May/2015:03:22:00 -0300] GET /query?db=sofia&q=select+mean%28tmp01%29%2Cmean%28stp01%29%2Cmean%28tmp02%29%2Cmean%28stp02%29%2Cmean%28tmp03%29%2Cmean%28stp03%29%2Cmean%28tmp04%29%2Cmean%28stp04%29%2Cmean%28tmp05%29%2Cmean%28stp05%29%2Cmean%28tmp06%29%2Cmean%28stp06%29%2Cmean%28tmp07%29%2Cmean%28stp07%29%2Cmean%28tmp08%29%2Cmean%28stp08%29%2Cmean%28tmp09%29%2Cmean%28stp09%29%2Cmean%28tmp10%29%2Cmean%28stp10%29%2Cmean%28tmp11%29%2Cmean%28stp11%29%2Cmean%28tmp12%29%2Cmean%28stp12%29%2Cmean%28tmp13%29%2Cmean%28stp13%29%2Cmean%28tmp14%29%2Cmean%28stp14%29%2Cmean%28tmp15%29%2Cmean%28stp15%29%2Cmean%28tmp16%29%2Cmean%28stp16%29%2Cmean%28tmp17%29%2Cmean%28stp17%29%2Cmean%28tmp18%29%2Cmean%28stp18%29%2Cmean%28tmp19%29%2Cmean%28stp19%29%2Cmean%28tmp20%29%2Cmean%28stp20%29%2Cmean%28tmp21%29%2Cmean%28stp21%29%2Cmean%28tmp22%29%2Cmean%28stp22%29%2Cmean%28tmp23%29%2Cmean%28stp23%29%2Cmean%28tmp24%29%2Cmean%28stp24%29%2Cmean%28tmp25%29%2Cmean%28stp25%29%2Cmean%28tmp26%29%2Cmean%28stp26%29%2Cmean%28tmp27%29%2Cmean%28stp27%29%2Cmean%28tmp28%29%2Cmean%28stp28%29%2Cmean%28tmp29%29%2Cmean%28stp29%29%2Cmean%28tmp30%29%2Cmean%28stp30%29+from+controldata+where+furnace%3D%27hf%27+and+time+%3E%3D+%272015-05-26+05%3A44%3A29.480889%27+and+time+%3C%3D+%272015-05-26+06%3A21%3A59.862359%27%3Bselect+mean%28stp01%29%2Cmean%28stp02%29%2Cmean%28stp03%29%2Cmean%28stp04%29%2Cmean%28stp05%29%2Cmean%28stp06%29%2Cmean%28stp07%29%2Cmean%28stp08%29%2Cmean%28stp09%29%2Cmean%28stp10%29%2Cmean%28stp11%29%2Cmean%28stp12%29%2Cmean%28stp13%29%2Cmean%28stp14%29%2Cmean%28stp15%29%2Cmean%28stp16%29%2Cmean%28stp17%29%2Cmean%28stp18%29%2Cmean%28stp19%29%2Cmean%28stp20%29%2Cmean%28stp21%29%2Cmean%28stp22%29%2Cmean%28stp23%29%2Cmean%28stp24%29%2Cmean%28stp25%29%2Cmean%28stp26%29%2Cmean%28stp27%29%2Cmean%28stp28%29%2Cmean%28stp29%29%2Cmean%28stp30%29+from+sofiacontroldata+where+furnace%3D%27hf%27+and+time+%3E%3D+%272015-05-26+05%3A44%3A29.480889%27+and+time+%3C%3D+%272015-05-26+06%3A21%3A59.862359%27 HTTP/1.1 200 459 - sofiacoordinator 8482f7bd-036f-11e5-908a-000000000000 24.761985ms
[http] 2015/05/26 03:22:10 127.0.0.1 - - [26/May/2015:03:22:10 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 8a404a61-036f-11e5-9094-000000000000 9.684853ms
[http] 2015/05/26 03:22:10 127.0.0.1 - - [26/May/2015:03:22:10 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 8a423764-036f-11e5-9095-000000000000 8.821489ms
[http] 2015/05/26 03:22:13 127.0.0.1 - - [26/May/2015:03:22:13 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 8c552445-036f-11e5-909a-000000000000 120.120043ms
[http] 2015/05/26 03:22:13 127.0.0.1 - - [26/May/2015:03:22:13 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 8c6b28eb-036f-11e5-909b-000000000000 18.938366ms
[http] 2015/05/26 03:22:13 127.0.0.1 - - [26/May/2015:03:22:13 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 8c71be5b-036f-11e5-909c-000000000000 231.91294ms
[http] 2015/05/26 03:22:24 127.0.0.1 - - [26/May/2015:03:22:24 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27tf2%27+and+model%3D%27cabe%C3%A7a%27+and+po%3D%27LQ00131015%27+and+number%3D%27303123%27 HTTP/1.1 200 569 - tsuchiyama 930b6338-036f-11e5-90a8-000000000000 51.783054ms
[http] 2015/05/26 03:22:24 127.0.0.1 - - [26/May/2015:03:22:24 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27tf2%27+and+model%3D%27centro%27+and+po%3D%27LQ00131015%27+and+number%3D%27303123%27 HTTP/1.1 200 560 - tsuchiyama 931417ac-036f-11e5-90a9-000000000000 48.635901ms
[http] 2015/05/26 03:22:25 127.0.0.1 - - [26/May/2015:03:22:24 -0300] GET /query?db=sofia&q=select+current+from+tubes+where+furnace%3D%27tf2%27+and+model%3D%27p%C3%A9%27+and+po%3D%27LQ00131015%27+and+number%3D%27303123%27 HTTP/1.1 200 557 - tsuchiyama 931cab08-036f-11e5-90aa-000000000000 167.409427ms
[http] 2015/05/26 03:22:25 127.0.0.1 - - [26/May/2015:03:22:25 -0300] GET /query?db=sofia&q=select+mean%28tmp01%29%2Cmean%28stp01%29%2Cmean%28tmp02%29%2Cmean%28stp02%29%2Cmean%28tmp03%29%2Cmean%28stp03%29%2Cmean%28tmp04%29%2Cmean%28stp04%29%2Cmean%28tmp05%29%2Cmean%28stp05%29%2Cmean%28tmp06%29%2Cmean%28stp06%29%2Cmean%28tmp07%29%2Cmean%28stp07%29%2Cmean%28tmp08%29%2Cmean%28stp08%29%2Cmean%28tmp09%29%2Cmean%28stp09%29%2Cmean%28tmp10%29%2Cmean%28stp10%29%2Cmean%28tmp11%29%2Cmean%28stp11%29%2Cmean%28tmp12%29%2Cmean%28stp12%29%2Cmean%28tmp13%29%2Cmean%28stp13%29%2Cmean%28tmp14%29%2Cmean%28stp14%29%2Cmean%28tmp15%29%2Cmean%28stp15%29%2Cmean%28tmp16%29%2Cmean%28stp16%29%2Cmean%28tmp17%29%2Cmean%28stp17%29%2Cmean%28tmp18%29%2Cmean%28stp18%29%2Cmean%28tmp19%29%2Cmean%28stp19%29%2Cmean%28tmp20%29%2Cmean%28stp20%29+from+controldata+where+furnace%3D%27tf2%27+and+time+%3E%3D+%272015-05-26+05%3A43%3A14.197471%27+and+time+%3C%3D+%272015-05-26+06%3A22%3A24.3132%27%3Bselect+mean%28stp01%29%2Cmean%28stp02%29%2Cmean%28stp03%29%2Cmean%28stp04%29%2Cmean%28stp05%29%2Cmean%28stp06%29%2Cmean%28stp07%29%2Cmean%28stp08%29%2Cmean%28stp09%29%2Cmean%28stp10%29%2Cmean%28stp11%29%2Cmean%28stp12%29%2Cmean%28stp13%29%2Cmean%28stp14%29%2Cmean%28stp15%29%2Cmean%28stp16%29%2Cmean%28stp17%29%2Cmean%28stp18%29%2Cmean%28stp19%29%2Cmean%28stp20%29+from+sofiacontroldata+where+furnace%3D%27tf2%27+and+time+%3E%3D+%272015-05-26+05%3A43%3A14.197471%27+and+time+%3C%3D+%272015-05-26+06%3A22%3A24.3132%27 HTTP/1.1 200 228 - sofiacoordinator 9341b362-036f-11e5-90ab-000000000000 5.431863ms
[http] 2015/05/26 03:22:41 127.0.0.1 - - [26/May/2015:03:22:41 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 9cb8f2c5-036f-11e5-90bc-000000000000 7.192156ms
[http] 2015/05/26 03:22:43 127.0.0.1 - - [26/May/2015:03:22:43 -0300] POST /write HTTP/1.1 204 0 - sofiatrends 9e36cecd-036f-11e5-90c0-000000000000 26.17291ms

Pre create shard starts (I don't know if related with the problem but in the same second as the last write):

[srvr] 2015/05/26 03:22:43 shard group pre-create check commencing
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 1, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 2, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 3, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 4, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 5, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 6, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 7, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 8, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 9, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 10, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 11, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 12, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 13, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 14, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 16, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 17, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 18, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 19, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 20, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 21, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 22, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 23, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 24, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 25, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 26, retention policy raw, database sofia
[srvr] 2015/05/26 03:22:43 pre-creating shard group for 27, retention policy raw, database sofia

No more response for the broker CQ request and no more loggs of writes or reads from my clients (and I know they are doing it):

[srvr] 2015/05/26 03:22:45 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:22:47 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:22:49 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:22:51 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:22:53 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:22:55 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:22:57 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:22:59 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:02 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:04 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[broker] 2015/05/26 03:23:05 executing truncation check for 28 topic(s)
[srvr] 2015/05/26 03:23:06 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:08 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:08 retention policy enforcement check commencing
[srvr] 2015/05/26 03:23:10 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:12 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:14 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:16 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:18 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:21 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:23 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:25 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:27 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:29 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:31 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:33 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:35 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:37 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:39 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:42 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:44 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:46 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:48 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:50 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:52 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:54 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:56 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:23:58 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:00 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:03 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:05 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:07 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:09 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:11 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:13 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:15 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:17 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:19 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:21 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:24 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:26 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:28 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:30 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:32 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:34 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:36 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:38 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:40 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:42 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:45 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:47 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:49 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:51 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:24:53 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection

Then after a log of the previous error, the process reach the file limit and nothing else works:

[srvr] 2015/05/26 03:50:27 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:27 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:50:29 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:29 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:50:30 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:30 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 10ms
[srvr] 2015/05/26 03:50:31 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:31 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:50:32 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:32 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 10ms
[srvr] 2015/05/26 03:50:33 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:33 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:50:34 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:35 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:35 message stream error: next segment: read segments: open /var/influxdb/raft/22: too many open files
[srvr] 2015/05/26 03:50:35 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 10ms
[messaging] 2015/05/26 03:50:35 connected to broker: http://127.0.0.1:8086/messaging/messages?index=22562052&streaming=true&topicID=22
[srvr] 2015/05/26 03:50:36 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:50:36 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:36 message stream error: next segment: read segments: open /var/influxdb/raft/7: too many open files
[srvr] 2015/05/26 03:50:36 message stream error: next segment: read segments: open /var/influxdb/raft/0: too many open files
[srvr] 2015/05/26 03:50:36 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:36 message stream error: segment by index: read segments: open /var/influxdb/raft/0: too many open files
[messaging] 2015/05/26 03:50:36 connected to broker: http://127.0.0.1:8086/messaging/messages?index=34232436&streaming=true&topicID=0
[messaging] 2015/05/26 03:50:36 connected to broker: http://127.0.0.1:8086/messaging/messages?index=2994728&streaming=true&topicID=7
[messaging] 2015/05/26 03:50:36 connected to broker: http://127.0.0.1:8086/messaging/messages?index=34232436&streaming=true&topicID=0
[srvr] 2015/05/26 03:50:37 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:37 message stream error: next segment: read segments: open /var/influxdb/raft/17: too many open files
[srvr] 2015/05/26 03:50:37 message stream error: next segment: read segments: open /var/influxdb/raft/9: too many open files
[srvr] 2015/05/26 03:50:37 message stream error: next segment: read segments: open /var/influxdb/raft/9: too many open files
[srvr] 2015/05/26 03:50:37 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:37 message stream error: next segment: read segments: open /var/influxdb/raft/4: too many open files
[messaging] 2015/05/26 03:50:37 connected to broker: http://127.0.0.1:8086/messaging/messages?index=4652311&streaming=true&topicID=9
[srvr] 2015/05/26 03:50:37 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:37 message stream error: next segment: read segments: open /var/influxdb/raft/7: too many open files
[messaging] 2015/05/26 03:50:37 connected to broker: http://127.0.0.1:8086/messaging/messages?index=13403886&streaming=true&topicID=17
[srvr] 2015/05/26 03:50:37 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:37 message stream error: next segment: read segments: open /var/influxdb/raft/9: too many open files
[messaging] 2015/05/26 03:50:37 connected to broker: http://127.0.0.1:8086/messaging/messages?index=4652311&streaming=true&topicID=9
[messaging] 2015/05/26 03:50:37 connected to broker: http://127.0.0.1:8086/messaging/messages?index=2994728&streaming=true&topicID=7
[messaging] 2015/05/26 03:50:37 connected to broker: http://127.0.0.1:8086/messaging/messages?index=1150975&streaming=true&topicID=4
[messaging] 2015/05/26 03:50:37 connected to broker: http://127.0.0.1:8086/messaging/messages?index=4652311&streaming=true&topicID=9
[srvr] 2015/05/26 03:50:38 broker cq: error hitting data node: http://127.0.0.1:8086: Post http://127.0.0.1:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/05/26 03:50:38 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
[srvr] 2015/05/26 03:50:39 http: Accept error: accept tcp 127.0.0.1:8086: too many open files; retrying in 5ms
toddboom commented 9 years ago

This should be fixed with RC32 later today. I'm going to close it out for now to keep the v0.9.0 milestone clear, but let me know if this reappears.