Open philippkahr opened 4 years ago
philippkahr
commented
4 years ago So today, I retried adding the destination.geo.location fields and the filebeat seem to function properly. Maybe it had something to do with auditbeat running on the same host? I have no idea on how to reproduce this properly
philippkahr
commented
4 years ago I have filebeat 7.5.0 with netflow module running on a 4core (Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz) 8GB RAM virtual machine. I have a 3 node elasticsearch cluster with 8 cores (Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz) and 8GB RAM (4GB to Java HEAP) and 1TB of SSD. I know that this might be undersized when doing large aggregations but searching is fast enough for us.
As soon as I add that processor to filebeat and restart it. It stops taking in all the flows. We have around ~5000 flows per second. 5000*2 (I am checking destination and source) DNS requests should not be any problem? Since I create such a huge capacity. May it be that it drops all the new messages that it hasn't processed, due to being in the cache for 1 second until DNS mapping has been successful?
- dns:
type: reverse
fields:
server.ip: server.hostname
client.ip: client.hostname
source.ip: source.hostname
destination.ip: destination.hostname
success_cache:
capacity.initial: 1000
capacity.max: 100000
failure_cache:
capacity.initial: 1000
capacity.max: 10000
nameservers: ['internal', 'internal']
timeout: 1s
tag_on_failure: [_dns_reverse_lookup_failed]To tackle this, I thought that the queue_size is way too low and it may max out at some point, so I set it to 2^20, which would be netflow messages for ~2-3 minutes. So as a packet is around 1300 bytes (1,26KBytes) as shown with tcpdump. My queue size if full would need around 1.331.200KBytes (1.300MBytes, ~1,1GBytes) memory, since I have 8GB RAM, that should not be any problem.
- module: netflow
log:
enabled: true
var:
netflow_host: 0.0.0.0
netflow_port: 2055
queue_size: 1048576 #I use an elasticsearch geoip-info pipeline.
{
"geoip-info" : {
"description" : "Add geoip info",
"processors" : [
{
"geoip" : {
"field" : "client.ip",
"target_field" : "client.geo",
"ignore_missing" : true
}
},
{
"geoip" : {
"field" : "source.ip",
"target_field" : "source.geo",
"ignore_missing" : true
}
},
{
"geoip" : {
"field" : "destination.ip",
"target_field" : "destination.geo",
"ignore_missing" : true
}
},
{
"geoip" : {
"field" : "server.ip",
"target_field" : "server.geo",
"ignore_missing" : true
}
},
{
"geoip" : {
"field" : "host.ip",
"target_field" : "host.geo",
"ignore_missing" : true
}
}
]
}
}Here are some screenshots from the index where it is being written too. There is nothing else than this one beat writing to that index, and the beats monitoring tab.
So there is a super weird behavior going on. At some point in time the filebeat processors grows and grows and grows, but never get's rid of the memory. E.g. the behavior shown at ~14:40, where the Filebeat is getting restarted to do memory issues (I guess).
The funny thing is, at the point in time ~15:12, where the Process Total overtakes the GC Next, it stops indexing (marked it with a red circle). Looking at the Discovery page, I cannot see anything going in there, after that point in time, and nothing is shown in Kibana until filebeat restarts. The interesting thing is that Filebeat still says that it collects a lot of documents and has no failure or error rate and that the indexing rate is constant and working properly.
To summarize, I think that there is an issue with the queue and the garbage collector and it has nothing to do with the processor or whatever you use. In my infrastructure, the filebeat usually crashes around ~13 million documents. 13000000*1,2KBytes/1024/1024 = 14,87GB of RAM, I guess with some overhead added, that is around the RAM I have available in the server 7,64GB RAM and 7,87GB swap (total 15,51).
philippkahr
commented
4 years ago Hi again,
I looked into the monitoring tab in Kibana again and it really seems to be some sort of GC / memory leak that breaks a lot of things. Here you go for a full 16-hour view of our monitoring pages. In the normal journalctl log with the info log level, I cannot see anything interesting https://gist.github.com/philippkahr/474488281112cca5aa9cbe8bb9fcd435. What really bugs me is that the message dropped: 9824. Why does it drop that many netflows? The queue is not full, to be honest, I cannot figure this one out.
"input": {
"netflow": {
"flows": 1397,
"packets": {
"dropped": 9824,
"received": 49
}
}{
"monitoring": {
"metrics": {
"beat": {
"cpu": {
"system": {
"ticks": 196390,
"time": {
"ms": 706
}
},
"total": {
"ticks": 423500,
"time": {
"ms": 1508
},
"value": 423500
},
"user": {
"ticks": 227110,
"time": {
"ms": 802
}
}
},
"handles": {
"limit": {
"hard": 4096,
"soft": 1024
},
"open": 21
},
"info": {
"ephemeral_id": "8538121a-efed-498e-9643-039816079aab",
"uptime": {
"ms": 8190039
}
},
"memstats": {
"gc_next": 189570048,
"memory_alloc": 126939600,
"memory_total": 53656353912
},
"runtime": {
"goroutines": 125
}
},
"filebeat": {
"events": {
"active": 23,
"added": 3008,
"done": 2985
},
"harvester": {
"open_files": 2,
"running": 2
},
"input": {
"netflow": {
"flows": 1397,
"packets": {
"dropped": 9824,
"received": 49
}
}
}
},
"libbeat": {
"config": {
"module": {
"running": 0
}
},
"output": {
"events": {
"acked": 2985,
"batches": 75,
"total": 2985
},
"read": {
"bytes": 700151
},
"write": {
"bytes": 4856415
}
},
"pipeline": {
"clients": 17,
"events": {
"active": 26,
"published": 3008,
"total": 3008
},
"queue": {
"acked": 2985
}
}
},
"processor": {
"dns": {
"1": {
"1_0_0_1": {
"failure": 2
},
"1_1_1_1": {
"failure": 1,
"response": {
"ptr": {
"count": 257,
"mean": 17117.39785991609,
"median": 91.5,
"p75": 73537.5,
"stddev": -1873.8704617023468
}
},
"success": 257
},
"cache": {
"hits": 2482,
"misses": 259
}
}
}
},
"registrar": {
"states": {
"current": 13,
"update": 69
},
"writes": {
"success": 11,
"total": 11
}
},
"system": {
"load": {
"1": 0.03,
"15": 0.05,
"5": 0.03,
"norm": {
"1": 0.0038,
"15": 0.0063,
"5": 0.0038
}
}
}
}
}
}Increasing the log level to debug is just spamming everything full, with the JSONs send to elasticsearch, is there any way to say give me debug, without publish?
ct0br0
commented
4 years ago Not sure if it's entirely related but having a similar issue and see there's a ton open already.
We have 1-2 directories with 5-10 json files we collect, sometimes 1 sub directory with lower file count and size of files. Some of these can reach 100s of GB a day and get rotated around UTC midnight.
A box may be fine for a day or few, but eventually filebeat will clog up memory and constantly crash. Virtual memory will jump from 1-2-5---200GB and filebeat will repeatedly die within a few minutes to an hour after the initial crash. The only thing that we find will fix it is a reboot. Until it happens again. This happens on 100-200 machines.
Filebeat (and whole stack) 7.5.1 Oracle Linux 7.7
adriansr
commented
4 years ago @philippkahr, I see you use 7.5.0. Have you tried with version 7.5.2 that was released some weeks ago? Just in case some fixes I added (#15351) cause some improvement, as it's a nasty bug that could create a deadlock and I've seen you've experienced a pause in ingestion at least once. Otherwise, we'll have to investigate what's going on with memory continually increasing, it looks to me that adding extra processors (DNS, geoip) is causing ingestion rates to drop and events from Netflow pile up due to lack of flow control between the Exporter (network device) and Collector (Filebeat). Hopefully this situation will be improved when TCP is available as an input (#14270).
@ct0br0 it's better if you create a new issue as you're not using Netflow.
ct0br0
commented
4 years ago thanks, will do
philippkahr
commented
4 years ago Hi @adriansr thanks for adding more and more features and tweaks. I just waited for the 7.6 release and jumped right into trying. I am still facing the same issue. I started filebeat with the netflow module and without any processors (so no lookups, no DNS queries or whatever, just basic netflow module) at around 09:05 and since then it has reached around 7GB of physical RAM and ~11.5GB of virtual memory within 25 minutes. The swap is growing and growing. The index used by the netflow collector filebeat-7.6.0 is only used by that filebeat. There is nothing else writing to it. The cluster is still the same three node cluster and everything is updated to 7.6.0.
I sadly, cannot provide you with a pcap file as it stores productive data. However, I can give you all the information from the monitoring tab you want to have.
At least, it does not go all the way up to 400 GB of RAM ;).
adriansr
commented
4 years ago @philippkahr can you run filebeat with -httpprof :8888 and, once it's using a high amount of memory, do:
curl 'http://localhost:8888/debug/pprof/heap?gc=1 -o heap.txt'
Then share the heap.txt file? This will help see what is using so much memory.
You can visualize the contents with:
go tool pprof -http :9999 /path/to/filebeat/binary heap.txt
This should pop a browser to visualize the contents of the heap dump.
I fail to see what could possibly be leaking memory. The Netflow input doesn't allocate much memory to operate. Using the same settings as yours and flooding it with netflow packets I get 2k eps and memory usage maxes at ~830MB.
philippkahr
commented
4 years ago Hi @adriansr sure! here you go. I know debugging a memory leak is the worst thing there is. Once filebeat reached ~12gb virtual memory in htop, I started a 15s interval call for curl for around 23 minutes. Maybe that will help you show how it increased over time. I also wrote the physical memory usage for some heaps out. The 12000mb is the moment where the virtual memory took off and "all" physical memory was used, the golden point seems to be around 7400-7800mb.
12000mb-heap.txt 7400mb-goroutine.txt 7400mb-allocs.txt 7400mb-heap.txt 7390mb-heap.txt 7130mb-heap.txt 6570mb-heap.txt 5770mb-heap.txt 4380mb-heap.txt 200mb-heap.txt
I was able to catch it crashing, this is the last heap I was able to pull off successfully. every 15s reached 16000mb virtual heap.zip
I started my 15s interval again when it reached 15.9GB of virtual RAM and looked like this:
I dug into the pprof, but as I am not so experienced in Go, I could not figure out much. However, I found that it seems all memory is "lost" in alloc_space. I especially have the feeling that the MapStr.Clone seems to be unhappy.
philippkahr
commented
4 years ago Sadly the behavior is still the same in 7.8.0
adriansr
commented
4 years ago Hi @philippkahr
Sorry that I forgot to follow up on this for so long.
The alloc_space graph you've attached is misleading because this counts total allocations. Most of that memory has been deallocated. It just indicates that the bulk of allocations are caused by events generated by Netflow, which is expected.
The interesting graph is inuse_space, which shows the currently allocated memory. For 7400mb-allocs.txt file we have:
This indicates that almost all of the allocated memory is UDP packets queued for processing by the netflow input. What I don't understand here is that according to your config, you're using:
max_message_size: 10000 # This was increased from 1000 according to a comment above.
queue_size: 32768This should limit memory usage to ~320MB, not 5GB. Have you increased these values since then? (The UDP input always allocates the max size for a packet before reading).
If that's not the case then something is wrong in our UDP packet queuing.
Also, can you run Filebeat with -d netflow so that we get this stats in the log:
Stats total:[ packets=....
This will tell us the state of the internal packet queue (note to self, export this as a monitoring metric).
All in all, what is happening is that UDP packets are being received faster than they can be ingested, eventually the queue fills up and this leads to a large memory usage and the dropped packets you are observing.
(Also note that there is not a 1:1 relation b/w packets and flows. A single UDP packet can contain a lot of flows, each one of them is a new document to be ingested in ES.)
This leads us to the following problem: 98.5% of the packets are being dropped, while your ingestion rate is about 1000-2000 events per second.
Something is hurting the ingestion rate here and causing backpressure. Can you disable the Cisco ASA temporarily and monitor the number of dropped and queued netflow packets?
philippkahr
commented
4 years ago Hi @adriansr
sure, anything to help. I disabled all the modules and inputs except netflow.
Are we already at the conclusion that the processors in the beats config are the root cause? Maybe, just maybe there is an issue with conflicting templates something like this issue? Elastiflow duplicated index templates Just a wild guess.
I did not have time to wait for it to crash and observe. I just let it run for ~8 minutes and captured the -d netflow flag.
I have exactly 90 add_field. I had to mask the real data thus every IP range points to the same location (a McDonalds in Graz).
here is the heap file. heap.txt
Here is the log file. filebeat.netflow2.log
Here is a monitoring dashboard screenshot
Filebeat config
```yaml
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
setup.template.settings:
index.number_of_shards: 1
#-------- Elasticsearch output ------------------------------
output.elasticsearch:
hosts: ["es1","es2","es3"]
protocol: "https"
username: ...
password: ...
processors:
- add_host_metadata: ~
- dns:
type: reverse
action: append
fields:
server.ip: server.hostname
client.ip: client.hostname
source.ip: source.hostname
destination.ip: destination.hostname
winlog.event_data.IpAddress: winlog.event_data.hostname
success_cache:
capacity.initial: 1000
capacity.max: 100000
failure_cache:
capacity.initial: 1000
capacity.max: 10000
ttl: 5m
nameservers: ['192.168.8.13', '192.168.8.12']
timeout: 5s
tag_on_failure: [_dns_reverse_lookup_failed]
- registered_domain:
field: dns.question.name
target_field: dns.question.registered_domain
ignore_missing: true
ignore_failure: true
- add_fields:
when.network.source.ip: 10.0.0.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.1.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.2.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.3.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.4.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.5.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.6.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.7.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.8.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.9.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.10.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.11.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.12.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.13.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.14.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.15.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.16.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.17.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.18.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.19.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.20.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.21.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.22.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.23.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.24.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.25.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.26.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.27.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.28.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.29.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.30.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.31.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.32.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.33.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.34.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.35.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.36.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.37.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.38.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.39.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.40.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.41.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.42.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.43.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.source.ip: 10.0.44.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.0.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.1.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.2.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.3.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.4.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.5.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.6.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.7.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.8.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.9.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.10.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.11.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.12.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.13.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.14.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.15.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.16.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.17.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.18.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.19.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.20.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.21.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.22.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.23.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.24.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.25.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.26.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.27.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.28.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.29.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.30.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.31.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.32.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.33.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.34.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.35.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.36.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.37.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.38.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.39.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.40.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.41.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.42.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.43.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
- add_fields:
when.network.destination.ip: 10.0.44.0/24
fields:
source.geo.location:
lat: 47.066720
lat: 15.441582
target: ''
setup.kibana.host: "https://kibana:5601"
setup.kibana.ssl.enabled: true
setup.kibana.ssl.certificate_authorities: ["/etc/ssl/certs/cachain.pem"]
setup.kibana.ssl.verification_mode: none
monitoring:
enabled: true
elasticsearch:
username: ...
password: ...
```
philippkahr
commented
4 years ago @adriansr
just for the sake of it, I deployed an elastic cloud trial version and pointed my filebeat to that and let it run for ~10 minutes. The issue is the same, so my elastic servers are not the bottleneck in this case.
filebeat log filebeat.netflow.elasticcloud.log
Elastic cloud heap filebeat heapElasticCloud.txt
Index monitoring tab:
For confirmed bugs, please report:
So I am not quite sure if this will be easy to reproduce. I am writing this issue as I am debugging a weird RAM behavior while tinkering with Netflow. We have around 1000-1500 flows per second, maxing out at 3000, deployed filebeat 7.5.0 with NetFlow module, on a CentOS 7, 8 cores, 8 GB RAM. That filebeat is receiving some ASA events, so there might be a ground noise around 200-500 events per second. My three-node ES cluster with 7.5.0 and Kibana 7.5.0. My ES cluster can handle around 14-16k documents per second on ingest. So backpressure should not be any problem. Somewhere around 14:30 today, CET, we started with the filebeat and NetFlow module, with the following configuration.
Super long config!
``` filebeat.config.modules: # Glob pattern for configuration loading path: ${path.config}/modules.d/*.yml # Set to true to enable config reloading reload.enabled: false # Period on which files under path should be checked for changes #reload.period: 10s setup.template.settings: index.number_of_shards: 1 filebeat.inputs: - type: syslog protocol.udp: host: "0.0.0.0:514" filebeat.modules: - module: netflow log: enabled: true var: netflow_host: 0.0.0.0 netflow_port: 2055 max_message_size: 1000 queue_size: 32768 #-------- Elasticsearch output ------------------------------ output.elasticsearch: hosts: ["es1:9200","es2:9200","es3:9200"] protocol: "https" username: "" password: "" ssl.certificate_authorities: ["/etc/ssl/certs/cachain.pem"] pipeline: geoip-info processors: - add_host_metadata: ~ - add_cloud_metadata: ~ - dns: type: reverse action: append fields: server.ip: server.hostname client.ip: client.hostname source.ip: source.hostname destination.ip: destination.hostname success_cache: capacity.initial: 10000 capacity.max: 100000 failure_cache: capacity.initial: 1000 capacity.max: 10000 ttl: 1m nameservers: ['1.1.1.1', 'someinternal'] timeout: 5s tag_on_failure: [_dns_reverse_lookup_failed] - add_fields: when.network.source.ip: 192.132.0.0/16 fields: source.geo.location: lat: 1.377502 lon: 103.872831 target: '' - add_fields: when.network.source.ip: 192.61.0.0/16 fields: source.geo.location: lat: 51.200045 lon: 4.435278 target: '' - add_fields: when.network.source.ip: 192.151.0.0/16 fields: source.geo.location: lat: 39.923802 lon: 116.415285 target: '' - add_fields: when.network.source.ip: 192.137.0.0/16 fields: source.geo.location: lat: 14.169850 lon: 121.132613 target: '' - add_fields: when.network.source.ip: 192.64.0.0/16 fields: source.geo.location: lat: 52.228267 lon: 0.157753 target: '' - add_fields: when.network.source.ip: 192.23.0.0/16 fields: source.geo.location: lat: 37.321643 lon: -122.032561 target: '' - add_fields: when.network.source.ip: 192.45.0.0/16 fields: source.geo.location: lat: 51.411013 lon: 5.450380 target: '' - add_fields: when.network.source.ip: 192.63.0.0/16 fields: source.geo.location: lat: 32.659898 lon: -16.926293 target: '' - add_fields: when.network.source.ip: 192.139.0.0/16 fields: source.geo.location: lat: 17.434485 lon: 78.387469 target: '' - add_fields: when.network.source.ip: 192.65.0.0/16 fields: source.geo.location: lat: 50.891908 lon: 11.594972 target: '' - add_fields: when.network.source.ip: 192.129.0.0/16 fields: source.geo.location: lat: 1.339557 lon: 103.708979 target: '' - add_fields: when.network.source.ip: 192.36.10.0/24 fields: source.geo.location: lat: 46.101176 lon: 7.074462 target: '' - add_fields: when.network.source.ip: 192.26.0.0/16 fields: source.geo.location: lat: 40.232804 lon: -74.687870 target: '' - add_fields: when.network.source.ip: 192.32.0.0/16 fields: source.geo.location: lat: 45.463651 lon: 9.197683 target: '' - add_fields: when.network.source.ip: 192.40.10.0/24 fields: source.geo.location: lat: 48.111018 lon: 11.450526 target: '' - add_fields: when.network.source.ip: 192.62.0.0/16 fields: source.geo.location: lat: 47.721335 lon: 6.085081 target: '' - add_fields: when.network.source.ip: 192.33.0.0/16 fields: source.geo.location: lat: 45.194482 lon: 9.141818 target: '' - add_fields: when.network.source.ip: 192.31.0.0/16 fields: source.geo.location: lat: 43.684821 lon: 10.487538 target: '' - add_fields: when.network.source.ip: 192.20.0.0/16 fields: source.geo.location: lat: 43.684837 lon: 10.487381 target: '' - add_fields: when.network.source.ip: 192.35.10.0/24 fields: source.geo.location: lat: 50.389876 lon: -4.020370 target: '' - add_fields: when.network.source.ip: 192.11.0.0/16 fields: source.geo.location: lat: 46.975885 lon: 15.394378 target: '' - add_fields: when.network.source.ip: 192.251.0.0/16 fields: source.geo.location: lat: 46.975885 lon: 15.394378 target: '' - add_fields: when.network.source.ip: 192.17.0.0/16 fields: source.geo.location: lat: 46.975885 lon: 15.394378 target: '' - add_fields: when.network.source.ip: 192.10.0.0/16 fields: source.geo.location: lat: 46.975885 lon: 15.394378 target: '' - add_fields: when.network.source.ip: 192.12.0.0/16 fields: source.geo.location: lat: 46.975885 lon: 15.394378 target: '' - add_fields: when.network.source.ip: 192.14.0.0/16 fields: source.geo.location: lat: 46.975885 lon: 15.394378 target: '' - add_fields: when.network.source.ip: 192.16.0.0/16 fields: source.geo.location: lat: 46.975885 lon: 15.394378 target: '' - add_fields: when.network.source.ip: 192.168.0.0/16 fields: source.geo.location: lat: 46.975885 lon: 15.394378 target: '' - add_fields: when.network.source.ip: 192.19.0.0/16 fields: source.geo.location: lat: 46.975885 lon: 15.394378 target: '' - add_fields: when.network.source.ip: 192.34.0.0/16 fields: source.geo.location: lat: 47.224153 lon: 8.817450 target: '' - add_fields: when.network.source.ip: 192.43.0.0/16 fields: source.geo.location: lat: 48.496135 lon: 9.137428 target: '' - add_fields: when.network.source.ip: 192.38.0.0/16 fields: source.geo.location: lat: 47.303424 lon: 8.546351 target: '' - add_fields: when.network.source.ip: 192.27.0.0/16 fields: source.geo.location: lat: 37.385708 lon: -121.972477 target: '' - add_fields: when.network.source.ip: 192.148.0.0/16 fields: source.geo.location: lat: 37.556852 lon: 126.981990 target: '' - add_fields: when.network.source.ip: 192.152.0.0/16 fields: source.geo.location: lat: 31.228960 lon: 121.524242 target: '' - add_fields: when.network.source.ip: 192.154.0.0/16 fields: source.geo.location: lat: 22.534638 lon: 114.054346 target: '' - add_fields: when.network.source.ip: 192.66.0.0/16 fields: source.geo.location: lat: 51.659808 lon: -0.914749 target: '' - add_fields: when.network.source.ip: 192.44.0.0/16 fields: source.geo.location: lat: 49.079522 lon: 8.458652 target: '' - add_fields: when.network.source.ip: 192.153.0.0/16 fields: source.geo.location: lat: 31.298766 lon: 120.585207 target: '' - add_fields: when.network.source.ip: 192.144.0.0/16 fields: source.geo.location: lat: 25.081430 lon: 121.569353 target: '' - add_fields: when.network.source.ip: 192.134.0.0/16 fields: source.geo.location: lat: 1.377595 lon: 103.872874 target: '' - add_fields: when.network.source.ip: 192.135.0.0/16 fields: source.geo.location: lat: 1.377595 lon: 103.872874 target: '' - add_fields: when.network.source.ip: 192.146.0.0/16 fields: source.geo.location: lat: 1.111222 lon: 123.123123 target: '' - add_fields: when.network.source.ip: 192.37.0.0/16 fields: source.geo.location: lat: 39.479420 lon: -0.334650 target: '' - add_fields: when.network.source.ip: 192.41.0.0/16 fields: source.geo.location: lat: 52.326310 lon: 13.626244 target: '' setup.kibana.host: "https://kibana" setup.kibana.ssl.enabled: true setup.kibana.ssl.certificate_authorities: ["/etc/ssl/certs/cachain.pem"] monitoring: enabled: true elasticsearch: username: beats_system password: ```
I sometimes had this message.
So I altered the
max_message_size: 1000tomax_message_size: 10000which worked. The funny thing is, if I watch the beat in monitoring, I am confused with the values I see.Since 14:30, it used some memory. Ok in that graph you can not see it very well due to the peak at ~ 18:00.
If I scale down the time to 14:00 - 14:30, you can see that at 14:18, the first flows started coming in.
If I move mor to 14:10 - 15, filebeat took so much memory around 8gb?
Around 18:00, I changed the configuration to include all the
source.geolookups also to includedestination.geo, so I just copied myadd_fieldsfrom above and added those. As soon as I have done this, and restarted the filebeat, it acted out crazy.Also at the time, I saw some super weird stuff in the
journalctllogs.Super long and weird journalctl
``` Dec 03 18:05:12 my.file.beat.com filebeat[30053]: fatal error: runtime: out of memory Dec 03 18:05:12 my.file.beat.com filebeat[30053]: runtime stack: Dec 03 18:05:12 my.file.beat.com filebeat[30053]: runtime.throw(0x300364b, 0x16) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/panic.go:617 +0x72 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: runtime.sysMap(0xfea4000000, 0x4000000, 0x4e7c5f8) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/mem_linux.go:170 +0xc7 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: runtime.(*mheap).sysAlloc(0x4e62a20, 0x9c4000, 0x4e62a30, 0x4e2) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/malloc.go:633 +0x1cd Dec 03 18:05:12 my.file.beat.com filebeat[30053]: runtime.(*mheap).grow(0x4e62a20, 0x4e2, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/mheap.go:1222 +0x42 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: runtime.(*mheap).allocSpanLocked(0x4e62a20, 0x4e2, 0x4e7c608, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/mheap.go:1150 +0x37f Dec 03 18:05:12 my.file.beat.com filebeat[30053]: runtime.(*mheap).alloc_m(0x4e62a20, 0x4e2, 0x101, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/mheap.go:977 +0xc2 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: runtime.(*mheap).alloc.func1() Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/mheap.go:1048 +0x4c Dec 03 18:05:12 my.file.beat.com filebeat[30053]: runtime.(*mheap).alloc(0x4e62a20, 0x4e2, 0x7f56ad010101, 0x12a60e0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/mheap.go:1047 +0x8a Dec 03 18:05:12 my.file.beat.com filebeat[30053]: runtime.largeAlloc(0x9c4000, 0x101, 0x4c37980) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/malloc.go:1055 +0x99 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: runtime.mallocgc.func1() Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/malloc.go:950 +0x46 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: runtime.systemstack(0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/asm_amd64.s:351 +0x66 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: runtime.mstart() Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/proc.go:1153 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: goroutine 315 [running]: Dec 03 18:05:12 my.file.beat.com filebeat[30053]: runtime.systemstack_switch() Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc00094ee28 sp=0xc00094ee20 pc=0x12db2a0 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: runtime.mallocgc(0x9c4000, 0x2bef200, 0xc00094ee01, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/malloc.go:949 +0x872 fp=0xc00094eec8 sp=0xc00094ee28 pc=0x128b8f2 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: runtime.makeslice(0x2bef200, 0x9c4000, 0x9c4000, 0x33f7200) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/slice.go:49 +0x6c fp=0xc00094eef8 sp=0xc00094eec8 pc=0x12c452c Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/inputsource/udp.(*Server).run(0xc00005b5c0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:98 +0x81 fp=0xc00094efb0 sp=0xc00094eef8 pc=0x25c2f51 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/inputsource/udp.(*Server).Start.func1(0xc00005b5c0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:85 +0x57 fp=0xc00094efd8 sp=0xc00094efb0 pc=0x25c3477 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: runtime.goexit() Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00094efe0 sp=0xc00094efd8 pc=0x12dd371 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/inputsource/udp.(*Server).Start Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:83 +0x172 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: goroutine 1 [chan receive, 2 minutes]: Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/beater.(*signalWait).Wait(...) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/beater/signalwait.go:45 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/beater.(*Filebeat).Run(0xc0008b15a0, 0xc000259d40, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/beater/filebeat.go:410 +0xf13 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/cmd/instance.(*Beat).launch(0xc000259d40, 0x2fe465b, 0x8, 0x2fe465b, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, ...) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:435 +0x60e Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/cmd/instance.Run.func1(0x2fe465b, 0x8, 0x2fe465b, 0x8, 0x0, 0x0, 0xc0007ffc90, 0x30d5cf0, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:187 +0x536 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/cmd/instance.Run(0x2fe465b, 0x8, 0x2fe465b, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:188 +0x13a Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/cmd.genRunCmd.func1(0xc000286000, 0xc0001308f0, 0x0, 0xb) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/cmd/run.go:36 +0x80 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/vendor/github.com/spf13/cobra.(*Command).execute(0xc000286000, 0xc00004e0d0, 0xb, 0xb, 0xc000286000, 0xc00004e0d0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:704 +0x289 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc000286000, 0x2a0d419, 0xc0005e7f88, 0xc0000be058) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:785 +0x2ca Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/vendor/github.com/spf13/cobra.(*Command).Execute(...) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:738 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: main.main() Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/x-pack/filebeat/main.go:22 +0x30 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: goroutine 20 [syscall, 2 minutes]: Dec 03 18:05:12 my.file.beat.com filebeat[30053]: os/signal.signal_recv(0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/sigqueue.go:139 +0x9c Dec 03 18:05:12 my.file.beat.com filebeat[30053]: os/signal.loop() Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/os/signal/signal_unix.go:23 +0x22 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: created by os/signal.init.0 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/os/signal/signal_unix.go:29 +0x41 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: goroutine 21 [chan receive]: Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x4e5ad40) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/vendor/k8s.io/klog/klog.go:1000 +0x8b Dec 03 18:05:12 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/vendor/k8s.io/klog.init.0 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/vendor/k8s.io/klog/klog.go:404 +0x6c Dec 03 18:05:12 my.file.beat.com filebeat[30053]: goroutine 11 [select]: Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/vendor/go.opencensus.io/stats/view.(*worker).start(0xc000050190) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/vendor/go.opencensus.io/stats/view/worker.go:154 +0x100 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/vendor/go.opencensus.io/stats/view.init.0 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/vendor/go.opencensus.io/stats/view/worker.go:32 +0x57 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: goroutine 1127 [select]: Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*persistConn).writeLoop(0xc003d9c6c0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1978 +0x113 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: created by net/http.(*Transport).dialConn Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1360 +0xb0d Dec 03 18:05:12 my.file.beat.com filebeat[30053]: goroutine 497 [IO wait]: Dec 03 18:05:12 my.file.beat.com filebeat[30053]: internal/poll.runtime_pollWait(0x7f56b8d96dd8, 0x72, 0xffffffffffffffff) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/netpoll.go:182 +0x56 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).wait(0xc00bcb1618, 0x72, 0xe700, 0xe78e, 0xffffffffffffffff) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b Dec 03 18:05:12 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).waitRead(...) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: internal/poll.(*FD).Read(0xc00bcb1600, 0xc1dbc56000, 0xe78e, 0xe78e, 0x0, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net.(*netFD).Read(0xc00bcb1600, 0xc1dbc56000, 0xe78e, 0xe78e, 0x203000, 0x0, 0xe781) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/fd_unix.go:202 +0x4f Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net.(*conn).Read(0xc00018a110, 0xc1dbc56000, 0xe78e, 0xe78e, 0x0, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/net.go:177 +0x69 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: crypto/tls.(*atLeastReader).Read(0xc000780900, 0xc1dbc56000, 0xe78e, 0xe78e, 0x40, 0x2534, 0xc003d618d8) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:761 +0x60 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: bytes.(*Buffer).ReadFrom(0xc000325758, 0x33cb6c0, 0xc000780900, 0x1289475, 0x2cf3640, 0x2f21b60) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/bytes/buffer.go:207 +0xbd Dec 03 18:05:12 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readFromUntil(0xc000325500, 0x33cfae0, 0xc00018a110, 0x5, 0xc00018a110, 0x40) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readRecordOrCCS(0xc000325500, 0x30da000, 0xc000325638, 0xc003d61b38) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:590 +0x125 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readRecord(...) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:558 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).Read(0xc000325500, 0xc00fd5a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs/transport.(*statsConn).Read(0xc00fcc3140, 0xc00fd5a000, 0x1000, 0x1000, 0xc003d61c50, 0x1284b8c, 0xc00fc01980) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/transport/stats.go:44 +0x55 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*persistConn).Read(0xc00bdab8c0, 0xc00fd5a000, 0x1000, 0x1000, 0xc003d61c88, 0x1284665, 0xc00fc01980) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1526 +0x7b Dec 03 18:05:12 my.file.beat.com filebeat[30053]: bufio.(*Reader).fill(0xc0007a6a20) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/bufio/bufio.go:100 +0x10f Dec 03 18:05:12 my.file.beat.com filebeat[30053]: bufio.(*Reader).Peek(0xc0007a6a20, 0x1, 0x0, 0x0, 0x1, 0xc00bd90d00, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/bufio/bufio.go:138 +0x4f Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*persistConn).readLoop(0xc00bdab8c0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1679 +0x1a3 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: created by net/http.(*Transport).dialConn Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1359 +0xae8 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: goroutine 1126 [IO wait]: Dec 03 18:05:12 my.file.beat.com filebeat[30053]: internal/poll.runtime_pollWait(0x7f56b8d96f78, 0x72, 0xffffffffffffffff) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/netpoll.go:182 +0x56 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).wait(0xc00022e498, 0x72, 0xe700, 0xe78e, 0xffffffffffffffff) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b Dec 03 18:05:12 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).waitRead(...) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: internal/poll.(*FD).Read(0xc00022e480, 0xc00fe86000, 0xe78e, 0xe78e, 0x0, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net.(*netFD).Read(0xc00022e480, 0xc00fe86000, 0xe78e, 0xe78e, 0x203000, 0x0, 0xe781) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/fd_unix.go:202 +0x4f Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net.(*conn).Read(0xc000784ac0, 0xc00fe86000, 0xe78e, 0xe78e, 0x0, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/net.go:177 +0x69 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: crypto/tls.(*atLeastReader).Read(0xc000781300, 0xc00fe86000, 0xe78e, 0xe78e, 0x40, 0x2d53, 0xc003d658d8) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:761 +0x60 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: bytes.(*Buffer).ReadFrom(0xc0006493d8, 0x33cb6c0, 0xc000781300, 0x1289475, 0x2cf3640, 0x2f21b60) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/bytes/buffer.go:207 +0xbd Dec 03 18:05:12 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readFromUntil(0xc000649180, 0x33cfae0, 0xc000784ac0, 0x5, 0xc000784ac0, 0x40) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readRecordOrCCS(0xc000649180, 0x30da000, 0xc0006492b8, 0xc003d65b38) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:590 +0x125 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readRecord(...) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:558 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).Read(0xc000649180, 0xc0000fc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs/transport.(*statsConn).Read(0xc00bdc6fe0, 0xc0000fc000, 0x1000, 0x1000, 0xc003d65c50, 0x1284b8c, 0xc007ee69c0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/transport/stats.go:44 +0x55 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*persistConn).Read(0xc003d9c6c0, 0xc0000fc000, 0x1000, 0x1000, 0xc003d65c88, 0x1284665, 0xc007ee69c0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1526 +0x7b Dec 03 18:05:12 my.file.beat.com filebeat[30053]: bufio.(*Reader).fill(0xc001eb80c0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/bufio/bufio.go:100 +0x10f Dec 03 18:05:12 my.file.beat.com filebeat[30053]: bufio.(*Reader).Peek(0xc001eb80c0, 0x1, 0x0, 0x0, 0x1, 0xc00bd90f00, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/bufio/bufio.go:138 +0x4f Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*persistConn).readLoop(0xc003d9c6c0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1679 +0x1a3 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: created by net/http.(*Transport).dialConn Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1359 +0xae8 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: goroutine 278 [select]: Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/queue/memqueue.(*bufferingEventLoop).run(0xc000170d20) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/eventloop.go:316 +0x1db Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/queue/memqueue.NewBroker.func1(0xc00047a4d0, 0x33f0ac0, 0xc000170d20) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:175 +0x5d Dec 03 18:05:12 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/queue/memqueue.NewBroker Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:173 +0x374 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: goroutine 279 [select]: Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/queue/memqueue.(*ackLoop).run(0xc000150d70) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/ackloop.go:60 +0x107 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/queue/memqueue.NewBroker.func2(0xc00047a4d0, 0xc000150d70) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:179 +0x57 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/queue/memqueue.NewBroker Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:177 +0x3a3 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: goroutine 280 [select]: Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*eventConsumer).loop(0xc000150e60, 0x33f0b00, 0xc0008b0000) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:190 +0x2c9 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.newEventConsumer Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:77 +0x155 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: goroutine 281 [select, 2 minutes]: Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*retryer).loop(0xc000150eb0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:139 +0x1df Dec 03 18:05:12 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.newRetryer Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:91 +0x13d Dec 03 18:05:12 my.file.beat.com filebeat[30053]: goroutine 282 [select]: Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*persistConn).roundTrip(0xc003d9c6c0, 0xc01bdb2cc0, 0x0, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:2194 +0x75c Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*Transport).roundTrip(0xc000857200, 0xc0007bc800, 0xc01bdb2c90, 0xd597d66d8c, 0xd597d66d80) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:481 +0xa1b Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*Transport).RoundTrip(0xc000857200, 0xc0007bc800, 0xc000857200, 0xbf71c3628ed793f3, 0x2be91491df) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/roundtrip.go:17 +0x35 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.send(0xc0007bcb00, 0x33cfba0, 0xc000857200, 0xbf71c3628ed793f3, 0x2be91491df, 0x4e5a700, 0xc000010630, 0xbf71c3628ed793f3, 0x1, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:250 +0x461 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*Client).send(0xc0005bb9b0, 0xc0007bcb00, 0xbf71c3628ed793f3, 0x2be91491df, 0x4e5a700, 0xc000010630, 0x0, 0x1, 0x10) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:174 +0xfb Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*Client).do(0xc0005bb9b0, 0xc0007bcb00, 0x0, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:641 +0x279 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*Client).Do(...) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:509 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs/elasticsearch.(*Connection).execHTTPRequest(0xc00023ed80, 0xc0007bcb00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/client.go:826 +0x34c Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs/elasticsearch.(*Connection).sendBulkRequest(...) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/bulkapi.go:203 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs/elasticsearch.(*Client).publishEvents(0xc00023ed80, 0xc01baf0000, 0x32, 0x800, 0xc1dbaad0e0, 0xc1dbaad0e0, 0x0, 0xc00023f680, 0xc00023f708) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/client.go:342 +0x341 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs/elasticsearch.(*Client).Publish(0xc00023ed80, 0x3448080, 0xc0005b6780, 0xc000802d20, 0xc000799f78) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/client.go:293 +0x43 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs.(*backoffClient).Publish(0xc0005bb9e0, 0x3448080, 0xc0005b6780, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/backoff.go:60 +0x4b Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*netClientWorker).run(0xc0007ed5c0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:119 +0x315 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.makeClientWorker Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:48 +0xe0 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: goroutine 283 [select]: Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*persistConn).roundTrip(0xc00fcc57a0, 0xd467c45620, 0x0, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:2194 +0x75c Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*Transport).roundTrip(0xc000857440, 0xc00077e700, 0xd467c455f0, 0xc01bd2c59c, 0xc01bd2c590) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:481 +0xa1b Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*Transport).RoundTrip(0xc000857440, 0xc00077e700, 0xc000857440, 0xbf71c362499583b9, 0x2ba837b7a7) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/roundtrip.go:17 +0x35 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.send(0xc0007bce00, 0x33cfba0, 0xc000857440, 0xbf71c362499583b9, 0x2ba837b7a7, 0x4e5a700, 0xc00081c1a8, 0xbf71c362499583b9, 0x1, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:250 +0x461 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*Client).send(0xc0005bbc20, 0xc0007bce00, 0xbf71c362499583b9, 0x2ba837b7a7, 0x4e5a700, 0xc00081c1a8, 0x0, 0x1, 0x10) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:174 +0xfb Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*Client).do(0xc0005bbc20, 0xc0007bce00, 0x0, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:641 +0x279 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*Client).Do(...) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:509 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs/elasticsearch.(*Connection).execHTTPRequest(0xc00023ef00, 0xc0007bce00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/client.go:826 +0x34c Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs/elasticsearch.(*Connection).sendBulkRequest(...) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/bulkapi.go:203 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs/elasticsearch.(*Client).publishEvents(0xc00023ef00, 0xc013fc8000, 0x2c, 0x800, 0xc0008f4300, 0xc0008f4300, 0x0, 0xc00023f800, 0xc00023f888) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/client.go:342 +0x341 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs/elasticsearch.(*Client).Publish(0xc00023ef00, 0x3448080, 0xc0005b66c0, 0xc000802d20, 0xc00fe5df78) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/client.go:293 +0x43 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs.(*backoffClient).Publish(0xc0005bbc50, 0x3448080, 0xc0005b66c0, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/backoff.go:60 +0x4b Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*netClientWorker).run(0xc0007ed600) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:119 +0x315 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.makeClientWorker Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:48 +0xe0 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: goroutine 284 [select]: Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*persistConn).roundTrip(0xc00bdab8c0, 0xd467d29620, 0x0, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:2194 +0x75c Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*Transport).roundTrip(0xc000857d40, 0xc0007bc700, 0xd467d295f0, 0xd597d6642c, 0xd597d66420) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:481 +0xa1b Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*Transport).RoundTrip(0xc000857d40, 0xc0007bc700, 0xc000857d40, 0xbf71c3620a1d249f, 0x2b6d248f5e) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/roundtrip.go:17 +0x35 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.send(0xc0007bd100, 0x33cfba0, 0xc000857d40, 0xbf71c3620a1d249f, 0x2b6d248f5e, 0x4e5a700, 0xc0000105f8, 0xbf71c3620a1d249f, 0x1, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:250 +0x461 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*Client).send(0xc0005bbf20, 0xc0007bd100, 0xbf71c3620a1d249f, 0x2b6d248f5e, 0x4e5a700, 0xc0000105f8, 0x0, 0x1, 0x10) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:174 +0xfb Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*Client).do(0xc0005bbf20, 0xc0007bd100, 0x0, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:641 +0x279 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: net/http.(*Client).Do(...) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:509 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs/elasticsearch.(*Connection).execHTTPRequest(0xc00023f080, 0xc0007bd100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/client.go:826 +0x34c Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs/elasticsearch.(*Connection).sendBulkRequest(...) Dec 03 18:05:12 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/bulkapi.go:203 Dec 03 18:05:12 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs/elasticsearch.(*Client).publishEvents(0xc00023f080, 0xc013f3a000, 0x21, 0x800, 0xc0007caba0, 0xc0007caba0, 0x0, 0xc00023f980, 0xc00023fa08) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/client.go:342 +0x341 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs/elasticsearch.(*Client).Publish(0xc00023f080, 0x3448080, 0xc000894900, 0xc000802d20, 0xc00fe59f78) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/client.go:293 +0x43 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs.(*backoffClient).Publish(0xc0005bbf50, 0x3448080, 0xc000894900, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/backoff.go:60 +0x4b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*netClientWorker).run(0xc0007ed640) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:119 +0x315 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.makeClientWorker Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:48 +0xe0 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 286 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/monitoring/report/log.(*reporter).snapshotLoop(0xc0007ba510) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:129 +0x3c5 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/monitoring/report/log.MakeReporter.func1(0xc0007ba510) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:107 +0x53 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/monitoring/report/log.MakeReporter Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:105 +0x159 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 790 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input/log.(*Input).stopWhenDone(0xc000983380) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/input.go:753 +0x11c Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input/log.NewInput Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/input.go:155 +0x936 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 779 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).writeLoop(0xc0009706c0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1978 +0x113 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by net/http.(*Transport).dialConn Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1360 +0xb0d Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 946 [chan receive, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/x-pack/filebeat/input/netflow.(*netflowInput).recvRoutine(0xc0003ba870) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/x-pack/filebeat/input/netflow/input.go:237 +0x85 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/x-pack/filebeat/input/netflow.(*netflowInput).Run Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/x-pack/filebeat/input/netflow/input.go:167 +0x33d Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 376 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/queue/memqueue.(*directEventLoop).run(0xc000402630) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/eventloop.go:100 +0x1d9 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/queue/memqueue.NewBroker.func1(0xc00023bf80, 0x33f0b40, 0xc000402630) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:175 +0x5d Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/queue/memqueue.NewBroker Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:173 +0x374 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 377 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/queue/memqueue.(*ackLoop).run(0xc00015f6d0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/ackloop.go:60 +0x107 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/queue/memqueue.NewBroker.func2(0xc00023bf80, 0xc00015f6d0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:179 +0x57 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/queue/memqueue.NewBroker Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:177 +0x3a3 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 378 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*eventConsumer).loop(0xc00015f7c0, 0x33f0b00, 0xc000743a60) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:190 +0x2c9 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.newEventConsumer Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:77 +0x155 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 379 [select, 1 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*retryer).loop(0xc00015f810) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:139 +0x1df Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.newRetryer Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:91 +0x13d Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 380 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).roundTrip(0xc000800240, 0xc01bb27cb0, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:2194 +0x75c Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*Transport).roundTrip(0xc00080cea0, 0xc0007bca00, 0xc01bb27a10, 0xd467ee36d0, 0xd467ee36c4) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:481 +0xa1b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*Transport).RoundTrip(0xc00080cea0, 0xc0007bca00, 0xc00080cea0, 0xbf71c35b762aa2fb, 0x254fdebf20) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/roundtrip.go:17 +0x35 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.send(0xc0007bc900, 0x33cfba0, 0xc00080cea0, 0xbf71c35b762aa2fb, 0x254fdebf20, 0x4e5a700, 0xc0007c0338, 0xbf71c35b762aa2fb, 0x1, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:250 +0x461 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*Client).send(0xc00083b320, 0xc0007bc900, 0xbf71c35b762aa2fb, 0x254fdebf20, 0x4e5a700, 0xc0007c0338, 0x0, 0x1, 0x10) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:174 +0xfb Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*Client).do(0xc00083b320, 0xc0007bc900, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:641 +0x279 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*Client).Do(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:509 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs/elasticsearch.(*Connection).execHTTPRequest(0xc0004eb380, 0xc0007bc900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/client.go:826 +0x34c Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs/elasticsearch.(*Connection).sendBulkRequest(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/bulkapi.go:203 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs/elasticsearch.(*Connection).BulkWith(0xc0004eb380, 0xc0008ed8c0, 0x1e, 0x0, 0x0, 0x0, 0x0, 0xc00079fa60, 0x2, 0x2, ...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/bulkapi.go:75 +0x17d Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/monitoring/report/elasticsearch.(*publishClient).publishBulk(0xc000743680, 0xbf71c34c711442d2, 0x17528106c7, 0x4e5a700, 0xc01bb15620, 0xc01bb15650, 0x0, 0x0, 0x0, 0x0, ...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/monitoring/report/elasticsearch/client.go:233 +0x5cb Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/monitoring/report/elasticsearch.(*publishClient).Publish(0xc000743680, 0x3448080, 0xc02fb13280, 0xc00051a958, 0xc0004ebc38) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/monitoring/report/elasticsearch/client.go:140 +0x6b2 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs.(*failoverClient).Publish(0xc000743920, 0x3448080, 0xc02fb13280, 0xc0007c90e0, 0xc00079ff78) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/failover.go:99 +0x63 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs.(*backoffClient).Publish(0xc00083b5f0, 0x3448080, 0xc02fb13280, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/backoff.go:60 +0x4b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*netClientWorker).run(0xc00005af80) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:119 +0x315 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.makeClientWorker Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:48 +0xe0 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 381 [chan receive, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/service.HandleSignals.func1(0xc0008e39e0, 0xc000745ae0, 0xc00016b510, 0xc0008d6720) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/service/service.go:49 +0x41 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/service.HandleSignals Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/service/service.go:48 +0x115 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 789 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dc00, 0xc001ecda10, 0xc0007c87e0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 383 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*pipelineEventCB).worker(0xc00005b080) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline_ack.go:245 +0x1a1 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.newPipelineEventCB Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline_ack.go:196 +0x15b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 384 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/registrar.(*Registrar).Run(0xc00051b5c0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/registrar/registrar.go:291 +0x161 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/registrar.(*Registrar).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/registrar/registrar.go:272 +0x13a Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 385 [select, 1 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc00015aa30) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 386 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dbc0, 0xc000852b40, 0xc0007c94a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 387 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Run(0xc00051b860) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:147 +0x145 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Start.func1(0xc0007c6ef0, 0xc00051b860) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:132 +0x5b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input.(*Runner).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:125 +0x16b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 388 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc00015b690) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 389 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dbc0, 0xc000858540, 0xc0007c94a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 390 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Run(0xc00015aea0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:147 +0x145 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Start.func1(0xc0007c7480, 0xc00015aea0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:132 +0x5b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input.(*Runner).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:125 +0x16b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 313 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/x-pack/filebeat/input/netflow/decoder/v9.(*SessionMap).CleanupLoop(0xc00015b6f8, 0x1a3185c5000, 0xc000796420) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/x-pack/filebeat/input/netflow/decoder/v9/session.go:182 +0xf5 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/x-pack/filebeat/input/netflow/decoder/v9.(*NetflowV9Protocol).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/x-pack/filebeat/input/netflow/decoder/v9/v9.go:62 +0x99 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 314 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/x-pack/filebeat/input/netflow/decoder/v9.(*SessionMap).CleanupLoop(0xc00015b758, 0x1a3185c5000, 0xc000796480) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/x-pack/filebeat/input/netflow/decoder/v9/session.go:182 +0xf5 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/x-pack/filebeat/input/netflow/decoder/v9.(*NetflowV9Protocol).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/x-pack/filebeat/input/netflow/decoder/v9/v9.go:62 +0x99 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 263 [IO wait]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.runtime_pollWait(0x7f56b8d96b68, 0x72, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/netpoll.go:182 +0x56 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).wait(0xc000922098, 0x72, 0x2800, 0x2800, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).waitRead(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*FD).ReadFrom(0xc000922080, 0xc003ed4000, 0x2800, 0x2800, 0x0, 0x0, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_unix.go:219 +0x194 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*netFD).readFrom(0xc000922080, 0xc003ed4000, 0x2800, 0x2800, 0x30d9518, 0xc0007f1e60, 0xc0007f1e50, 0x1308f6d, 0xc000922080) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/fd_unix.go:208 +0x5b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*UDPConn).readFrom(0xc00018a0e0, 0xc003ed4000, 0x2800, 0x2800, 0xbf71c3970b978826, 0x5ccace3a13, 0x4e5a700, 0xe9) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/udpsock_posix.go:47 +0x6a Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*UDPConn).ReadFrom(0xc00018a0e0, 0xc003ed4000, 0x2800, 0x2800, 0x0, 0x0, 0x4e5a700, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/udpsock.go:121 +0x5e Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/inputsource/udp.(*Server).run(0xc00005b240) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:106 +0x126 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/inputsource/udp.(*Server).Start.func1(0xc00005b240) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:85 +0x57 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/inputsource/udp.(*Server).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:83 +0x172 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 316 [IO wait]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.runtime_pollWait(0x7f56b8d96d08, 0x72, 0xffffffffffffffff) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/netpoll.go:182 +0x56 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).wait(0xc000779198, 0x72, 0x200, 0x200, 0xffffffffffffffff) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).waitRead(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*FD).Read(0xc000779180, 0xc027ebec00, 0x200, 0x200, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*netFD).Read(0xc000779180, 0xc027ebec00, 0x200, 0x200, 0xc0005a1120, 0x128961a, 0x2d9f540) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/fd_unix.go:202 +0x4f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*conn).Read(0xc0007ac268, 0xc027ebec00, 0x200, 0x200, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/net.go:177 +0x69 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/vendor/github.com/miekg/dns.(*Conn).Read(0xc033afb380, 0xc027ebec00, 0x200, 0x200, 0xc0007ac268, 0xc00005e501, 0xc00005e500) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/vendor/github.com/miekg/dns/client.go:261 +0x23f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/vendor/github.com/miekg/dns.(*Conn).ReadMsgHeader(0xc033afb380, 0x0, 0x0, 0x12a05e7d4, 0xc00fc86360, 0x12a05e7d4, 0xc0005a12c8) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/vendor/github.com/miekg/dns/client.go:225 +0xc7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/vendor/github.com/miekg/dns.(*Conn).ReadMsg(0xc033afb380, 0xbf71c352d59b052c, 0x1d44c04d26, 0x4e5a700) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/vendor/github.com/miekg/dns/client.go:187 +0x49 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/vendor/github.com/miekg/dns.(*Client).exchange(0xc00047a0e0, 0xc0005a1510, 0xc0004653c0, 0xf, 0x0, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/vendor/github.com/miekg/dns/client.go:173 +0x27a Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/vendor/github.com/miekg/dns.(*Client).Exchange(0xc00047a0e0, 0xc0005a1510, 0xc0004653c0, 0xf, 0x0, 0x12a089d40, 0x33cfaa0, 0xc0007b8b40) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/vendor/github.com/miekg/dns/client.go:129 +0x2bc Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/processors/dns.(*MiekgResolver).LookupPTR(0xc00057a910, 0xc013acae40, 0xe, 0x4e5a700, 0xc013acae40, 0xe) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/processors/dns/resolver.go:137 +0x235 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/processors/dns.PTRLookupCache.LookupPTR(0xc0005bb110, 0xc0005bb140, 0x0, 0x33cbf80, 0xc00057a910, 0xc0004653d0, 0xc0004653d8, 0xc013acae40, 0xe, 0x0, ...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/processors/dns/cache.go:195 +0x130 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/processors/dns.(*processor).processField(0xc000592bb0, 0xc000465360, 0xe, 0xc00033d380, 0x14, 0x12e6600, 0xc02ffed1c0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/processors/dns/dns.go:103 +0xc1 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/processors/dns.(*processor).Run(0xc000592bb0, 0xc02ffed1c0, 0xc02ffed1c0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/processors/dns/dns.go:84 +0x111 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/processing.(*group).Run(0xc00058ef60, 0xc02ffed1c0, 0xc02ffed1c0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/processing/processors.go:104 +0x94 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/processing.(*group).Run(0xc000858390, 0xc02ffed1c0, 0xed5788c59, 0x0, 0xc1dbb58f00) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/processing/processors.go:104 +0x94 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*client).publish(0xc0007a1080, 0x0, 0xed5788c59, 0x0, 0x0, 0xd597cea210, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client.go:89 +0x5c3 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*client).Publish(0xc0007a1080, 0x0, 0xed5788c59, 0x0, 0x0, 0xd597cea210, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client.go:68 +0x9c Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.(*outlet).OnEvent(0xc000858540, 0x0, 0xed5788c59, 0x0, 0x0, 0xd597cea210, 0x0, 0x0, 0x0, 0x2887fa01) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/outlet.go:64 +0x80 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/harvester.(*Forwarder).Send(0xc0003f3ea0, 0x0, 0xed5788c59, 0x0, 0x0, 0xd597cea210, 0x0, 0x0, 0x0, 0x0, ...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/harvester/forwarder.go:50 +0x65 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/x-pack/filebeat/input/netflow.(*netflowInput).Publish(0xc00015fd10, 0xd597ce8000, 0x1d, 0x1d, 0xd467b40b10, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/x-pack/filebeat/input/netflow/input.go:138 +0x111 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/x-pack/filebeat/input/netflow.(*netflowInput).recvRoutine(0xc00015fd10) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/x-pack/filebeat/input/netflow/input.go:248 +0x32c Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/x-pack/filebeat/input/netflow.(*netflowInput).Run Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/x-pack/filebeat/input/netflow/input.go:167 +0x33d Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 950 [IO wait, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.runtime_pollWait(0x7f56b8d96688, 0x72, 0xffffffffffffffff) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/netpoll.go:182 +0x56 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).wait(0xc0008aff98, 0x72, 0xdf00, 0xdf50, 0xffffffffffffffff) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).waitRead(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*FD).Read(0xc0008aff80, 0xc003d50000, 0xdf50, 0xdf50, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*netFD).Read(0xc0008aff80, 0xc003d50000, 0xdf50, 0xdf50, 0x203000, 0x0, 0x33) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/fd_unix.go:202 +0x4f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*conn).Read(0xc00081c020, 0xc003d50000, 0xdf50, 0xdf50, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/net.go:177 +0x69 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*atLeastReader).Read(0xc00080a220, 0xc003d50000, 0xdf50, 0xdf50, 0x4, 0xa, 0xc00078d938) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:761 +0x60 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: bytes.(*Buffer).ReadFrom(0xc00059f058, 0x33cb6c0, 0xc00080a220, 0x1289475, 0x2cf3640, 0x2f21b60) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/bytes/buffer.go:207 +0xbd Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readFromUntil(0xc00059ee00, 0x33cfae0, 0xc00081c020, 0x5, 0xc00081c020, 0x203000) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readRecordOrCCS(0xc00059ee00, 0x30da000, 0xc00059ef38, 0xc00078db88) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:590 +0x125 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readRecord(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:558 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).Read(0xc00059ee00, 0xc001e7f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).Read(0xc00072ca20, 0xc001e7f000, 0x1000, 0x1000, 0xc00078dc88, 0x1284665, 0xc000802360) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1526 +0x7b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: bufio.(*Reader).fill(0xc001e96000) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/bufio/bufio.go:100 +0x10f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: bufio.(*Reader).Peek(0xc001e96000, 0x1, 0x0, 0x0, 0x1, 0xc0007b5f00, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/bufio/bufio.go:138 +0x4f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).readLoop(0xc00072ca20) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1679 +0x1a3 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by net/http.(*Transport).dialConn Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1359 +0xae8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 951 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).writeLoop(0xc00072ca20) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1978 +0x113 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by net/http.(*Transport).dialConn Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1360 +0xb0d Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 891 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/reader/readfile.(*TimeoutReader).Next(0xc00005b840, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc0008580a8, ...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/readfile/timeout.go:79 +0x12f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/reader/multiline.(*Reader).readNext(0xc001e5a9c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc037c2ad20, ...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/multiline/multiline.go:166 +0x6e Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/reader/multiline.(*Reader).Next(0xc001e5a9c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc002923d48, ...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/multiline/multiline.go:134 +0x6a Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/reader/readfile.(*LimitReader).Next(0xc0008d6e20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x257a01b, ...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/readfile/limit.go:40 +0x6b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input/log.(*Harvester).Run(0xc000286780, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:260 +0x2a1 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/harvester.(*Registry).Start.func1(0xc00005b900, 0x340dd80, 0xc000286780) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/harvester/registry.go:96 +0x73 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/harvester.(*Registry).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/harvester/registry.go:90 +0x128 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 847 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input/log.(*Log).wait(0xc00041ad20) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/log.go:195 +0xbc Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input/log.(*Log).Read(0xc00041ad20, 0xc0008d2000, 0x4000, 0x4000, 0x1, 0xffffffffffffffff, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/log.go:108 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/reader/readfile.(*LineReader).advance(0xc00077a200, 0x8, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/readfile/line.go:133 +0x12f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/reader/readfile.(*LineReader).Next(0xc00077a200, 0xc0007dd5d8, 0x128b369, 0xc000160fc0, 0x60, 0x60, 0x2f18940) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/readfile/line.go:78 +0x2f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/reader/readfile.EncoderReader.Next(0xc00077a200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1284d2c, ...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/readfile/encode.go:53 +0x63 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/reader/readfile.(*StripNewline).Next(0xc0005e7b00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x13edd64, ...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/readfile/strip_newline.go:50 +0x6b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/reader/readfile.(*TimeoutReader).Next.func1(0xc0007ec6c0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/readfile/timeout.go:70 +0x48 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/reader/readfile.(*TimeoutReader).Next Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/readfile/timeout.go:68 +0x287 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 721 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input/log.(*Input).stopWhenDone(0xc0008c29c0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/input.go:753 +0x11c Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input/log.NewInput Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/input.go:155 +0x936 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 786 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc001e62bb0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 845 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dc00, 0xc0008091a0, 0xc0007b4b40) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 778 [IO wait, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.runtime_pollWait(0x7f56b8d968f8, 0x72, 0xffffffffffffffff) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/netpoll.go:182 +0x56 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).wait(0xc000813e18, 0x72, 0x1a00, 0x1af9, 0xffffffffffffffff) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).waitRead(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*FD).Read(0xc000813e00, 0xc00089e000, 0x1af9, 0x1af9, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*netFD).Read(0xc000813e00, 0xc00089e000, 0x1af9, 0x1af9, 0x17b8d460eb9bfb, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/fd_unix.go:202 +0x4f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*conn).Read(0xc00081c000, 0xc00089e000, 0x1af9, 0x1af9, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/net.go:177 +0x69 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*atLeastReader).Read(0xc0005e4000, 0xc00089e000, 0x1af9, 0x1af9, 0x4, 0xa, 0xc00094d938) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:761 +0x60 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: bytes.(*Buffer).ReadFrom(0xc00059e958, 0x33cb6c0, 0xc0005e4000, 0x1289475, 0x2cf3640, 0x2f21b60) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/bytes/buffer.go:207 +0xbd Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readFromUntil(0xc00059e700, 0x33cfae0, 0xc00081c000, 0x5, 0xc00081c000, 0x203000) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readRecordOrCCS(0xc00059e700, 0x30da000, 0xc00059e838, 0xc00094db88) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:590 +0x125 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readRecord(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:558 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).Read(0xc00059e700, 0xc00086c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).Read(0xc0009706c0, 0xc00086c000, 0x1000, 0x1000, 0xc00094dc88, 0x1284665, 0xc0007b4a80) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1526 +0x7b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: bufio.(*Reader).fill(0xc00015a720) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/bufio/bufio.go:100 +0x10f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: bufio.(*Reader).Peek(0xc00015a720, 0x1, 0x0, 0x0, 0x1, 0xc000802600, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/bufio/bufio.go:138 +0x4f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).readLoop(0xc0009706c0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1679 +0x1a3 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by net/http.(*Transport).dialConn Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1359 +0xae8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 846 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input/log.(*Harvester).Run.func2(0xc000286500, 0xc001edc7c0, 0x31) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:239 +0xca Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input/log.(*Harvester).Run Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:232 +0x1d2 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 844 [chan receive, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.SubOutlet.func1(0xc0008091a0, 0x340dbc0, 0xc001ebdce0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:44 +0x150 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.SubOutlet Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 780 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/reader/readfile.(*TimeoutReader).Next(0xc0007ec6c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x33cb8e0, ...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/readfile/timeout.go:79 +0x12f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/reader/multiline.(*Reader).readFirst(0xc0008f2410, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, ...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/multiline/multiline.go:139 +0x6b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/reader/multiline.(*Reader).Next(0xc0008f2410, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x49c9651, ...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/multiline/multiline.go:134 +0x6a Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/reader/readfile.(*LimitReader).Next(0xc0008b1a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x12afc2b, ...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/readfile/limit.go:40 +0x6b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input/log.(*Harvester).Run(0xc000286500, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:260 +0x2a1 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/harvester.(*Registry).Start.func1(0xc001ea51c0, 0x340dd80, 0xc000286500) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/harvester/registry.go:96 +0x73 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/harvester.(*Registry).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/harvester/registry.go:90 +0x128 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 840 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dbc0, 0xc0005906c0, 0xc0007c94a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 839 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc000231ab0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 289 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).writeLoop(0xc000800240) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1978 +0x113 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by net/http.(*Transport).dialConn Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1360 +0xb0d Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 288 [IO wait]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.runtime_pollWait(0x7f56b8d96c38, 0x72, 0xffffffffffffffff) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/netpoll.go:182 +0x56 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).wait(0xc000188418, 0x72, 0xc00, 0xc72, 0xffffffffffffffff) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).waitRead(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*FD).Read(0xc000188400, 0xc000452000, 0xc72, 0xc72, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*netFD).Read(0xc000188400, 0xc000452000, 0xc72, 0xc72, 0x203004, 0x0, 0x34) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/fd_unix.go:202 +0x4f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*conn).Read(0xc00018a0d0, 0xc000452000, 0xc72, 0xc72, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/net.go:177 +0x69 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*atLeastReader).Read(0xc013f31b80, 0xc000452000, 0xc72, 0xc72, 0x3, 0xa, 0xc00078f938) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:761 +0x60 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: bytes.(*Buffer).ReadFrom(0xc0004d8958, 0x33cb6c0, 0xc013f31b80, 0x1289475, 0x2cf3640, 0x2f21b60) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/bytes/buffer.go:207 +0xbd Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readFromUntil(0xc0004d8700, 0x33cfae0, 0xc00018a0d0, 0x5, 0xc00018a0d0, 0x203007) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readRecordOrCCS(0xc0004d8700, 0x30da000, 0xc0004d8838, 0xc00078fb88) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:590 +0x125 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readRecord(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:558 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).Read(0xc0004d8700, 0xc0003ea000, 0x1000, 0x1000, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).Read(0xc000800240, 0xc0003ea000, 0x1000, 0x1000, 0xc00078fc88, 0x1284665, 0xc000184240) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1526 +0x7b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: bufio.(*Reader).fill(0xc0008e2360) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/bufio/bufio.go:100 +0x10f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: bufio.(*Reader).Peek(0xc0008e2360, 0x1, 0x0, 0x0, 0x1, 0xc0000bff00, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/bufio/bufio.go:138 +0x4f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).readLoop(0xc000800240) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1679 +0x1a3 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by net/http.(*Transport).dialConn Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1359 +0xae8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 787 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dbc0, 0xc001ecd9e0, 0xc0007c94a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 788 [chan receive, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.SubOutlet.func1(0xc001ecda10, 0x340dbc0, 0xc001ecd9e0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:44 +0x150 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.SubOutlet Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 513 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc0008f5d50) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 514 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dbc0, 0xc00083a6c0, 0xc0007c94a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 515 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc0001609d0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 516 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dbc0, 0xc0008526c0, 0xc0007c94a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 569 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc00015bf30) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 570 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dbc0, 0xc00084b5c0, 0xc0007c94a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 272 [chan receive, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/beater.waitChannel.func1() Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/beater/signalwait.go:79 +0x33 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/beater.(*signalWait).Add.func1(0xc0009250f0, 0xc0001526b0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/beater/signalwait.go:52 +0x27 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/beater.(*signalWait).Add Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/beater/signalwait.go:51 +0x4c Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 271 [chan receive, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/cfgfile.(*Reloader).Run(0xc00015b9e0, 0x33cba00, 0xc00015fd60) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/cfgfile/reload.go:228 +0x43e Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/crawler.(*Crawler).Start.func2(0xc000380510) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/crawler/crawler.go:102 +0x41 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/crawler.(*Crawler).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/crawler/crawler.go:101 +0x45f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 803 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).writeLoop(0xc001eec240) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1978 +0x113 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by net/http.(*Transport).dialConn Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1360 +0xb0d Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 802 [IO wait, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.runtime_pollWait(0x7f56b8d969c8, 0x72, 0xffffffffffffffff) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/netpoll.go:182 +0x56 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).wait(0xc000189d98, 0x72, 0xc00, 0xc72, 0xffffffffffffffff) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).waitRead(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*FD).Read(0xc000189d80, 0xc000452c80, 0xc72, 0xc72, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*netFD).Read(0xc000189d80, 0xc000452c80, 0xc72, 0xc72, 0x203000, 0x12a0475, 0xc0008d6320) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/fd_unix.go:202 +0x4f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*conn).Read(0xc00018a070, 0xc000452c80, 0xc72, 0xc72, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/net.go:177 +0x69 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*atLeastReader).Read(0xc0008d6320, 0xc000452c80, 0xc72, 0xc72, 0x4e63020, 0x7f56b8dfb908, 0xc000789938) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:761 +0x60 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: bytes.(*Buffer).ReadFrom(0xc000324cd8, 0x33cb6c0, 0xc0008d6320, 0x1289475, 0x2cf3640, 0x2f21b60) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/bytes/buffer.go:207 +0xbd Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readFromUntil(0xc000324a80, 0x33cfae0, 0xc00018a070, 0x5, 0xc00018a070, 0x203000) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readRecordOrCCS(0xc000324a80, 0x30da000, 0xc000324bb8, 0xc000789b88) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:590 +0x125 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readRecord(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:558 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).Read(0xc000324a80, 0xc003ccc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).Read(0xc001eec240, 0xc003ccc000, 0x1000, 0x1000, 0xc000789c88, 0x1284665, 0xc0007b5200) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1526 +0x7b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: bufio.(*Reader).fill(0xc0007a6e40) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/bufio/bufio.go:100 +0x10f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: bufio.(*Reader).Peek(0xc0007a6e40, 0x1, 0x0, 0x0, 0x1, 0xc0001d0600, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/bufio/bufio.go:138 +0x4f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).readLoop(0xc001eec240) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1679 +0x1a3 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by net/http.(*Transport).dialConn Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1359 +0xae8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 270 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input/log.(*Input).stopWhenDone(0xc0008c21a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/input.go:753 +0x11c Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input/log.NewInput Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/input.go:155 +0x936 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 617 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc0008055d0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 618 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dbc0, 0xc000910630, 0xc0007c94a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 619 [chan receive, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.SubOutlet.func1(0xc000910660, 0x340dbc0, 0xc000910630) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:44 +0x150 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.SubOutlet Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 620 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dc00, 0xc000910660, 0xc0007c87e0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 621 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input/log.(*Input).stopWhenDone(0xc0009824e0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/input.go:753 +0x11c Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input/log.NewInput Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/input.go:155 +0x936 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 622 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc0007ca910) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 623 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dbc0, 0xc0007eef90, 0xc0007c94a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 624 [chan receive, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.SubOutlet.func1(0xc0007eefc0, 0x340dbc0, 0xc0007eef90) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:44 +0x150 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.SubOutlet Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 625 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dc00, 0xc0007eefc0, 0xc0007c87e0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 626 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input/log.(*Input).stopWhenDone(0xc000982820) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/input.go:753 +0x11c Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input/log.NewInput Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/input.go:155 +0x936 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 1404 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.setRequestCancel.func3(0x0, 0xd467d295f0, 0xc0003ba320, 0xd597d6642c, 0xc0007c8120) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:321 +0xcf Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by net/http.setRequestCancel Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:320 +0x251 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 155 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).writeLoop(0xc000909680) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1978 +0x113 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by net/http.(*Transport).dialConn Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1360 +0xb0d Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 269 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dc00, 0xc0007834d0, 0xc0007c87e0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 666 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc0007b7a50) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 667 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dbc0, 0xc0005e7020, 0xc0007c94a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 267 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dbc0, 0xc0007834a0, 0xc0007c94a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 268 [chan receive, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.SubOutlet.func1(0xc0007834d0, 0x340dbc0, 0xc0007834a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:44 +0x150 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.SubOutlet Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 266 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc00051abb0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 265 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input/log.(*Input).stopWhenDone(0xc000983040) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/input.go:753 +0x11c Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input/log.NewInput Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/input.go:155 +0x936 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 867 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Run(0xc000230840) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:147 +0x145 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Start.func1(0xc000810060, 0xc000230840) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:132 +0x5b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input.(*Runner).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:125 +0x16b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 715 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc00088db70) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 716 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dbc0, 0xc0008977d0, 0xc0007c94a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 717 [chan receive, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.SubOutlet.func1(0xc000897800, 0x340dbc0, 0xc0008977d0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:44 +0x150 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.SubOutlet Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 718 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dc00, 0xc000897800, 0xc0007c87e0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 719 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/monitoring/report/elasticsearch.(*reporter).snapshotLoop(0xc00047a540, 0x2fdf81c, 0x5, 0x2fdf81c, 0x5, 0xdf8475800, 0xc013e759c0, 0x16) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/monitoring/report/elasticsearch/elasticsearch.go:282 +0x2f4 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/monitoring/report/elasticsearch.(*reporter).initLoop Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/monitoring/report/elasticsearch/elasticsearch.go:265 +0x473 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 720 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/monitoring/report/elasticsearch.(*reporter).snapshotLoop(0xc00047a540, 0x2fdf826, 0x5, 0x2fe2c86, 0x7, 0x2540be400, 0xc013e759c0, 0x16) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/monitoring/report/elasticsearch/elasticsearch.go:282 +0x2f4 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/monitoring/report/elasticsearch.(*reporter).initLoop Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/monitoring/report/elasticsearch/elasticsearch.go:267 +0x4e9 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 774 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc001eb8c70) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 775 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dbc0, 0xc001ebdce0, 0xc0007c94a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 776 [chan receive, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.SubOutlet.func1(0xc001ebdd10, 0x340dbc0, 0xc001ebdce0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:44 +0x150 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.SubOutlet Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 777 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dc00, 0xc001ebdd10, 0xc0007c87e0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 781 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Run(0xc000231620) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:147 +0x145 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Start.func1(0xc0004658b0, 0xc000231620) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:132 +0x5b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input.(*Runner).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:125 +0x16b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 160 [IO wait]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.runtime_pollWait(0x7f56b8d97048, 0x72, 0xffffffffffffffff) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/netpoll.go:182 +0x56 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).wait(0xc00fb75698, 0x72, 0xf900, 0xf950, 0xffffffffffffffff) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).waitRead(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*FD).Read(0xc00fb75680, 0xc013d00000, 0xf950, 0xf950, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*netFD).Read(0xc00fb75680, 0xc013d00000, 0xf950, 0xf950, 0x203000, 0x0, 0xf943) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/fd_unix.go:202 +0x4f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*conn).Read(0xc00081c060, 0xc013d00000, 0xf950, 0xf950, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/net.go:177 +0x69 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*atLeastReader).Read(0xc000781200, 0xc013d00000, 0xf950, 0xf950, 0x40, 0x17ab, 0xc00008d8d8) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:761 +0x60 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: bytes.(*Buffer).ReadFrom(0xc00059fad8, 0x33cb6c0, 0xc000781200, 0x1289475, 0x2cf3640, 0x2f21b60) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/bytes/buffer.go:207 +0xbd Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readFromUntil(0xc00059f880, 0x33cfae0, 0xc00081c060, 0x5, 0xc00081c060, 0x40) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readRecordOrCCS(0xc00059f880, 0x30da000, 0xc00059f9b8, 0xc00008db38) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:590 +0x125 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readRecord(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:558 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).Read(0xc00059f880, 0xc00fd56000, 0x1000, 0x1000, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/outputs/transport.(*statsConn).Read(0xc00bd98ac0, 0xc00fd56000, 0x1000, 0x1000, 0xc00008dc50, 0x1284b8c, 0xc00fc01860) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/outputs/transport/stats.go:44 +0x55 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).Read(0xc00fcc57a0, 0xc00fd56000, 0x1000, 0x1000, 0xc00008dc88, 0x1284665, 0xc00fc01860) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1526 +0x7b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: bufio.(*Reader).fill(0xc0007a3620) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/bufio/bufio.go:100 +0x10f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: bufio.(*Reader).Peek(0xc0007a3620, 0x1, 0x0, 0x0, 0x1, 0xc000797500, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/bufio/bufio.go:138 +0x4f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).readLoop(0xc00fcc57a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1679 +0x1a3 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by net/http.(*Transport).dialConn Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1359 +0xae8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 161 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).writeLoop(0xc00fcc57a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1978 +0x113 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by net/http.(*Transport).dialConn Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1360 +0xb0d Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 794 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Run(0xc001eb83c0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:147 +0x145 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Start.func1(0xc00033e0a0, 0xc001eb83c0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:132 +0x5b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input.(*Runner).Start Dec 03 18:05:13 my.file.beat.com systemd[1]: filebeat.service: main process exited, code=exited, status=2/INVALIDARGUMENT Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:125 +0x16b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 795 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Run(0xc001e62360) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:147 +0x145 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Start.func1(0xc00033e1f0, 0xc001e62360) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:132 +0x5b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input.(*Runner).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:125 +0x16b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 800 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/x-pack/filebeat/input/netflow/decoder/v9.(*SessionMap).CleanupLoop(0xc000231b18, 0x1a3185c5000, 0xc0000bf020) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/x-pack/filebeat/input/netflow/decoder/v9/session.go:182 +0xf5 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/x-pack/filebeat/input/netflow/decoder/v9.(*NetflowV9Protocol).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/x-pack/filebeat/input/netflow/decoder/v9/v9.go:62 +0x99 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 868 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Run(0xc0002313e0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:147 +0x145 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Start.func1(0xc000810080, 0xc0002313e0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:132 +0x5b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input.(*Runner).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:125 +0x16b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 801 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/x-pack/filebeat/input/netflow/decoder/v9.(*SessionMap).CleanupLoop(0xc000231b78, 0x1a3185c5000, 0xc0000bf080) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/x-pack/filebeat/input/netflow/decoder/v9/session.go:182 +0xf5 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/x-pack/filebeat/input/netflow/decoder/v9.(*NetflowV9Protocol).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/x-pack/filebeat/input/netflow/decoder/v9/v9.go:62 +0x99 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 849 [IO wait, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.runtime_pollWait(0x7f56b8d96828, 0x72, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/netpoll.go:182 +0x56 Dec 03 18:05:13 my.file.beat.com systemd[1]: Unit filebeat.service entered failed state. Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).wait(0xc000812118, 0x72, 0x2800, 0x2800, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).waitRead(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*FD).ReadFrom(0xc000812100, 0xc000872800, 0x2800, 0x2800, 0x0, 0x0, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_unix.go:219 +0x194 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*netFD).readFrom(0xc000812100, 0xc000872800, 0x2800, 0x2800, 0x30d9518, 0xc001ef1e60, 0xc001ef1e50, 0x1308f6d, 0xc000812100) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/fd_unix.go:208 +0x5b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*UDPConn).readFrom(0xc0007ac028, 0xc000872800, 0x2800, 0x2800, 0xbf71c37e75734622, 0x45e7cdd9f6, 0x4e5a700, 0xe9) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/udpsock_posix.go:47 +0x6a Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*UDPConn).ReadFrom(0xc0007ac028, 0xc000872800, 0x2800, 0x2800, 0x0, 0x0, 0x4e5a700, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/udpsock.go:121 +0x5e Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/inputsource/udp.(*Server).run(0xc0007e7040) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:106 +0x126 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/inputsource/udp.(*Server).Start.func1(0xc0007e7040) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:85 +0x57 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/inputsource/udp.(*Server).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:83 +0x172 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 931 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc000230d90) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 932 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dbc0, 0xc00058f830, 0xc0007c94a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com systemd[1]: filebeat.service failed. Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 933 [chan receive, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.SubOutlet.func1(0xc00058f860, 0x340dbc0, 0xc00058f830) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:44 +0x150 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.SubOutlet Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 934 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dc00, 0xc00058f860, 0xc0007c87e0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 935 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input/log.(*Input).stopWhenDone(0xc0000dc680) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/input.go:753 +0x11c Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input/log.NewInput Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/input.go:155 +0x936 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 936 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc000231990) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 937 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dbc0, 0xc0005e6c90, 0xc0007c94a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 938 [chan receive, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.SubOutlet.func1(0xc0005e6cc0, 0x340dbc0, 0xc0005e6c90) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:44 +0x150 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.SubOutlet Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 939 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dc00, 0xc0005e6cc0, 0xc0007c87e0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 940 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input/log.(*Input).stopWhenDone(0xc0000dc9c0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/input.go:753 +0x11c Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input/log.NewInput Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/input.go:155 +0x936 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 1570 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.setRequestCancel.func3(0x0, 0xc01bb27a10, 0xc000050aa0, 0xd467ee36d0, 0xc0001d0300) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:321 +0xcf Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by net/http.setRequestCancel Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:320 +0x251 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 154 [IO wait, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.runtime_pollWait(0x7f56b8d96758, 0x72, 0xffffffffffffffff) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/netpoll.go:182 +0x56 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).wait(0xc00077ae98, 0x72, 0x2000, 0x2011, 0xffffffffffffffff) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).waitRead(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*FD).Read(0xc00077ae80, 0xc00020a500, 0x2011, 0x2011, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*netFD).Read(0xc00077ae80, 0xc00020a500, 0x2011, 0x2011, 0x203000, 0x0, 0x32) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/fd_unix.go:202 +0x4f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*conn).Read(0xc00018a078, 0xc00020a500, 0x2011, 0x2011, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/net.go:177 +0x69 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*atLeastReader).Read(0xc0008b0100, 0xc00020a500, 0x2011, 0x2011, 0x4e63020, 0x7f56b8d97fd0, 0xc000941938) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:761 +0x60 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: bytes.(*Buffer).ReadFrom(0xc000648958, 0x33cb6c0, 0xc0008b0100, 0x1289475, 0x2cf3640, 0x2f21b60) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/bytes/buffer.go:207 +0xbd Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readFromUntil(0xc000648700, 0x33cfae0, 0xc00018a078, 0x5, 0xc00018a078, 0x203000) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readRecordOrCCS(0xc000648700, 0x30da000, 0xc000648838, 0xc000941b88) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:590 +0x125 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).readRecord(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:558 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: crypto/tls.(*Conn).Read(0xc000648700, 0xc000748000, 0x1000, 0x1000, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).Read(0xc000909680, 0xc000748000, 0x1000, 0x1000, 0xc000941c88, 0x1284665, 0xc0007b4de0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1526 +0x7b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: bufio.(*Reader).fill(0xc000451380) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/bufio/bufio.go:100 +0x10f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: bufio.(*Reader).Peek(0xc000451380, 0x1, 0x0, 0x0, 0x1, 0xc0007c9c00, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/bufio/bufio.go:138 +0x4f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).readLoop(0xc000909680) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1679 +0x1a3 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by net/http.(*Transport).dialConn Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1359 +0xae8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 493 [running]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine running on other thread; stack unavailable Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/inputsource/udp.(*Server).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:83 +0x172 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 1154 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.(*persistConn).writeLoop(0xc00bdab8c0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1978 +0x113 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by net/http.(*Transport).dialConn Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/transport.go:1360 +0xb0d Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 952 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Run(0xc0007b65a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:147 +0x145 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Start.func1(0xc0007929b0, 0xc0007b65a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:132 +0x5b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input.(*Runner).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:125 +0x16b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 896 [IO wait, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.runtime_pollWait(0x7f56b8d96418, 0x72, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/netpoll.go:182 +0x56 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).wait(0xc0008ae098, 0x72, 0x2800, 0x2800, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).waitRead(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*FD).ReadFrom(0xc0008ae080, 0xc003cce000, 0x2800, 0x2800, 0x0, 0x0, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_unix.go:219 +0x194 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*netFD).readFrom(0xc0008ae080, 0xc003cce000, 0x2800, 0x2800, 0x30d9518, 0xc000633e60, 0xc000633e50, 0x1308f6d, 0xc0008ae080) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/fd_unix.go:208 +0x5b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*UDPConn).readFrom(0xc0007c0070, 0xc003cce000, 0x2800, 0x2800, 0xbf71c37e8090aec9, 0x45ee860cc2, 0x4e5a700, 0xe9) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/udpsock_posix.go:47 +0x6a Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*UDPConn).ReadFrom(0xc0007c0070, 0xc003cce000, 0x2800, 0x2800, 0x0, 0x0, 0x4e5a700, 0xc000633fa8, 0xc000633fa8) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/udpsock.go:121 +0x5e Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/inputsource/udp.(*Server).run(0xc0007ed7c0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:106 +0x126 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/inputsource/udp.(*Server).Start.func1(0xc0007ed7c0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:85 +0x57 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/inputsource/udp.(*Server).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:83 +0x172 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 1556 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.setRequestCancel.func3(0x0, 0xd467c455f0, 0xc0007a4000, 0xc01bd2c59c, 0xc00bd91140) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:321 +0xcf Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by net/http.setRequestCancel Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:320 +0x251 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 953 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Run(0xc0007b6de0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:147 +0x145 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Start.func1(0xc000792a00, 0xc0007b6de0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:132 +0x5b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input.(*Runner).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:125 +0x16b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 1124 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input/log.(*Log).wait(0xc00023aa10) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/log.go:195 +0xbc Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input/log.(*Log).Read(0xc00023aa10, 0xc02ff52000, 0x4000, 0x4000, 0x1, 0xffffffffffffffff, 0x4b3) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/log.go:108 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/reader/readfile.(*LineReader).advance(0xc000778600, 0x4b3, 0x500) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/readfile/line.go:133 +0x12f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/reader/readfile.(*LineReader).Next(0xc000778600, 0x4b3, 0x500, 0x2fdd3ae, 0x3, 0xc03bb10500, 0x4b3) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/readfile/line.go:78 +0x2f Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/reader/readfile.EncoderReader.Next(0xc000778600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/readfile/encode.go:53 +0x63 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/reader/readfile.(*StripNewline).Next(0xc00084a870, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/readfile/strip_newline.go:50 +0x6b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/reader/readfile.(*TimeoutReader).Next.func1(0xc00005b840) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/readfile/timeout.go:70 +0x48 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/reader/readfile.(*TimeoutReader).Next Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/reader/readfile/timeout.go:68 +0x287 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 954 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Run(0xc0007b75c0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:147 +0x145 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input.(*Runner).Start.func1(0xc000792b70, 0xc0007b75c0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:132 +0x5b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input.(*Runner).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/input.go:125 +0x16b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 1123 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/input/log.(*Harvester).Run.func2(0xc000286780, 0xc00016e8a0, 0x11) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:239 +0xca Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/input/log.(*Harvester).Run Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:232 +0x1d2 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 1122 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dc00, 0xc000858090, 0xc0000bf440) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 1058 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc0007b6b50) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 1059 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dbc0, 0xc000777770, 0xc0007c94a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 1060 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc0007b7330) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 1061 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dbc0, 0xc000972de0, 0xc0007c94a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 1114 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc000161810) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 1115 [select, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x340dbc0, 0xc0008c8150, 0xc0007c94a0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.CloseOnSignal Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 955 [IO wait, 2 minutes]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.runtime_pollWait(0x7f56b8d964e8, 0x72, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/runtime/netpoll.go:182 +0x56 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).wait(0xc0007a0298, 0x72, 0x2800, 0x2800, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*pollDesc).waitRead(...) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: internal/poll.(*FD).ReadFrom(0xc0007a0280, 0xc003cb8000, 0x2800, 0x2800, 0x0, 0x0, 0x0, 0x0, 0x0) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/internal/poll/fd_unix.go:219 +0x194 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*netFD).readFrom(0xc0007a0280, 0xc003cb8000, 0x2800, 0x2800, 0x30d9518, 0xc001ef9e60, 0xc001ef9e50, 0x1308f6d, 0xc0007a0280) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/fd_unix.go:208 +0x5b Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*UDPConn).readFrom(0xc00081c0c0, 0xc003cb8000, 0x2800, 0x2800, 0xbf71c37e808b2425, 0x45ee808206, 0x4e5a700, 0xe9) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/udpsock_posix.go:47 +0x6a Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net.(*UDPConn).ReadFrom(0xc00081c0c0, 0xc003cb8000, 0x2800, 0x2800, 0x0, 0x0, 0x4e5a700, 0xc001ef9fa8, 0x13edd64) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/udpsock.go:121 +0x5e Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/inputsource/udp.(*Server).run(0xc0005b6480) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:106 +0x126 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/inputsource/udp.(*Server).Start.func1(0xc0005b6480) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:85 +0x57 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/inputsource/udp.(*Server).Start Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:83 +0x172 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 945 [chan receive]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: github.com/elastic/beats/filebeat/channel.SubOutlet.func1(0xc000858090, 0x340dbc0, 0xc0005e6c90) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:44 +0x150 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by github.com/elastic/beats/filebeat/channel.SubOutlet Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8 Dec 03 18:05:13 my.file.beat.com filebeat[30053]: goroutine 1405 [select]: Dec 03 18:05:13 my.file.beat.com filebeat[30053]: net/http.setRequestCancel.func3(0x0, 0xc01bdb2c90, 0xc0003ba550, 0xd597d66d8c, 0xc0007c8420) Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:321 +0xcf Dec 03 18:05:13 my.file.beat.com filebeat[30053]: created by net/http.setRequestCancel Dec 03 18:05:13 my.file.beat.com filebeat[30053]: /usr/local/go/src/net/http/client.go:320 +0x251 Dec 03 18:05:13 my.file.beat.com systemd[1]: filebeat.service holdoff time over, scheduling restart. Dec 03 18:05:13 my.file.beat.com systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch.. Dec 03 18:05:13 my.file.beat.com systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch.. ```
So I did remove the
add_fieldsfor the additionaldestination.geoand it worked again. At the time where filebeat was consuming that large amount of memory, I saw in htop that RAM and swap were 100% used, within 10-15 seconds, after that filebeat was restarted, the RAM was emptied and it started again. Since then it looks like thisSo to conclude:
add_fieldsoperations, filebeat may break?