search

molu8bits / squid-filebeat-kibana

Filebeat module for Squid access.log + Kibana dashboards. ELK 7.x
Apache License 2.0
17 stars 14 forks source link

Could not locate that index-pattern-field (id: squid.access.geoip.location) #11

Open gbarre opened 3 years ago

gbarre commented 3 years ago

I've just installed ELK 7.11 (noob with it). I have squid3 for a while. I've installed filebeat on my squid server, using the files from this repo.

Everything seems good except geoip. What did I missed ?

molu8bits commented 3 years ago

There is a part in pipeline which gets IP address for destination and transform it into geoip fields. Usually the problem happens when filebeat instance already sent something to store as a filebeat index without applying at first metadata structure (fields.yml from thie repo). That causes that when correct filebeat entry with squid data enters elasticsearch it cannot modify existing geoip structure. Only saves informations as string/text etc. String/text is not correct type to present geoip data, hence probably error found on visualization. The simplest way to fix it is usually setup a new clean elasticsearch cluster and send filebeats squid logs to it (do not start other clients writing to that ES). Other way would be to remove all the indexes, metadata, schemas, pipelines from existing elastichsearch which is not quite simple.

numptyboy commented 3 years ago

Hi,

I have same problem as @gbarre, except I'm running 7.12.

I've deleted everything in /var/lib/elasticsearch/* and installed this module as per instructions - that didn't work.

Ive tried using the version of fields.yml provided here, and integrating the squid section into the version supplied with filebeat 7.12 - again, same result.

There is a squid module provided with 7.12 - i'm not sure if this is new but I've removed everything to do with it to be safe.

What I'm seeing is, regardless of what I do, the squid.access.geoip.location value is being interpreted as two Integers [.lat and .lon] rather than a geoip.

Hope this makes sense and helps with diagnosis.

I'm sure I must be doing something really dumb here, but any help appreciated.

Thanks

ChIP

molu8bits commented 3 years ago

@numptyboy I guess that 7.12 brings some overlapping settings which are not compatible with my squid module. I didn't know that they have been developing other squid module - it might be a reason why nobody from ELK team didn't have time to even look at my PR for a few weeks last year. I suggest to use older version of Filebeat/Elastic/Kibana with my module or switch completely to module provided by newest filebeat and build your own dashboard based. In future I'm rather going to switch for using Loki which is much more light solutions for simple log patterns like Squid.

numptyboy commented 3 years ago

Thanks for taking the time to get back to me Sir.

Much appreciated.

Now I know I'm not just being dumb I'll continue to smack my head against this a little longer. If I make any useful progress I'll let you know.

Hang in there ...

ChIP