Time field name for Grafana

domainaware / parsedmarc

A Python package and CLI for parsing aggregate and forensic DMARC reports

https://domainaware.github.io/parsedmarc/

Apache License 2.0

984 stars 212 forks source link

Time field name for Grafana #72

Closed bhozar closed 5 years ago

bhozar commented 5 years ago

Hi. We tend to use Grafana for all our dashboards rather than Kibana, so I was attempting to setup parsedmarc and build your existing dashboard into Grafana, but I have falled at the first hurdle. I am unable to setup a data source as there is no time field in the index. I did attempt to use date_range. All other indexes I have uses @timestamp.

Any ideas how I resolve it or am I stuck having to install Kibana somewhere?

seanthegeek commented 5 years ago

Hi,

I looked at grafana when I first started this project, as I would have preferred to use it instead of Kibana too. Unfortunately, Grafana doesn't have a way of understanding date ranges in data that has already been aggregated. DMARC aggregate reports have a starting timestamp and ending timestamp, so you're stuck with Kibana until Grafana supports date ranges in data Fields. Please bring this up with their dev team. I would love to see it supported.

bhozar commented 5 years ago

Despite getting the error message in Grafana it did actually work and I was able to build the dashboard successfully in Grafana. Comparing like for like with Kibana it seems to be correct. Also setup the forensic dashboard, but as a different data source as the time field name is different.

Did you have a request already open with the Grafana devs to add date ranges as an option?

seanthegeek commented 5 years ago

I haven't had a chance to. If you could, that would be great.

Also, is there any way you could export the dashboard configuration so it can be shared and imported, similar to what I did with the Kibana dashboards?

seanthegeek commented 5 years ago

Hmm. I just tried, and if I go to Explore in Grafana, I still get errors and no data :(

For aggregate:

"Error connecting to datasource: No date field named date_range found"

For forensic:

"Error connecting to datasource: No date field named arrival_date found"

No idea what I'm doing differently :(

bhozar commented 5 years ago

Grafana-DMARC Summary-1556030024908.zip

I've exported the dashboard from Grafana and put it in the Zip (attached). Obviously a work in progress.

Data source setup is different for each ES index:

bhozar commented 5 years ago

I should mention you need Word Map and Pie Chart plugins for the Grafana dashboard.

bhozar commented 5 years ago

Grafana-DMARC Summary-1556030024908.zip Like I said, work in progress. Fixed a few things on the dashboard. I only have the single forensic report sample from you to try out the forensic report, but think I have the dashboard correct.

bhozar commented 5 years ago

I finally received some legitimate forensic reports so have updated the dashboard further. Also fixed a few other issues. Grafana-DMARC_Reports-1556528576252.zip Let me know if you see any issues, but I've been using the dashboard for a while to pickup and fix various issues. So much easier than reading the raw reports, so thanks for Parsedmarc, it's a great application.

seanthegeek commented 5 years ago

Added to the repo under the grafana directory. Thanks!

sledzik1984 commented 3 years ago

@seanthegeek did you manage to make it work? When I'm setting Elasticsearch datasource in Grafana it throws error No date field named date_range

sledzik1984 commented 3 years ago

Ok... I am running Grafana 8.0.6 and it will not accept Index name as:

[dmarc_ag*-]YYYY-MM-DD

but will run ok with Index name like:

[dmarc_aggregate-]YYYY-MM-DD