Dashboards ProblemHomeMonitor 4.5 with Enterprise 6.3

[machinasdei]

Hi, I've uninstalled Splunk Enterprise, and Homemonitor and reformatted my Mac Mini running El Capitan to ensure a CLEAN Slate to work with. So its Bare bones vanilla machine, with not much on it, just to solely use as a server and start clean with Splunk Enterprise 6.3 and HomeMonitor 4.5. I've also installed the Splunk App for Stream in which ALL dashboards work.

Thanks for upgrading HomeMonitor. I'd love to get it to work one day. I've followed All videos, blogs, resources of yours that I can find, and I'm still having issues getting the Dashboards to populate. I AM getting data from 514 and running as Root. I used the input screen you included in 4.5 HomeMonitor to ensure I have it correctly as well as checked the props and transforms file to make sure they all match to your documentation - they do. and data is showing in home monitor search. (I've snapshot my Data Inputs and a Search below to check for reference)

On Overview Dashboards: I get a partial dashboard on the Home Network Overview Dashboard where it only shows: As you see, it is missing any Bandwidth data (the Data Input is selected correctly for Linux/Mac) and missing Intrusion Detection, Inbound, Outbound, and Blocked Events data.

I get a partial dashboard on the Bandwidth Overview Dashboard;

As you can see on the Bandwidth Overview Dashboard--- Average Downloads vs. Average Uploads is NOT populating, and you see the Stream fields which are are populating with no issues.

On Overview Dashboards: Check for Intrusions in your network - Sourcetype=fios - No results found Blocked Traffic - - Sourcetype=fios - No results found all Panels Network Event Overview -- - Sourcetype=fios - No results found all Panels Network Overview In bound - - Sourcetype=fios - No results found all Panels Network Overview Out bound -- - Sourcetype=fios - No results found all Panels

Device Specific Panel - fios - This one actually seems OK - i didn't make config changes so its reporting correctly it appears;

Experimental Views Dashboards: Home Tag cloud- Looks like the panels are fed from Stream - No results found all Panels Force Directed - Looks like the panels are fed from Stream - No results found all Panels Sankey Netwok Diagram - Looks like the panels are fed from Stream - No results found all Panels

Map of Connections Dashboard Panels: Source for these are fios and are not populating.

I did notice that any time from the time picker I chose did not yield any results as well......at one point it was only pulling some partial data when 'all time' was chosen' but that was before the reinstall, now the Dashboards pull the above results.

Just for completeness, here is my what my Data Input looks like; I assume that the Data inputs are correct based on everything i've seen and tried.....the source type override field still seems clear as mud as what should actually be in there, or not be in there, leave blank? as well as the set host button to choose, in this case i just left the default as thats what the set up guide did for you.

I know that this is a labor of love project for you and you do it on your own time. I greatly appreciate your work and love the potential this has when it works. Any time you have to think on assisting to remedy the issues is certainly appreciated.

Thanks man

[amiracle]

Great work so far getting to this point and thank you for documenting all of your steps. I think I know what the issue might be around the missing bandwidth data and non-populating dashboards. Let me know when we can go through a Webex and knock this out.  Thanks,Kamilo

Kamilo Amir (240)466-1385

On Thu, Mar 17, 2016 at 7:00 PM -0700, "machinasdei" notifications@github.com wrote:

Hi,

I've uninstalled Splunk Enterprise, and Homemonitor and reformatted my Mac Mini running El Capitan to ensure a CLEAN Slate to work with. So its Bare bones vanilla machine, with not much on it, just to solely use as a server and start clean with Splunk Enterprise 6.3 and HomeMonitor 4.5. I've also installed the Splunk App for Stream in which ALL dashboards work.

Thanks for upgrading HomeMonitor. I'd love to get it to work one day. I've followed All videos, blogs, resources of yours that I can find, and I'm still having issues getting the Dashboards to populate. I AM getting data from 514 and running as Root. I used the input screen you included in 4.5 HomeMonitor to ensure I have it correctly as well as checked the props and transforms file to make sure they all match to your documentation - they do. and data is showing in home monitor search. (I've snapshot my Data Inputs and a Search below to check for reference)

On Overview Dashboards:

I get a partial dashboard on the Home Network Overview Dashboard where it only shows:

As you see, it is missing any Bandwidth data (the Data Input is selected correctly for Linux/Mac) and missing Intrusion Detection, Inbound, Outbound, and Blocked Events data.

I get a partial dashboard on the Bandwidth Overview Dashboard;

As you can see on the Bandwidth Overview Dashboard--- Average Downloads vs. Average Uploads is NOT populating, and you see the Stream fields which are are populating with no issues.

On Overview Dashboards:

Check for Intrusions in your network - Sourcetype=fios - No results found

Blocked Traffic - - Sourcetype=fios - No results found all Panels

Network Event Overview -- - Sourcetype=fios - No results found all Panels

Network Overview In bound - - Sourcetype=fios - No results found all Panels

Network Overview Out bound -- - Sourcetype=fios - No results found all Panels

Device Specific Panel - fios - This one actually seems OK - i didn't make config changes so its reporting correctly it appears;

Experimental Views Dashboards:

Home Tag cloud- Looks like the panels are fed from Stream - No results found all Panels

Force Directed - Looks like the panels are fed from Stream - No results found all Panels

Sankey Netwok Diagram - Looks like the panels are fed from Stream - No results found all Panels

Map of Connections Dashboard Panels:

Source for these are fios and are not populating.

I did notice that any time from the time picker I chose did not yield any results as well......at one point it was only pulling some partial data when 'all time' was chosen' but that was before the reinstall, now the Dashboards pull the above results.

Just for completeness, here is my what my Data Input looks like;

I assume that the Data inputs are correct based on everything i've seen and tried.....the source type override field still seems clear as mud as what should actually be in there, or not be in there, leave blank? as well as the set host button to choose, in this case i just left the default as thats what the set up guide did for you.

I know that this is a labor of love project for you and you do it on your own time. I greatly appreciate your work and love the potential this has when it works. Any time you have to think on assisting to remedy the issues is certainly appreciated.

Thanks man

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub

[machinasdei]

Hey man thanks I should be able to do something after seven tonight or Saturday night or Sunday night and of course whatever works for you thanks let me know much appreciated!

Sent from my iPhone 6S

On Mar 18, 2016, at 8:06 AM, Kamilo Amir notifications@github.com wrote:

Great work so far getting to this point and thank you for documenting all of your steps. I think I know what the issue might be around the missing bandwidth data and non-populating dashboards. Let me know when we can go through a Webex and knock this out. Thanks,Kamilo

Kamilo Amir (240)466-1385

On Thu, Mar 17, 2016 at 7:00 PM -0700, "machinasdei" notifications@github.com wrote:

Hi,

I've uninstalled Splunk Enterprise, and Homemonitor and reformatted my Mac Mini running El Capitan to ensure a CLEAN Slate to work with. So its Bare bones vanilla machine, with not much on it, just to solely use as a server and start clean with Splunk Enterprise 6.3 and HomeMonitor 4.5. I've also installed the Splunk App for Stream in which ALL dashboards work.

Thanks for upgrading HomeMonitor. I'd love to get it to work one day. I've followed All videos, blogs, resources of yours that I can find, and I'm still having issues getting the Dashboards to populate. I AM getting data from 514 and running as Root. I used the input screen you included in 4.5 HomeMonitor to ensure I have it correctly as well as checked the props and transforms file to make sure they all match to your documentation - they do. and data is showing in home monitor search. (I've snapshot my Data Inputs and a Search below to check for reference)

On Overview Dashboards:

I get a partial dashboard on the Home Network Overview Dashboard where it only shows:

As you see, it is missing any Bandwidth data (the Data Input is selected correctly for Linux/Mac) and missing Intrusion Detection, Inbound, Outbound, and Blocked Events data.

I get a partial dashboard on the Bandwidth Overview Dashboard;

As you can see on the Bandwidth Overview Dashboard--- Average Downloads vs. Average Uploads is NOT populating, and you see the Stream fields which are are populating with no issues.

On Overview Dashboards:

Check for Intrusions in your network - Sourcetype=fios - No results found

Blocked Traffic - - Sourcetype=fios - No results found all Panels

Network Event Overview -- - Sourcetype=fios - No results found all Panels

Network Overview In bound - - Sourcetype=fios - No results found all Panels

Network Overview Out bound -- - Sourcetype=fios - No results found all Panels

Device Specific Panel - fios - This one actually seems OK - i didn't make config changes so its reporting correctly it appears;

Experimental Views Dashboards:

Home Tag cloud- Looks like the panels are fed from Stream - No results found all Panels

Force Directed - Looks like the panels are fed from Stream - No results found all Panels

Sankey Netwok Diagram - Looks like the panels are fed from Stream - No results found all Panels

Map of Connections Dashboard Panels:

Source for these are fios and are not populating.

I did notice that any time from the time picker I chose did not yield any results as well......at one point it was only pulling some partial data when 'all time' was chosen' but that was before the reinstall, now the Dashboards pull the above results.

Just for completeness, here is my what my Data Input looks like;

I assume that the Data inputs are correct based on everything i've seen and tried.....the source type override field still seems clear as mud as what should actually be in there, or not be in there, leave blank? as well as the set host button to choose, in this case i just left the default as thats what the set up guide did for you.

I know that this is a labor of love project for you and you do it on your own time. I greatly appreciate your work and love the potential this has when it works. Any time you have to think on assisting to remedy the issues is certainly appreciated.

Thanks man

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub

[amiracle]

Able to fix this by correcting some of the field extractions ( see http://amiracle19.blogspot.com/2016/02/adding-dd-wrt-sourcetype.html for guide on how to fix them for your router.) Fixed the bandwidth script to use the hardcoded path to the script.

[machinasdei]

Thanks for your help the other night again. Maybe I'm a bit slow or something, most everything is working but the following. I've now been having issues with 2 things - one that will fix many --on 1. Dashboards that is anything Port related -which is used in a few of the dashboards, and 2. with the Maps of Connections Dashboards all together.

I apologize in advance, as I haven't been able to get much farther. I've been playing with the field extractor quite a bit and messing with the props.conf file in the local directory only. (the src_port and dest_port are already created so they are grayed out, nothing to do there really that I can see) maybe its using some other name other than src_port or dest_port now? not sure.

So when we were working the other night, I noticed that we left the props.conf file with simply two extracted fields which were EXTRACT-direction = ^(?:[^ \n]* ){9}(?P\w+) EXTRACT-action = ^(?:[^ \n]* ){10}(?P\w+)

and most things worked at that point, I messed with it a little and got the blocked ports to populate somehow using all of the following below;

My Props.conf file looks like this now :

EXTRACT-direction = ^(?:[^ \n]* ){9}(?P\w+) EXTRACT-protocol = ^[^(\n]((?P\w+) EXTRACT-action = ^(?:[^ \n] ){10}(?P\w+) EXTRACT-state = ^[^]\n]_]\s+(?P\w+\s+\w+) EXTRACT-natip = ^[^>\n]>(?P\d+.\d+.\d+.\d+) EXTRACT-interface = ^[^/\n]/\w+\w+\d+\s+(?P\w+) EXTRACT-reason = ^(?:[^:\n]:){5}\s+\w+\s+(?P[^ ]+) EXTRACT-srcip = ^[^(\n](\w+\s+(?P\d+.\d+.\d+.\d+) EXTRACT-srcport = ^(?:[^.\n].){4}\d+:(?P\d+) EXTRACT-destip = ^[^>\n]>(?P\d+.\d+.\d+.\d+) EXTRACT-destport = ^(?:[^.\n].){4}\d+:(?P\d+) LOOKUP-fios = action_lookup action OUTPUTNEW action2 LOOKUP-rdns = dnsLookup ip AS dest_ip OUTPUTNEW host as rdns_host

---

I guess i still don't understand the spunk logic here in these expressions, although I thought i did from referencing quick reference guides...... been trying to figure out where action2 is defined as well.

I thought I was on the right track with the extractions and being able to get the blocked stuff up. the src_port and dest_port have me stumped.

thanks again.

[machinasdei]

i think i figured it out finally.
I tried out the Extractor again and recreated the src_port and the dest_port expressions and noticed one number in the middle changed ( i still don't know what the heck it is - perhaps you can tell me?)

so from the above props.conf: EXTRACT-src_port = ^(?:[^.\n].){4}\d+:(?P\d+) EXTRACT-dest_port = ^(?:[^.\n].){4}\d+:(?P\d+)

--which both have the number 4 in them ....i guess its some sort of unique identifier

and i just changed it to;

EXTRACT-srcport = ^(?:[^.\n].){3}\d+:(?P\d+) EXTRACT-destport = ^(?:[^.\n].){6}\d+:(?P\d+)

so the 4's were replaced with the 3 and 6 and it works.

so that fixed the first issue with the Ports! all good there now!

The Maps of Connections Dashboards is the only issue that remains now! it appears that this is pulling data from the source_ip and Action2 and implication fields......not too familiar with the later two in spunk yet......

[amiracle]

Action2 is a lookup that looks at action and normalizes it to BLOCK and ACCEPT. Since other routers use allow, deny etc., this just makes my life easier. I hope that helps. 

Kamilo Amir (240)466-1385

On Sun, Mar 20, 2016 at 12:45 PM -0700, "machinasdei" notifications@github.com wrote:

i think i figured it out finally.

I tried out the Extractor again and recreated the src_port and the dest_port expressions and noticed one number in the middle changed ( i still don't know what the heck it is - perhaps you can tell me?)

so from the above props.conf:

EXTRACT-src_port = ^(?:[^. ].){4}\d+:(?P\d+)

EXTRACT-dest_port = ^(?:[^. ].){4}\d+:(?P\d+)

--which both have 4 in them ....i guess its some sort of unique identifier

and i just changed it to;

EXTRACT-src_port = ^(?:[^. ].){3}\d+:(?P\d+)

EXTRACT-dest_port = ^(?:[^. ].){6}\d+:(?P\d+)

so that fixed the first issue with the Ports! all good there now!

The Maps of Connections Dashboards is the only issue that remains now! it appears that this is pulling data from the source_ip and Action2 and implication fields......not too familiar with the later two in spunk yet......

— You are receiving this because you modified the open/close state. Reply to this email directly or view it on GitHub

[machinasdei]

cool, i understand the lookup tables, just didn't think to look for it there, but anyhow, I found it and see it and understand it now.

SO do you think that I should be able to fix the Maps of Connections Dashboards the same way as the other Dashboards issues like I/we have been doing? If not, please drop a hint :)

Thanks!

[machinasdei]

Ah never mind that last comment about the Maps, i got them to work for the first time, bonehead mistake - forgot to enable the last field of ignore private networks, it all works now.....maybe one day i'll get a switch and use the Experimental views!

Again, awesome App, and I plan on enhancing it further too, very exciting!

Thanks man!