Captured data does not flow from Hedgedog to Malcolm

[Zokol]

Describe the bug I see PCAPs in Hedgedog local disk Hedgedog can see Malcolm No arkime sessions or traffic is visible in Arkime

To Reproduce Steps to reproduce the behavior:

1. Follow guide here
2. Add traffic to network monitored by Hedgehog
3. Check that pcap-files are visible in /home/sensor/net_cap
4. Open Arkime in Malcolm
5. See no traffic

Expected behavior I would expect to see the contents of the pcap-files in Hedgedog to be visible in Malcolm/Arkime dashboard after short while.

Malcolm Version:

• Version v24.05.0

How are you running Malcolm? ISO installed (Malcolm on VM, Hedgedog on Raspberry Pi 4 8GB)

[Zokol]

I noticed that arkime-container has no ports open for the Hedgedog to access. Could this be the reason for it not to be able to upload the PCAPs?

$ docker ps
CONTAINER ID   IMAGE                                                COMMAND                  CREATED      STATUS                 PORTS                                          NAMES
b0a82dd0bcca   ghcr.io/idaholab/malcolm/nginx-proxy:24.05.0         "/sbin/tini -- /usr/…"   5 days ago   Up 2 hours (healthy)   0.0.0.0:443->443/tcp, 0.0.0.0:9200->9200/tcp   malcolm-nginx-proxy-1
f2009539be92   ghcr.io/idaholab/malcolm/dashboards:24.05.0          "/usr/bin/tini -- /u…"   5 days ago   Up 2 hours (healthy)   5601/tcp                                       malcolm-dashboards-1
9b25c1a079e4   ghcr.io/idaholab/malcolm/netbox:24.05.0              "/usr/bin/tini -- /u…"   5 days ago   Up 2 hours (healthy)   9001/tcp                                       malcolm-netbox-1
ef90e1f3675f   ghcr.io/idaholab/malcolm/pcap-monitor:24.05.0        "/usr/bin/tini -- /u…"   5 days ago   Up 2 hours (healthy)   30441/tcp                                      malcolm-pcap-monitor-1
b3339fcea600   ghcr.io/idaholab/malcolm/logstash-oss:24.05.0        "/usr/bin/tini -- /u…"   5 days ago   Up 2 hours (healthy)   9001/tcp, 0.0.0.0:5044->5044/tcp, 9600/tcp     malcolm-logstash-1
fe6966bf60eb   ghcr.io/idaholab/malcolm/zeek:24.05.0                "/usr/bin/tini -- /u…"   5 days ago   Up 2 hours (healthy)                                                  malcolm-zeek-1
ca4cca21f17a   ghcr.io/idaholab/malcolm/arkime:24.05.0              "/usr/bin/tini -- /u…"   5 days ago   Up 2 hours (healthy)   8000/tcp, 8005/tcp, 8081/tcp                   malcolm-arkime-1
adba28348adb   ghcr.io/idaholab/malcolm/dashboards-helper:24.05.0   "/sbin/tini -- /usr/…"   5 days ago   Up 2 hours (healthy)   28991/tcp                                      malcolm-dashboards-helper-1
5efd029a2321   ghcr.io/idaholab/malcolm/opensearch:24.05.0          "/usr/bin/tini -- /u…"   5 days ago   Up 2 hours (healthy)   9200/tcp, 9300/tcp, 9600/tcp, 9650/tcp         malcolm-opensearch-1
a001cec5f055   ghcr.io/idaholab/malcolm/zeek:24.05.0                "/usr/bin/tini -- /u…"   5 days ago   Up 2 hours                                                            malcolm-zeek-live-1
97971de84bf1   ghcr.io/idaholab/malcolm/filebeat-oss:24.05.0        "/usr/bin/tini -- /u…"   5 days ago   Up 2 hours (healthy)   0.0.0.0:5045->5045/tcp                         malcolm-filebeat-1
b6b25026ad9a   ghcr.io/idaholab/malcolm/suricata:24.05.0            "/usr/bin/tini -- /u…"   5 days ago   Up 2 hours                                                            malcolm-suricata-live-1
92aa4ffe010f   ghcr.io/idaholab/malcolm/file-monitor:24.05.0        "/usr/bin/tini -- /u…"   5 days ago   Up 2 hours (healthy)   3310/tcp, 8440/tcp                             malcolm-file-monitor-1
a7056ab282ec   ghcr.io/idaholab/malcolm/postgresql:24.05.0          "/sbin/tini -- /usr/…"   5 days ago   Up 2 hours (healthy)   5432/tcp                                       malcolm-netbox-postgres-1
3101899d6255   ghcr.io/idaholab/malcolm/redis:24.05.0               "/sbin/tini -- /usr/…"   5 days ago   Up 2 hours (healthy)   6379/tcp                                       malcolm-netbox-redis-1
97c490466a96   ghcr.io/idaholab/malcolm/redis:24.05.0               "/sbin/tini -- /usr/…"   5 days ago   Up 2 hours (healthy)   6379/tcp                                       malcolm-netbox-redis-cache-1
b2aab0708bb6   ghcr.io/idaholab/malcolm/freq:24.05.0                "/usr/bin/tini -- /u…"   5 days ago   Up 2 hours (healthy)   10004/tcp                                      malcolm-freq-1
77b40795fd95   ghcr.io/idaholab/malcolm/pcap-capture:24.05.0        "/usr/bin/tini -- /u…"   5 days ago   Up 2 hours                                                            malcolm-pcap-capture-1
97bf3b101e65   ghcr.io/idaholab/malcolm/htadmin:24.05.0             "/usr/bin/tini -- /u…"   5 days ago   Up 2 hours (healthy)   80/tcp                                         malcolm-htadmin-1
3bb803dea3c8   ghcr.io/idaholab/malcolm/file-upload:24.05.0         "/usr/bin/tini -- /u…"   5 days ago   Up 2 hours (healthy)   80/tcp, 127.0.0.1:8022->22/tcp                 malcolm-upload-1
f86b975aa52d   ghcr.io/idaholab/malcolm/api:24.05.0                 "/usr/bin/tini -- /u…"   5 days ago   Up 2 hours (healthy)   5000/tcp                                       malcolm-api-1
56214a1237a4   ghcr.io/idaholab/malcolm/arkime:24.05.0              "/usr/bin/tini -- /u…"   5 days ago   Up 2 hours                                                            malcolm-arkime-live-1
f07d519e74a5   ghcr.io/idaholab/malcolm/suricata:24.05.0            "/usr/bin/tini -- /u…"   5 days ago   Up 2 hours (healthy)                                                  malcolm-suricata-1

[mmguero]

No, as arkime-capture on hedgehog writes directly to opensearch (port 9200).

Hedgehog is not just globally broken, as we have a lot of regression and acceptance testing across a variety of platforms before release. So we just need to find out what's wrong with your system/configuration.

Checking /opt/sensor/sensor_ctl/logs as you mentioned in #451 is a good idea. You should be able to see the arkime capture logs and filebeat logs there, if they're complaining about the connection you'd see it there.

Also, looking at the logs on Malcolm (./scripts/logs) there may be some indications there of what's going on (authentication issues, maybe)?

[Zokol]

It seems that I was fooled by Arkime's default 1 hour window. The actual issue was that Hedgehog is three hours behind Malcolm's clock, so the data it sent was not visible until I changed the timeline filter.

However, I see no reason why Hedgehog is actually running different time, as I configured it to fetch time from Malcolm's IP in setup phase. This still might be something that needs some looking into.

Just to document the steps I took investigating the issue, in case it would be helpful for someone in the future:

• Verified that Hedgehog has open connection to Malcolm's port 9200 using netstat
• Verified that arkime-capture log in /opt/sensor/sensor-ctl/log contains connections to Malcolm's port 9200 with word bulk and code 200, which would indicate that the captured data is successfully sent to opensearch
• Opened stats-tab in Malcolm's Arkime web UI and saw that the node has actually sent several packets to Malcolm, from which I started to suspect that the issue might actually reside on time difference between the machines

[mmguero]

Hmm, okay, I somewhat recently (within the last 6 months or so) changed Hedgehog's code so that it should sync immediately upon setting up time synchronization from Malcolm using htpdate, but I wonder if you already had services running when it happened if it woudn't take effect until they've been restarted. I'll double-check that. Thanks for the info.