Multi-Wan: Routing Breaks if the autogenerated gateways are edited

ilikenwf commented 8 months ago

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

[X] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
[X] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

I have two WAN interfaces, each get their IP via DHCP. If I edit the system generated Gateways for them to enable monitor IP addresses, packets appear to go out, but they do not return to the requesting machine on the LAN. Likewise, if I don't enable monitoring, and then instead just setup a group, I have connectivity again, with the relevant policy based routing rules in place.

In addition, for vlans that I allow cross communication on, I have to calculate and add static routes to the DHCP server to allow them to talk back.

Tip: to validate your setup was working with the previous version, use opnsense-revert (https://docs.opnsense.org/manual/opnsense_tools.html#opnsense-revert)

To Reproduce

Steps to reproduce the behavior:

Have two DHCP WAN interfaces enabled
Edit each of their generated gateways and enable monitoring ip addresses (or just hit save even)
Suddenly, no internet access for any of the clients, be it through a default gateway or group
In the firewall I see allowed outgoing requests, but the responses don't seem to be getting back to the LAN client machines

Expected behavior

Working monitoring and LAN failover, with LAN clients able to access the internet.

Environment

Software version used and hardware type if relevant, e.g.:

OPNsense 24.1_1-amd64 FreeBSD 13.2-RELEASE-p9 OpenSSL 3.0.12

ilikenwf commented 8 months ago

This is really frustrating as it means I can't setup monitor IP addresses and can only use the other options for routing.

fichtner commented 8 months ago

TBH, it sounds like a setup issue. Did you enable default gateway swiching?

Cheers, Franco

ilikenwf commented 8 months ago

Yes, otherwise the group with both wans as tier 1 wouldn't switch IPs occasionally in nonsticky round robin mode? That part works so I know the actual WAN switching works, however the gateways just break with monitoring turned on.

I have to delete the entire gateway and interface and then reimport the config to even be able to get them to route again.

Unfortunately, unchecking the disable box and setting a monitor IP is all it takes to nuke all internet access on either WAN. One WAN IP is static but assigned by DHCP, the other is also DHCP. Not sure if that affects it at all:

fichtner commented 8 months ago

It sounds way too fragile to be a sole software bug. System logs might help but I fear this will quickly exceed community support time.

ilikenwf commented 8 months ago

I thought community support was the forums and real bugs happen here?

fichtner commented 8 months ago

Bugs usually indicate a code line or function that is misbehaving. Unless we have that confirmed it requires support time. In this case the chance is at least 50% setup issue due to lack of diagnostics output and you also have to factor in the time to analyse and reproduce.

ilikenwf commented 8 months ago

I can reproduce the issue and share the syslog if that would assist. I assume that is the /etc/syslog/latest.log file? I'll just anonymize the private parts.

ilikenwf commented 8 months ago

What's interesting is this only happens on modification of the gateway configs themselves, so I'm not sure if it is a dpinger issue or some other config file(s) get touched there?

What is interesting is that the firewall itself retains internet access, and usually packets going out from clients go out but then never get back to the clients who requested them it seems?

Feels like a routing problem. I'd be happy to share more with whoever might want to dig into this as it feels like a routing issue.

ilikenwf commented 8 months ago

Could it be related to this?

https://github.com/opnsense/core/issues/7191

ilikenwf commented 8 months ago

Here's the gateway log:

<12>1 2024-02-17T01:21:15-06:00 fw.local.lan dpinger 31322 - [meta sequenceId="1"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 9.9.9.9  bind_addr x.x.x.x  identifier "WAN_TMOBILE_DHCP "
<12>1 2024-02-17T01:22:52-06:00 fw.local.lan dpinger 31322 - [meta sequenceId="1"] exiting on signal 15
<12>1 2024-02-17T02:27:18-06:00 fw.local.lan dpinger 94265 - [meta sequenceId="1"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 9.9.9.9  bind_addr x.x.x.x  identifier "WAN_TMOBILE_DHCP "
<12>1 2024-02-17T02:27:21-06:00 fw.local.lan dpinger 94265 - [meta sequenceId="2"] exiting on signal 15
<12>1 2024-02-17T02:27:21-06:00 fw.local.lan dpinger 1323 - [meta sequenceId="3"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 9.9.9.9  bind_addr x.x.x.x  identifier "WAN_TMOBILE_DHCP "
<165>1 2024-02-17T02:27:21-06:00 fw.local.lan dpinger 94898 - [meta sequenceId="4"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2024-02-17T02:27:22-06:00 fw.local.lan dpinger 1323 - [meta sequenceId="5"] exiting on signal 15
<12>1 2024-02-17T02:27:22-06:00 fw.local.lan dpinger 8051 - [meta sequenceId="6"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 9.9.9.9  bind_addr x.x.x.x  identifier "WAN_TMOBILE_DHCP "
<165>1 2024-02-17T02:27:22-06:00 fw.local.lan dpinger 94898 - [meta sequenceId="7"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2024-02-17T02:27:24-06:00 fw.local.lan dpinger 8051 - [meta sequenceId="8"] exiting on signal 15
<12>1 2024-02-17T02:27:24-06:00 fw.local.lan dpinger 15411 - [meta sequenceId="9"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 9.9.9.9  bind_addr x.x.x.x  identifier "WAN_TMOBILE_DHCP "
<165>1 2024-02-17T02:27:24-06:00 fw.local.lan dpinger 94898 - [meta sequenceId="10"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2024-02-17T02:27:24-06:00 fw.local.lan dpinger 15411 - [meta sequenceId="11"] exiting on signal 15
<12>1 2024-02-17T02:27:24-06:00 fw.local.lan dpinger 22400 - [meta sequenceId="12"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 9.9.9.9  bind_addr x.x.x.x  identifier "WAN_TMOBILE_DHCP "
<165>1 2024-02-17T02:27:24-06:00 fw.local.lan dpinger 94898 - [meta sequenceId="13"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2024-02-17T02:27:25-06:00 fw.local.lan dpinger 22400 - [meta sequenceId="14"] exiting on signal 15
<12>1 2024-02-17T02:27:25-06:00 fw.local.lan dpinger 28325 - [meta sequenceId="15"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 9.9.9.9  bind_addr x.x.x.x  identifier "WAN_TMOBILE_DHCP "
<165>1 2024-02-17T02:27:25-06:00 fw.local.lan dpinger 94898 - [meta sequenceId="16"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2024-02-17T02:27:25-06:00 fw.local.lan dpinger 28325 - [meta sequenceId="17"] exiting on signal 15
<12>1 2024-02-17T02:27:25-06:00 fw.local.lan dpinger 38813 - [meta sequenceId="18"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 9.9.9.9  bind_addr x.x.x.x  identifier "WAN_TMOBILE_DHCP "
<165>1 2024-02-17T02:27:25-06:00 fw.local.lan dpinger 94898 - [meta sequenceId="19"] Reloaded gateway watcher configuration on SIGHUP
<165>1 2024-02-17T02:27:28-06:00 fw.local.lan dpinger 94898 - [meta sequenceId="20"] ALERT: WAN_TMOBILE_DHCP (Addr: 9.9.9.9 Alarm: down -> none RTT: 54.5 ms RTTd: 5.0 ms Loss: 0.0 %)
<12>1 2024-02-17T02:28:27-06:00 fw.local.lan dpinger 38813 - [meta sequenceId="21"] exiting on signal 15
<12>1 2024-02-17T02:28:27-06:00 fw.local.lan dpinger 6310 - [meta sequenceId="22"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 1.1.1.1  bind_addr x.x.x.x  identifier "WAN_STARLINK_DHCP "
<12>1 2024-02-17T02:28:27-06:00 fw.local.lan dpinger 6704 - [meta sequenceId="23"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 9.9.9.9  bind_addr x.x.x.x  identifier "WAN_TMOBILE_DHCP "
<165>1 2024-02-17T02:28:27-06:00 fw.local.lan dpinger 94898 - [meta sequenceId="24"] Reloaded gateway watcher configuration on SIGHUP
<165>1 2024-02-17T02:28:29-06:00 fw.local.lan dpinger 94898 - [meta sequenceId="25"] ALERT: WAN_STARLINK_DHCP (Addr: 1.1.1.1 Alarm: down -> none RTT: 31.2 ms RTTd: 8.1 ms Loss: 0.0 %)
<12>1 2024-02-17T02:28:38-06:00 fw.local.lan dpinger 6310 - [meta sequenceId="26"] exiting on signal 15
<12>1 2024-02-17T02:28:38-06:00 fw.local.lan dpinger 6704 - [meta sequenceId="27"] exiting on signal 15
<12>1 2024-02-17T02:28:38-06:00 fw.local.lan dpinger 22756 - [meta sequenceId="28"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 1.1.1.1  bind_addr x.x.x.x  identifier "WAN_STARLINK_DHCP "
<12>1 2024-02-17T02:28:38-06:00 fw.local.lan dpinger 23568 - [meta sequenceId="29"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 9.9.9.9  bind_addr x.x.x.x  identifier "WAN_TMOBILE_DHCP "
<165>1 2024-02-17T02:28:38-06:00 fw.local.lan dpinger 94898 - [meta sequenceId="30"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2024-02-17T02:31:33-06:00 fw.local.lan dpinger 22756 - [meta sequenceId="1"] exiting on signal 15
<12>1 2024-02-17T02:31:33-06:00 fw.local.lan dpinger 23568 - [meta sequenceId="2"] exiting on signal 15
<12>1 2024-02-17T02:31:33-06:00 fw.local.lan dpinger 94139 - [meta sequenceId="3"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 1.1.1.1  bind_addr x.x.x.x  identifier "WAN_STARLINK_DHCP "
<12>1 2024-02-17T02:31:33-06:00 fw.local.lan dpinger 94462 - [meta sequenceId="4"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 9.9.9.9  bind_addr x.x.x.x  identifier "WAN_TMOBILE_DHCP "
<165>1 2024-02-17T02:31:33-06:00 fw.local.lan dpinger 94898 - [meta sequenceId="5"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2024-02-17T02:32:05-06:00 fw.local.lan dpinger 94139 - [meta sequenceId="6"] exiting on signal 15
<12>1 2024-02-17T02:32:05-06:00 fw.local.lan dpinger 94462 - [meta sequenceId="7"] exiting on signal 15
<12>1 2024-02-17T02:32:05-06:00 fw.local.lan dpinger 23978 - [meta sequenceId="8"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 1.1.1.1  bind_addr x.x.x.x  identifier "WAN_STARLINK_DHCP "
<12>1 2024-02-17T02:32:05-06:00 fw.local.lan dpinger 24135 - [meta sequenceId="9"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 9.9.9.9  bind_addr x.x.x.x  identifier "WAN_TMOBILE_DHCP "
<165>1 2024-02-17T02:32:05-06:00 fw.local.lan dpinger 94898 - [meta sequenceId="10"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2024-02-17T02:32:20-06:00 fw.local.lan dpinger 23978 - [meta sequenceId="11"] exiting on signal 15
<12>1 2024-02-17T02:32:20-06:00 fw.local.lan dpinger 24135 - [meta sequenceId="12"] exiting on signal 15
<12>1 2024-02-17T02:32:20-06:00 fw.local.lan dpinger 37199 - [meta sequenceId="13"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 1.1.1.1  bind_addr x.x.x.x  identifier "WAN_STARLINK_DHCP "
<12>1 2024-02-17T02:32:20-06:00 fw.local.lan dpinger 37626 - [meta sequenceId="14"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 9.9.9.9  bind_addr x.x.x.x  identifier "WAN_TMOBILE_DHCP "
<165>1 2024-02-17T02:32:20-06:00 fw.local.lan dpinger 94898 - [meta sequenceId="15"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2024-02-17T02:33:30-06:00 fw.local.lan dpinger 37199 - [meta sequenceId="1"] exiting on signal 15
<12>1 2024-02-17T02:33:30-06:00 fw.local.lan dpinger 37626 - [meta sequenceId="2"] exiting on signal 15
<12>1 2024-02-17T02:33:30-06:00 fw.local.lan dpinger 31779 - [meta sequenceId="3"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 1.1.1.1  bind_addr x.x.x.x  identifier "WAN_STARLINK_DHCP "
<12>1 2024-02-17T02:33:30-06:00 fw.local.lan dpinger 32114 - [meta sequenceId="4"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 9.9.9.9  bind_addr x.x.x.x  identifier "WAN_TMOBILE_DHCP "
<165>1 2024-02-17T02:33:30-06:00 fw.local.lan dpinger 94898 - [meta sequenceId="5"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2024-02-17T02:33:51-06:00 fw.local.lan dpinger 31779 - [meta sequenceId="6"] exiting on signal 15
<12>1 2024-02-17T02:33:51-06:00 fw.local.lan dpinger 32114 - [meta sequenceId="7"] exiting on signal 15
<12>1 2024-02-17T02:33:51-06:00 fw.local.lan dpinger 48050 - [meta sequenceId="8"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 1.1.1.1  bind_addr x.x.x.x  identifier "WAN_STARLINK_DHCP "
<12>1 2024-02-17T02:33:51-06:00 fw.local.lan dpinger 48207 - [meta sequenceId="9"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 9.9.9.9  bind_addr x.x.x.x  identifier "WAN_TMOBILE_DHCP "
<165>1 2024-02-17T02:33:51-06:00 fw.local.lan dpinger 94898 - [meta sequenceId="10"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2024-02-17T02:34:19-06:00 fw.local.lan dpinger 48050 - [meta sequenceId="11"] exiting on signal 15
<12>1 2024-02-17T02:34:19-06:00 fw.local.lan dpinger 48207 - [meta sequenceId="12"] exiting on signal 15
<12>1 2024-02-17T02:34:19-06:00 fw.local.lan dpinger 74434 - [meta sequenceId="13"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 1.1.1.1  bind_addr x.x.x.x  identifier "WAN_STARLINK_DHCP "
<12>1 2024-02-17T02:34:19-06:00 fw.local.lan dpinger 74833 - [meta sequenceId="14"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 9.9.9.9  bind_addr x.x.x.x  identifier "WAN_TMOBILE_DHCP "
<165>1 2024-02-17T02:34:19-06:00 fw.local.lan dpinger 94898 - [meta sequenceId="15"] Reloaded gateway watcher configuration on SIGHUP
<165>1 2024-02-17T02:34:21-06:00 fw.local.lan dpinger 94898 - [meta sequenceId="16"] ALERT: WAN_STARLINK_DHCP (Addr: 1.1.1.1 Alarm: none -> down RTT: 37.5 ms RTTd: 0.8 ms Loss: 0.0 %)
<12>1 2024-02-17T02:34:40-06:00 fw.local.lan dpinger 74434 - [meta sequenceId="17"] exiting on signal 15
<12>1 2024-02-17T02:34:40-06:00 fw.local.lan dpinger 74833 - [meta sequenceId="18"] exiting on signal 15
<12>1 2024-02-17T02:34:40-06:00 fw.local.lan dpinger 16932 - [meta sequenceId="19"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 1.1.1.1  bind_addr x.x.x.x  identifier "WAN_STARLINK_DHCP "
<165>1 2024-02-17T02:34:40-06:00 fw.local.lan dpinger 94898 - [meta sequenceId="20"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2024-02-17T02:35:03-06:00 fw.local.lan dpinger 16932 - [meta sequenceId="21"] exiting on signal 15
<12>1 2024-02-17T02:37:43-06:00 fw.local.lan dpinger 40766 - [meta sequenceId="1"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 1.1.1.1  bind_addr x.x.x.x  identifier "WAN_STARLINK_DHCP "
<12>1 2024-02-17T02:37:43-06:00 fw.local.lan dpinger 41579 - [meta sequenceId="2"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 9.9.9.9  bind_addr x.x.x.x  identifier "WAN_TMOBILE_DHCP "
<165>1 2024-02-17T02:37:54-06:00 fw.local.lan dpinger 42097 - [meta sequenceId="3"] ALERT: WAN_STARLINK_DHCP (Addr: 1.1.1.1 Alarm: down -> none RTT: 28.4 ms RTTd: 5.7 ms Loss: 0.0 %)
<165>1 2024-02-17T02:37:54-06:00 fw.local.lan dpinger 42097 - [meta sequenceId="4"] ALERT: WAN_TMOBILE_DHCP (Addr: 9.9.9.9 Alarm: down -> none RTT: 54.8 ms RTTd: 6.2 ms Loss: 0.0 %)
<12>1 2024-02-17T02:38:21-06:00 fw.local.lan dpinger 40766 - [meta sequenceId="5"] exiting on signal 15
<12>1 2024-02-17T02:38:21-06:00 fw.local.lan dpinger 41579 - [meta sequenceId="6"] exiting on signal 15
<12>1 2024-02-17T02:38:21-06:00 fw.local.lan dpinger 76468 - [meta sequenceId="7"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 1.1.1.1  bind_addr x.x.x.x  identifier "WAN_STARLINK_DHCP "
<12>1 2024-02-17T02:38:21-06:00 fw.local.lan dpinger 76937 - [meta sequenceId="8"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 9.9.9.9  bind_addr x.x.x.x  identifier "WAN_TMOBILE_DHCP "
<165>1 2024-02-17T02:38:21-06:00 fw.local.lan dpinger 42097 - [meta sequenceId="9"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2024-02-17T02:39:09-06:00 fw.local.lan dpinger 76468 - [meta sequenceId="10"] exiting on signal 15

ilikenwf commented 8 months ago

And most of the syslog:


<13>1 2024-02-17T02:09:55-06:00 fw.local.lan dhclient 51071 - [meta sequenceId="1"] dhclient-script: Reason RENEW on em1 executing
<13>1 2024-02-17T02:09:55-06:00 fw.local.lan dhclient 51743 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:12:25-06:00 fw.local.lan dhclient 51137 - [meta sequenceId="1"] dhclient-script: Reason RENEW on em1 executing
<13>1 2024-02-17T02:12:25-06:00 fw.local.lan dhclient 51787 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:14:55-06:00 fw.local.lan dhclient 53297 - [meta sequenceId="1"] dhclient-script: Reason RENEW on em1 executing
<13>1 2024-02-17T02:14:55-06:00 fw.local.lan dhclient 53985 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:17:25-06:00 fw.local.lan dhclient 51981 - [meta sequenceId="1"] dhclient-script: Reason RENEW on em1 executing
<13>1 2024-02-17T02:17:25-06:00 fw.local.lan dhclient 53021 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:19:55-06:00 fw.local.lan dhclient 45547 - [meta sequenceId="1"] dhclient-script: Reason RENEW on em1 executing
<13>1 2024-02-17T02:19:55-06:00 fw.local.lan dhclient 46284 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:22:25-06:00 fw.local.lan dhclient 56425 - [meta sequenceId="1"] dhclient-script: Reason RENEW on em1 executing
<13>1 2024-02-17T02:22:25-06:00 fw.local.lan dhclient 56998 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:24:55-06:00 fw.local.lan dhclient 45029 - [meta sequenceId="1"] dhclient-script: Reason RENEW on em1 executing
<13>1 2024-02-17T02:24:55-06:00 fw.local.lan dhclient 45889 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:27:13-06:00 fw.local.lan configctl 28948 - [meta sequenceId="1"] event @ 1708158433.13 msg: Feb 17 02:27:13 fw.local.lan config[35180]: config-event: new_config /conf/backup/config-1708158433.1101.xml 
<13>1 2024-02-17T02:27:13-06:00 fw.local.lan configctl 28948 - [meta sequenceId="2"] event @ 1708158433.13 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:27:18-06:00 fw.local.lan opnsense 92778 - [meta sequenceId="3"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:27:18-06:00 fw.local.lan opnsense 92778 - [meta sequenceId="4"] /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: WAN_TMOBILE_DHCP
<13>1 2024-02-17T02:27:18-06:00 fw.local.lan opnsense 92778 - [meta sequenceId="5"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:27:18-06:00 fw.local.lan opnsense 92778 - [meta sequenceId="6"] /usr/local/etc/rc.routing_configure: ROUTING: setting inet default route to x.x.x.x
<13>1 2024-02-17T02:27:18-06:00 fw.local.lan opnsense 92778 - [meta sequenceId="7"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:27:18-06:00 fw.local.lan opnsense 92778 - [meta sequenceId="8"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:27:21-06:00 fw.local.lan opnsense 99280 - [meta sequenceId="9"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:27:21-06:00 fw.local.lan opnsense 99280 - [meta sequenceId="10"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:27:21-06:00 fw.local.lan opnsense 99280 - [meta sequenceId="11"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:27:21-06:00 fw.local.lan opnsense 99280 - [meta sequenceId="12"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:27:21-06:00 fw.local.lan opnsense 99280 - [meta sequenceId="13"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:27:22-06:00 fw.local.lan opnsense 5838 - [meta sequenceId="14"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:27:22-06:00 fw.local.lan opnsense 5838 - [meta sequenceId="15"] /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: WAN_TMOBILE_DHCP
<13>1 2024-02-17T02:27:22-06:00 fw.local.lan opnsense 5838 - [meta sequenceId="16"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:27:22-06:00 fw.local.lan opnsense 5838 - [meta sequenceId="17"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:27:22-06:00 fw.local.lan opnsense 5838 - [meta sequenceId="18"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:27:22-06:00 fw.local.lan opnsense 5838 - [meta sequenceId="19"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:27:24-06:00 fw.local.lan opnsense 13177 - [meta sequenceId="20"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:27:24-06:00 fw.local.lan opnsense 13177 - [meta sequenceId="21"] /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: WAN_TMOBILE_DHCP
<13>1 2024-02-17T02:27:24-06:00 fw.local.lan opnsense 13177 - [meta sequenceId="22"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:27:24-06:00 fw.local.lan opnsense 13177 - [meta sequenceId="23"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:27:24-06:00 fw.local.lan opnsense 13177 - [meta sequenceId="24"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:27:24-06:00 fw.local.lan opnsense 13177 - [meta sequenceId="25"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:27:24-06:00 fw.local.lan opnsense 20106 - [meta sequenceId="26"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:27:24-06:00 fw.local.lan opnsense 20106 - [meta sequenceId="27"] /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: WAN_TMOBILE_DHCP
<13>1 2024-02-17T02:27:24-06:00 fw.local.lan opnsense 20106 - [meta sequenceId="28"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:27:24-06:00 fw.local.lan opnsense 20106 - [meta sequenceId="29"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:27:24-06:00 fw.local.lan opnsense 20106 - [meta sequenceId="30"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:27:24-06:00 fw.local.lan opnsense 20106 - [meta sequenceId="31"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:27:25-06:00 fw.local.lan opnsense 26824 - [meta sequenceId="32"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:27:25-06:00 fw.local.lan opnsense 26824 - [meta sequenceId="33"] /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: WAN_TMOBILE_DHCP
<13>1 2024-02-17T02:27:25-06:00 fw.local.lan opnsense 26824 - [meta sequenceId="34"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:27:25-06:00 fw.local.lan opnsense 26824 - [meta sequenceId="35"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:27:25-06:00 fw.local.lan opnsense 26824 - [meta sequenceId="36"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:27:25-06:00 fw.local.lan opnsense 26824 - [meta sequenceId="37"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:27:25-06:00 fw.local.lan dhclient 30071 - [meta sequenceId="38"] dhclient-script: Reason RENEW on em1 executing
<13>1 2024-02-17T02:27:25-06:00 fw.local.lan dhclient 31294 - [meta sequenceId="39"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:27:25-06:00 fw.local.lan opnsense 36207 - [meta sequenceId="40"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:27:25-06:00 fw.local.lan opnsense 36207 - [meta sequenceId="41"] /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: WAN_TMOBILE_DHCP
<13>1 2024-02-17T02:27:25-06:00 fw.local.lan opnsense 36207 - [meta sequenceId="42"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:27:25-06:00 fw.local.lan opnsense 36207 - [meta sequenceId="43"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:27:25-06:00 fw.local.lan opnsense 36207 - [meta sequenceId="44"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:27:25-06:00 fw.local.lan opnsense 36207 - [meta sequenceId="45"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:27:28-06:00 fw.local.lan opnsense 44383 - [meta sequenceId="46"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:27:29-06:00 fw.local.lan opnsense 44383 - [meta sequenceId="47"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:27:29-06:00 fw.local.lan opnsense 44383 - [meta sequenceId="48"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:28:21-06:00 fw.local.lan configctl 28948 - [meta sequenceId="49"] event @ 1708158501.48 msg: Feb 17 02:28:21 fw.local.lan config[64761]: config-event: new_config /conf/backup/config-1708158501.4611.xml 
<13>1 2024-02-17T02:28:22-06:00 fw.local.lan configctl 28948 - [meta sequenceId="50"] event @ 1708158501.48 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:28:26-06:00 fw.local.lan opnsense 4742 - [meta sequenceId="51"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:28:27-06:00 fw.local.lan opnsense 4742 - [meta sequenceId="52"] /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: WAN_STARLINK_DHCP
<13>1 2024-02-17T02:28:27-06:00 fw.local.lan opnsense 4742 - [meta sequenceId="53"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt7
<13>1 2024-02-17T02:28:27-06:00 fw.local.lan opnsense 4742 - [meta sequenceId="54"] /usr/local/etc/rc.routing_configure: ROUTING: setting inet default route to x.x.x.x
<13>1 2024-02-17T02:28:27-06:00 fw.local.lan opnsense 4742 - [meta sequenceId="55"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:28:27-06:00 fw.local.lan opnsense 4742 - [meta sequenceId="56"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:28:29-06:00 fw.local.lan opnsense 13036 - [meta sequenceId="57"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:28:29-06:00 fw.local.lan opnsense 13036 - [meta sequenceId="58"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:28:29-06:00 fw.local.lan opnsense 13036 - [meta sequenceId="59"] /usr/local/etc/rc.routing_configure: ROUTING: setting inet default route to x.x.x.x
<13>1 2024-02-17T02:28:38-06:00 fw.local.lan opnsense 20019 - [meta sequenceId="60"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:28:38-06:00 fw.local.lan opnsense 20019 - [meta sequenceId="61"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:28:38-06:00 fw.local.lan opnsense 20019 - [meta sequenceId="62"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:28:38-06:00 fw.local.lan opnsense 20019 - [meta sequenceId="63"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:28:38-06:00 fw.local.lan opnsense 20019 - [meta sequenceId="64"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:29:55-06:00 fw.local.lan dhclient 91140 - [meta sequenceId="1"] dhclient-script: Reason RENEW on em1 executing
<13>1 2024-02-17T02:29:55-06:00 fw.local.lan dhclient 91595 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:31:25-06:00 fw.local.lan configctl 28948 - [meta sequenceId="1"] event @ 1708158685.08 msg: Feb 17 02:31:25 fw.local.lan config[76455]: config-event: new_config /conf/backup/config-1708158685.0511.xml 
<13>1 2024-02-17T02:31:25-06:00 fw.local.lan configctl 28948 - [meta sequenceId="2"] event @ 1708158685.08 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:31:31-06:00 fw.local.lan configctl 28948 - [meta sequenceId="3"] event @ 1708158691.43 msg: Feb 17 02:31:31 fw.local.lan config[64761]: config-event: new_config /conf/backup/config-1708158691.4174.xml 
<13>1 2024-02-17T02:31:32-06:00 fw.local.lan configctl 28948 - [meta sequenceId="4"] event @ 1708158691.43 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:31:33-06:00 fw.local.lan opnsense 91834 - [meta sequenceId="5"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:31:33-06:00 fw.local.lan opnsense 91834 - [meta sequenceId="6"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:31:33-06:00 fw.local.lan opnsense 91834 - [meta sequenceId="7"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:31:33-06:00 fw.local.lan opnsense 91834 - [meta sequenceId="8"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:31:33-06:00 fw.local.lan opnsense 91834 - [meta sequenceId="9"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:32:04-06:00 fw.local.lan configctl 28948 - [meta sequenceId="10"] event @ 1708158723.79 msg: Feb 17 02:32:03 fw.local.lan config[76455]: config-event: new_config /conf/backup/config-1708158723.7861.xml 
<13>1 2024-02-17T02:32:04-06:00 fw.local.lan configctl 28948 - [meta sequenceId="11"] event @ 1708158723.79 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:32:05-06:00 fw.local.lan opnsense 22098 - [meta sequenceId="12"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:32:05-06:00 fw.local.lan opnsense 22098 - [meta sequenceId="13"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:32:05-06:00 fw.local.lan opnsense 22098 - [meta sequenceId="14"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:32:05-06:00 fw.local.lan opnsense 22098 - [meta sequenceId="15"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:32:05-06:00 fw.local.lan opnsense 22098 - [meta sequenceId="16"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:32:19-06:00 fw.local.lan configctl 28948 - [meta sequenceId="17"] event @ 1708158739.05 msg: Feb 17 02:32:19 fw.local.lan config[64761]: config-event: new_config /conf/backup/config-1708158739.0355.xml 
<13>1 2024-02-17T02:32:19-06:00 fw.local.lan configctl 28948 - [meta sequenceId="18"] event @ 1708158739.05 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:32:20-06:00 fw.local.lan opnsense 35204 - [meta sequenceId="19"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:32:20-06:00 fw.local.lan opnsense 35204 - [meta sequenceId="20"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:32:20-06:00 fw.local.lan opnsense 35204 - [meta sequenceId="21"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:32:20-06:00 fw.local.lan opnsense 35204 - [meta sequenceId="22"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:32:20-06:00 fw.local.lan opnsense 35204 - [meta sequenceId="23"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:32:25-06:00 fw.local.lan dhclient 74412 - [meta sequenceId="24"] dhclient-script: Reason RENEW on em1 executing
<13>1 2024-02-17T02:32:25-06:00 fw.local.lan dhclient 75000 - [meta sequenceId="25"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:33:29-06:00 fw.local.lan configctl 28948 - [meta sequenceId="1"] event @ 1708158808.79 msg: Feb 17 02:33:28 fw.local.lan config[76455]: config-event: new_config /conf/backup/config-1708158808.7727.xml 
<13>1 2024-02-17T02:33:29-06:00 fw.local.lan configctl 28948 - [meta sequenceId="2"] event @ 1708158808.79 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:33:30-06:00 fw.local.lan opnsense 29066 - [meta sequenceId="3"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:33:30-06:00 fw.local.lan opnsense 29066 - [meta sequenceId="4"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:33:30-06:00 fw.local.lan opnsense 29066 - [meta sequenceId="5"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:33:30-06:00 fw.local.lan opnsense 29066 - [meta sequenceId="6"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:33:30-06:00 fw.local.lan opnsense 29066 - [meta sequenceId="7"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:33:50-06:00 fw.local.lan configctl 28948 - [meta sequenceId="8"] event @ 1708158829.56 msg: Feb 17 02:33:49 fw.local.lan config[76455]: config-event: new_config /conf/backup/config-1708158829.5384.xml 
<13>1 2024-02-17T02:33:50-06:00 fw.local.lan configctl 28948 - [meta sequenceId="9"] event @ 1708158829.56 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:33:51-06:00 fw.local.lan opnsense 45420 - [meta sequenceId="10"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:33:51-06:00 fw.local.lan opnsense 45420 - [meta sequenceId="11"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:33:51-06:00 fw.local.lan opnsense 45420 - [meta sequenceId="12"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:33:51-06:00 fw.local.lan opnsense 45420 - [meta sequenceId="13"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:33:51-06:00 fw.local.lan opnsense 45420 - [meta sequenceId="14"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:34:17-06:00 fw.local.lan configctl 28948 - [meta sequenceId="15"] event @ 1708158857.17 msg: Feb 17 02:34:17 fw.local.lan config[76455]: config-event: new_config /conf/backup/config-1708158857.1488.xml 
<13>1 2024-02-17T02:34:17-06:00 fw.local.lan configctl 28948 - [meta sequenceId="16"] event @ 1708158857.17 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:34:19-06:00 fw.local.lan opnsense 72033 - [meta sequenceId="17"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:34:19-06:00 fw.local.lan opnsense 72033 - [meta sequenceId="18"] /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: WAN_STARLINK_DHCP
<13>1 2024-02-17T02:34:19-06:00 fw.local.lan opnsense 72033 - [meta sequenceId="19"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt7
<13>1 2024-02-17T02:34:19-06:00 fw.local.lan opnsense 72033 - [meta sequenceId="20"] /usr/local/etc/rc.routing_configure: ROUTING: setting inet default route to x.x.x.x
<13>1 2024-02-17T02:34:19-06:00 fw.local.lan opnsense 72033 - [meta sequenceId="21"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:34:19-06:00 fw.local.lan opnsense 72033 - [meta sequenceId="22"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:34:21-06:00 fw.local.lan opnsense 80615 - [meta sequenceId="23"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:34:21-06:00 fw.local.lan opnsense 80615 - [meta sequenceId="24"] /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: WAN_STARLINK_DHCP
<13>1 2024-02-17T02:34:21-06:00 fw.local.lan opnsense 80615 - [meta sequenceId="25"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt7
<13>1 2024-02-17T02:34:21-06:00 fw.local.lan opnsense 80615 - [meta sequenceId="26"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:34:39-06:00 fw.local.lan configctl 28948 - [meta sequenceId="27"] event @ 1708158878.72 msg: Feb 17 02:34:38 fw.local.lan config[76455]: config-event: new_config /conf/backup/config-1708158878.6966.xml 
<13>1 2024-02-17T02:34:39-06:00 fw.local.lan configctl 28948 - [meta sequenceId="28"] event @ 1708158878.72 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:34:40-06:00 fw.local.lan opnsense 14798 - [meta sequenceId="29"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:34:40-06:00 fw.local.lan opnsense 14798 - [meta sequenceId="30"] /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: WAN_STARLINK_DHCP
<13>1 2024-02-17T02:34:40-06:00 fw.local.lan opnsense 14798 - [meta sequenceId="31"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt7
<13>1 2024-02-17T02:34:40-06:00 fw.local.lan opnsense 14798 - [meta sequenceId="32"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:34:40-06:00 fw.local.lan opnsense 14798 - [meta sequenceId="33"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:34:40-06:00 fw.local.lan opnsense 14798 - [meta sequenceId="34"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:34:55-06:00 fw.local.lan dhclient 25039 - [meta sequenceId="35"] dhclient-script: Reason RENEW on em1 executing
<13>1 2024-02-17T02:34:55-06:00 fw.local.lan dhclient 25694 - [meta sequenceId="36"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:35:01-06:00 fw.local.lan configctl 28948 - [meta sequenceId="37"] event @ 1708158901.22 msg: Feb 17 02:35:01 fw.local.lan config[76455]: config-event: new_config /conf/backup/config-1708158901.2073.xml 
<13>1 2024-02-17T02:35:01-06:00 fw.local.lan configctl 28948 - [meta sequenceId="38"] event @ 1708158901.22 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:35:02-06:00 fw.local.lan opnsense 44969 - [meta sequenceId="39"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:35:02-06:00 fw.local.lan opnsense 44969 - [meta sequenceId="40"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:35:02-06:00 fw.local.lan opnsense 44969 - [meta sequenceId="41"] /usr/local/etc/rc.routing_configure: ROUTING: setting inet default route to x.x.x.x
<13>1 2024-02-17T02:35:02-06:00 fw.local.lan opnsense 44969 - [meta sequenceId="42"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:35:02-06:00 fw.local.lan opnsense 44969 - [meta sequenceId="43"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:35:41-06:00 fw.local.lan configctl 28948 - [meta sequenceId="44"] event @ 1708158940.59 msg: Feb 17 02:35:40 fw.local.lan config[76455]: config-event: new_config /conf/backup/config-1708158940.5693.xml 
<13>1 2024-02-17T02:35:41-06:00 fw.local.lan configctl 28948 - [meta sequenceId="45"] event @ 1708158940.59 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:36:05-06:00 fw.local.lan configctl 28948 - [meta sequenceId="46"] event @ 1708158965.47 msg: Feb 17 02:36:05 fw.local.lan config[6497]: config-event: new_config /conf/backup/config-1708158965.4638.xml 
<13>1 2024-02-17T02:36:06-06:00 fw.local.lan configctl 28948 - [meta sequenceId="47"] event @ 1708158965.47 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:36:13-06:00 fw.local.lan configctl 28948 - [meta sequenceId="48"] event @ 1708158972.70 msg: Feb 17 02:36:12 fw.local.lan config[64405]: config-event: new_config /conf/backup/config-1708158972.6916.xml 
<13>1 2024-02-17T02:36:13-06:00 fw.local.lan configctl 28948 - [meta sequenceId="49"] event @ 1708158972.70 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:37:26-06:00 fw.local.lan dhclient 14051 - [meta sequenceId="1"] dhclient-script: Reason RENEW on em1 executing
<13>1 2024-02-17T02:37:26-06:00 fw.local.lan dhclient 14820 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:37:34-06:00 fw.local.lan configctl 28948 - [meta sequenceId="3"] event @ 1708159053.67 msg: Feb 17 02:37:33 fw.local.lan config[6497]: config-event: new_config /conf/backup/config-1708159053.6502.xml 
<13>1 2024-02-17T02:37:34-06:00 fw.local.lan configctl 28948 - [meta sequenceId="4"] event @ 1708159053.67 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:37:40-06:00 fw.local.lan configctl 28948 - [meta sequenceId="5"] event @ 1708159060.23 msg: Feb 17 02:37:40 fw.local.lan config[6497]: config-event: new_config /conf/backup/config-1708159060.2119.xml 
<13>1 2024-02-17T02:37:40-06:00 fw.local.lan configctl 28948 - [meta sequenceId="6"] event @ 1708159060.23 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:37:42-06:00 fw.local.lan opnsense 38994 - [meta sequenceId="7"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:37:43-06:00 fw.local.lan opnsense 38994 - [meta sequenceId="8"] /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: WAN_STARLINK_DHCP, WAN_TMOBILE_DHCP
<13>1 2024-02-17T02:37:43-06:00 fw.local.lan opnsense 38994 - [meta sequenceId="9"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:37:43-06:00 fw.local.lan opnsense 38994 - [meta sequenceId="10"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:37:54-06:00 fw.local.lan opnsense 51954 - [meta sequenceId="11"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:37:54-06:00 fw.local.lan opnsense 51954 - [meta sequenceId="12"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:37:54-06:00 fw.local.lan opnsense 51954 - [meta sequenceId="13"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:38:19-06:00 fw.local.lan configctl 28948 - [meta sequenceId="14"] event @ 1708159099.47 msg: Feb 17 02:38:19 fw.local.lan config[6497]: config-event: new_config /conf/backup/config-1708159099.4333.xml 
<13>1 2024-02-17T02:38:20-06:00 fw.local.lan configctl 28948 - [meta sequenceId="15"] event @ 1708159099.47 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:38:21-06:00 fw.local.lan opnsense 74589 - [meta sequenceId="16"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:38:21-06:00 fw.local.lan opnsense 74589 - [meta sequenceId="17"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:38:21-06:00 fw.local.lan opnsense 74589 - [meta sequenceId="18"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:38:21-06:00 fw.local.lan opnsense 74589 - [meta sequenceId="19"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:38:21-06:00 fw.local.lan opnsense 74589 - [meta sequenceId="20"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:39:08-06:00 fw.local.lan kernel - - [meta sequenceId="21"] 148.575240 [ 294] generic_netmap_unregister Emulated adapter for em0_vlan2 deactivated
<13>1 2024-02-17T02:39:08-06:00 fw.local.lan kernel - - [meta sequenceId="22"] 148.575642 [1076] generic_netmap_dtor       Emulated netmap adapter for em0_vlan2 destroyed
<45>1 2024-02-17T02:39:09-06:00 fw.local.lan syslog-ng 13312 - [meta sequenceId="23"] syslog-ng shutting down; version='4.6.0'
<13>1 2024-02-17T02:44:24-06:00 fw.local.lan configctl 49963 - [meta sequenceId="1"] event @ 1708159463.57 msg: Feb 17 02:44:23 fw.local.lan config[62793]: config-event: new_config /conf/backup/config-1708159463.56.xml 
<13>1 2024-02-17T02:44:24-06:00 fw.local.lan configctl 49963 - [meta sequenceId="2"] event @ 1708159463.57 exec: system event config_changed response: OK  
<27>1 2024-02-17T02:44:35-06:00 fw.local.lan dhclient 80009 - [meta sequenceId="3"] connection closed
<26>1 2024-02-17T02:44:35-06:00 fw.local.lan dhclient 80009 - [meta sequenceId="4"] exiting.
<13>1 2024-02-17T02:44:36-06:00 fw.local.lan configctl 49963 - [meta sequenceId="5"] event @ 1708159476.01 msg: Feb 17 02:44:36 fw.local.lan config[64355]: config-event: new_config /conf/backup/config-1708159475.9994.xml 
<13>1 2024-02-17T02:44:36-06:00 fw.local.lan configctl 49963 - [meta sequenceId="6"] event @ 1708159476.01 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:44:37-06:00 fw.local.lan kernel - - [meta sequenceId="7"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em1
<27>1 2024-02-17T02:44:38-06:00 fw.local.lan dhclient 8739 - [meta sequenceId="8"] connection closed
<26>1 2024-02-17T02:44:38-06:00 fw.local.lan dhclient 8739 - [meta sequenceId="9"] exiting.
<13>1 2024-02-17T02:44:38-06:00 fw.local.lan kernel - - [meta sequenceId="10"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:38-06:00 fw.local.lan kernel - - [meta sequenceId="11"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:38-06:00 fw.local.lan kernel - - [meta sequenceId="12"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:38-06:00 fw.local.lan kernel - - [meta sequenceId="13"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:38-06:00 fw.local.lan kernel - - [meta sequenceId="14"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="15"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="16"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="17"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="18"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan configctl 49963 - [meta sequenceId="19"] event @ 1708159478.55 msg: Feb 17 02:44:38 fw.local.lan config[62793]: config-event: new_config /conf/backup/config-1708159478.5386.xml 
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan configctl 49963 - [meta sequenceId="20"] event @ 1708159478.55 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="21"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="22"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="23"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="24"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em1
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="25"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="26"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="27"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="28"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="29"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="30"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="31"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="32"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="33"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="34"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="35"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:44:39-06:00 fw.local.lan kernel - - [meta sequenceId="36"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:45:48-06:00 fw.local.lan dhclient 19117 - [meta sequenceId="1244"] dhclient-script: Reason PREINIT on em1 executing
<13>1 2024-02-17T02:45:48-06:00 fw.local.lan dhclient 22775 - [meta sequenceId="1245"] dhclient-script: Reason EXPIRE on em1 executing
<13>1 2024-02-17T02:45:48-06:00 fw.local.lan dhclient 26800 - [meta sequenceId="1246"] dhclient-script: Reason PREINIT on em1 executing
<13>1 2024-02-17T02:45:48-06:00 fw.local.lan dhclient 30924 - [meta sequenceId="1247"] dhclient-script: Reason ARPSEND on em1 executing
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan dhclient 33534 - [meta sequenceId="1248"] dhclient-script: Reason ARPCHECK on em1 executing
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan dhclient 35362 - [meta sequenceId="1249"] dhclient-script: Reason BOUND on em1 executing
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan dhclient 37162 - [meta sequenceId="1250"] dhclient-script: New IP Address (em1): x.x.x.x
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan dhclient 38918 - [meta sequenceId="1251"] dhclient-script: New Subnet Mask (em1): 255.255.248.0
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan dhclient 40470 - [meta sequenceId="1252"] dhclient-script: New Broadcast Address (em1): x.x.x.x
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan dhclient 43045 - [meta sequenceId="1253"] dhclient-script: New Routers (em1): x.x.x.x
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan dhclient 44589 - [meta sequenceId="1254"] dhclient-script: New Classless Static Routes (em1):  192.168.100.1/32 0.0.0.0 34.120.255.244/32 0.0.0.0 default x.x.x.x
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan dhclient 46745 - [meta sequenceId="1255"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1256"] /interfaces.php: ROUTING: entering configure using 'opt6'
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1257"] /interfaces.php: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1258"] /interfaces.php: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1259"] /interfaces.php: plugins_configure monitor (,WAN_STARLINK_DHCP)
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1260"] /interfaces.php: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_STARLINK_DHCP))
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1261"] /usr/local/etc/rc.newwanip: IP renewal starting (new: x.x.x.x, old: x.x.x.x, interface: opt6, device: em1, force: yes)
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1262"] /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'opt6'
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1263"] /usr/local/etc/rc.newwanip: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1264"] /usr/local/etc/rc.newwanip: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1265"] /usr/local/etc/rc.newwanip: plugins_configure monitor (,WAN_STARLINK_DHCP)
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1266"] /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_STARLINK_DHCP))
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1267"] /interfaces.php: plugins_configure ipsec (,opt6)
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1268"] /interfaces.php: plugins_configure ipsec (execute task : ipsec_configure_do(,opt6))
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1269"] /interfaces.php: plugins_configure dhcp ()
<13>1 2024-02-17T02:45:50-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1270"] /interfaces.php: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
<13>1 2024-02-17T02:45:51-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1271"] /interfaces.php: plugins_configure dns ()
<13>1 2024-02-17T02:45:51-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1272"] /interfaces.php: plugins_configure dns (execute task : dnscryptproxy_configure_do())
<13>1 2024-02-17T02:45:51-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1273"] /interfaces.php: plugins_configure dns (execute task : dnsmasq_configure_do())
<13>1 2024-02-17T02:45:51-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1274"] /interfaces.php: plugins_configure dns (execute task : unbound_configure_do())
<13>1 2024-02-17T02:45:51-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1275"] /usr/local/etc/rc.newwanip: plugins_configure vpn (,opt6)
<13>1 2024-02-17T02:45:51-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1276"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : ipsec_configure_do(,opt6))
<13>1 2024-02-17T02:45:51-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1277"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : openvpn_configure_do(,opt6))
<13>1 2024-02-17T02:45:51-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1278"] /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN_STARLINK.
<13>1 2024-02-17T02:45:51-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1279"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : wireguard_configure_do(,opt6))
<13>1 2024-02-17T02:45:51-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1280"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (,opt6)
<13>1 2024-02-17T02:45:51-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1281"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : dnsmasq_configure_do())
<13>1 2024-02-17T02:45:51-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1282"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : ntpd_configure_do())
<13>1 2024-02-17T02:45:51-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1283"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : opendns_configure_do())
<13>1 2024-02-17T02:45:51-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1284"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : openssh_configure_do(,opt6))
<13>1 2024-02-17T02:45:51-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1285"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : unbound_configure_do(,opt6))
<13>1 2024-02-17T02:45:53-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1286"] /interfaces.php: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:45:53-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1287"] /interfaces.php: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:45:53-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1288"] /interfaces.php: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:45:53-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1289"] /interfaces.php: plugins_configure monitor ()
<13>1 2024-02-17T02:45:53-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1290"] /interfaces.php: plugins_configure monitor (execute task : dpinger_configure_do())
<13>1 2024-02-17T02:45:53-06:00 fw.local.lan kernel - - [meta sequenceId="1291"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:45:53-06:00 fw.local.lan kernel - - [meta sequenceId="1292"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em2
<13>1 2024-02-17T02:45:53-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1293"] /interfaces.php: plugins_configure newwanip (,opt6)
<13>1 2024-02-17T02:45:53-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1294"] /interfaces.php: plugins_configure newwanip (execute task : dnsmasq_configure_do())
<13>1 2024-02-17T02:45:53-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1295"] /interfaces.php: plugins_configure newwanip (execute task : ntpd_configure_do())
<13>1 2024-02-17T02:45:59-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1296"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : vxlan_configure_do())
<13>1 2024-02-17T02:45:59-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1297"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : webgui_configure_do(,opt6))
<13>1 2024-02-17T02:45:59-06:00 fw.local.lan opnsense 48434 - [meta sequenceId="1298"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : wireguard_sync(,opt6))
<27>1 2024-02-17T02:46:03-06:00 fw.local.lan dhclient 16499 - [meta sequenceId="1299"] connection closed
<26>1 2024-02-17T02:46:03-06:00 fw.local.lan dhclient 16499 - [meta sequenceId="1300"] exiting.
<13>1 2024-02-17T02:46:03-06:00 fw.local.lan kernel - - [meta sequenceId="1301"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em1
<13>1 2024-02-17T02:46:03-06:00 fw.local.lan kernel - - [meta sequenceId="1302"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em1
<13>1 2024-02-17T02:46:03-06:00 fw.local.lan dhclient 17619 - [meta sequenceId="1303"] dhclient-script: Reason PREINIT on em1 executing
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan dhclient 21187 - [meta sequenceId="1304"] dhclient-script: Reason REBOOT on em1 executing
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan dhclient 22763 - [meta sequenceId="1305"] dhclient-script: New IP Address (em1): x.x.x.x
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan dhclient 23937 - [meta sequenceId="1306"] dhclient-script: New Subnet Mask (em1): 255.255.248.0
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan dhclient 25461 - [meta sequenceId="1307"] dhclient-script: New Broadcast Address (em1): x.x.x.x
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan dhclient 26879 - [meta sequenceId="1308"] dhclient-script: New Routers (em1): x.x.x.x
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan dhclient 28967 - [meta sequenceId="1309"] dhclient-script: New Classless Static Routes (em1):  192.168.100.1/32 0.0.0.0 34.120.255.244/32 0.0.0.0 default x.x.x.x
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan dhclient 31382 - [meta sequenceId="1310"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1311"] /interfaces.php: ROUTING: entering configure using 'opt6'
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan kernel - - [meta sequenceId="1312"] <7>arpresolve: can't allocate llinfo for x.x.x.x on em1
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1313"] /interfaces.php: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1314"] /interfaces.php: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1315"] /interfaces.php: plugins_configure monitor (,WAN_STARLINK_DHCP)
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1316"] /interfaces.php: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_STARLINK_DHCP))
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1317"] /interfaces.php: plugins_configure ipsec (,opt6)
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1318"] /interfaces.php: plugins_configure ipsec (execute task : ipsec_configure_do(,opt6))
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1319"] /interfaces.php: plugins_configure dhcp ()
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1320"] /interfaces.php: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1321"] /interfaces.php: plugins_configure dns ()
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1322"] /interfaces.php: plugins_configure dns (execute task : dnscryptproxy_configure_do())
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1323"] /usr/local/etc/rc.newwanip: IP renewal starting (new: x.x.x.x, old: x.x.x.x, interface: opt6, device: em1, force: yes)
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1324"] /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'opt6'
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1325"] /usr/local/etc/rc.newwanip: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1326"] /usr/local/etc/rc.newwanip: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1327"] /usr/local/etc/rc.newwanip: plugins_configure monitor (,WAN_STARLINK_DHCP)
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1328"] /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_STARLINK_DHCP))
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1329"] /interfaces.php: plugins_configure dns (execute task : dnsmasq_configure_do())
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1330"] /interfaces.php: plugins_configure dns (execute task : unbound_configure_do())
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1331"] /usr/local/etc/rc.newwanip: plugins_configure vpn (,opt6)
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1332"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : ipsec_configure_do(,opt6))
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1333"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : openvpn_configure_do(,opt6))
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1334"] /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN_STARLINK.
<13>1 2024-02-17T02:46:04-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1335"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : wireguard_configure_do(,opt6))
<13>1 2024-02-17T02:46:05-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1336"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (,opt6)
<13>1 2024-02-17T02:46:05-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1337"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : dnsmasq_configure_do())
<13>1 2024-02-17T02:46:05-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1338"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : ntpd_configure_do())
<13>1 2024-02-17T02:46:06-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1339"] /interfaces.php: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:46:06-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1340"] /interfaces.php: ROUTING: configuring inet default gateway on opt6
<13>1 2024-02-17T02:46:06-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1341"] /interfaces.php: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:46:06-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1342"] /interfaces.php: plugins_configure monitor ()
<13>1 2024-02-17T02:46:06-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1343"] /interfaces.php: plugins_configure monitor (execute task : dpinger_configure_do())
<13>1 2024-02-17T02:46:06-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1344"] /interfaces.php: plugins_configure newwanip (,opt6)
<13>1 2024-02-17T02:46:06-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1345"] /interfaces.php: plugins_configure newwanip (execute task : dnsmasq_configure_do())
<13>1 2024-02-17T02:46:06-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1346"] /interfaces.php: plugins_configure newwanip (execute task : ntpd_configure_do())
<11>1 2024-02-17T02:46:11-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1347"] /interfaces.php: The command '/usr/local/sbin/ntpd -g -c '/var/etc/ntpd.conf'' returned exit code '1', the output was ''
<13>1 2024-02-17T02:46:11-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1348"] /interfaces.php: plugins_configure newwanip (execute task : opendns_configure_do())
<13>1 2024-02-17T02:46:11-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1349"] /interfaces.php: plugins_configure newwanip (execute task : openssh_configure_do(,opt6))
<13>1 2024-02-17T02:46:11-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1350"] /interfaces.php: plugins_configure newwanip (execute task : unbound_configure_do(,opt6))
<13>1 2024-02-17T02:46:11-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1351"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : opendns_configure_do())
<13>1 2024-02-17T02:46:11-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1352"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : openssh_configure_do(,opt6))
<13>1 2024-02-17T02:46:11-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1353"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : unbound_configure_do(,opt6))
<11>1 2024-02-17T02:46:11-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1354"] /interfaces.php: The command '/usr/local/sbin/ntpd -g -c '/var/etc/ntpd.conf'' returned exit code '1', the output was ''
<13>1 2024-02-17T02:46:11-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1355"] /interfaces.php: plugins_configure newwanip (execute task : opendns_configure_do())
<13>1 2024-02-17T02:46:11-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1356"] /interfaces.php: plugins_configure newwanip (execute task : openssh_configure_do(,opt6))
<13>1 2024-02-17T02:46:11-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1357"] /interfaces.php: plugins_configure newwanip (execute task : unbound_configure_do(,opt6))
<11>1 2024-02-17T02:46:12-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1358"] /interfaces.php: The command '/sbin/mount -t devfs devfs '/var/unbound/dev'' returned exit code '1', the output was 'mount: devfs: Device busy'
<11>1 2024-02-17T02:46:12-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1359"] /interfaces.php: The command '/sbin/mount -r -t nullfs '/usr/local/lib/python3.9' '/var/unbound/usr/local/lib/python3.9'' returned exit code '1', the output was 'mount_nullfs: /var/unbound/usr/local/lib/python3.9: Resource deadlock avoided'
<11>1 2024-02-17T02:46:12-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1360"] /usr/local/etc/rc.newwanip: The command '/sbin/mount -r -t nullfs '/usr/local/lib/python3.9' '/var/unbound/usr/local/lib/python3.9'' returned exit code '1', the output was 'mount_nullfs: /var/unbound/usr/local/lib/python3.9: Resource deadlock avoided'
<13>1 2024-02-17T02:46:13-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1361"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : vxlan_configure_do())
<13>1 2024-02-17T02:46:13-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1362"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : webgui_configure_do(,opt6))
<13>1 2024-02-17T02:46:13-06:00 fw.local.lan opnsense 32886 - [meta sequenceId="1363"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : wireguard_sync(,opt6))
<13>1 2024-02-17T02:46:13-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1364"] /interfaces.php: plugins_configure newwanip (execute task : vxlan_configure_do())
<13>1 2024-02-17T02:46:13-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1365"] /interfaces.php: plugins_configure newwanip (execute task : vxlan_configure_do())
<13>1 2024-02-17T02:46:13-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1366"] /interfaces.php: plugins_configure newwanip (execute task : webgui_configure_do(,opt6))
<13>1 2024-02-17T02:46:13-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1367"] /interfaces.php: plugins_configure newwanip (execute task : wireguard_sync(,opt6))
<13>1 2024-02-17T02:46:13-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1368"] /interfaces.php: plugins_configure newwanip (execute task : webgui_configure_do(,opt6))
<13>1 2024-02-17T02:46:13-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1369"] /interfaces.php: plugins_configure newwanip (execute task : wireguard_sync(,opt6))
<11>1 2024-02-17T02:46:14-06:00 fw.local.lan opnsense 62793 - [meta sequenceId="1370"] /interfaces.php: The command '/usr/sbin/daemon -f -p '/var/run/updaterrd.pid' '/var/db/rrd/updaterrd.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 96655'
<11>1 2024-02-17T02:46:14-06:00 fw.local.lan opnsense 64355 - [meta sequenceId="1371"] /interfaces.php: The command '/usr/sbin/daemon -f -p '/var/run/updaterrd.pid' '/var/db/rrd/updaterrd.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 96655'
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan configctl 49963 - [meta sequenceId="1"] event @ 1708159646.57 msg: Feb 17 02:47:26 fw.local.lan config[62793]: config-event: new_config /conf/backup/config-1708159646.5576.xml 
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan configctl 49963 - [meta sequenceId="2"] event @ 1708159646.57 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan dhclient 93934 - [meta sequenceId="3"] dhclient-script: Reason PREINIT on em2 executing
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan dhclient 96881 - [meta sequenceId="4"] dhclient-script: Reason REBOOT on em2 executing
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan dhclient 97818 - [meta sequenceId="5"] dhclient-script: New IP Address (em2): x.x.x.x
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan dhclient 20 - [meta sequenceId="6"] dhclient-script: New Subnet Mask (em2): 255.255.255.0
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan dhclient 1196 - [meta sequenceId="7"] dhclient-script: New Broadcast Address (em2): x.x.x.x
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan dhclient 3909 - [meta sequenceId="8"] dhclient-script: New Routers (em2): x.x.x.x
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan dhclient 7348 - [meta sequenceId="9"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="10"] /interfaces.php: ROUTING: entering configure using 'opt7'
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="11"] /interfaces.php: ROUTING: configuring inet default gateway on opt7
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="12"] /interfaces.php: ROUTING: setting inet default route to x.x.x.x
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="13"] /interfaces.php: plugins_configure monitor (,WAN_TMOBILE_DHCP)
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="14"] /interfaces.php: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_TMOBILE_DHCP))
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="15"] /interfaces.php: plugins_configure ipsec (,opt7)
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="16"] /interfaces.php: plugins_configure ipsec (execute task : ipsec_configure_do(,opt7))
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="17"] /interfaces.php: plugins_configure dhcp ()
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="18"] /interfaces.php: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="19"] /interfaces.php: plugins_configure dns ()
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="20"] /interfaces.php: plugins_configure dns (execute task : dnscryptproxy_configure_do())
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="21"] /usr/local/etc/rc.newwanip: IP renewal starting (new: x.x.x.x, old: x.x.x.x, interface: opt7, device: em2, force: yes)
<13>1 2024-02-17T02:47:27-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="22"] /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'opt7'
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="23"] /usr/local/etc/rc.newwanip: ROUTING: configuring inet default gateway on opt7
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="24"] /usr/local/etc/rc.newwanip: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="25"] /usr/local/etc/rc.newwanip: plugins_configure monitor (,WAN_TMOBILE_DHCP)
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="26"] /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_TMOBILE_DHCP))
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="27"] /interfaces.php: plugins_configure dns (execute task : dnsmasq_configure_do())
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="28"] /interfaces.php: plugins_configure dns (execute task : unbound_configure_do())
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="29"] /usr/local/etc/rc.newwanip: plugins_configure vpn (,opt7)
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="30"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : ipsec_configure_do(,opt7))
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="31"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : openvpn_configure_do(,opt7))
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="32"] /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN_TMOBILE.
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="33"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : wireguard_configure_do(,opt7))
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="34"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (,opt7)
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="35"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : dnsmasq_configure_do())
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="36"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : ntpd_configure_do())
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="37"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : opendns_configure_do())
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="38"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : openssh_configure_do(,opt7))
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="39"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : unbound_configure_do(,opt7))
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="40"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : vxlan_configure_do())
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="41"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : webgui_configure_do(,opt7))
<13>1 2024-02-17T02:47:28-06:00 fw.local.lan opnsense 9018 - [meta sequenceId="42"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : wireguard_sync(,opt7))
<13>1 2024-02-17T02:47:30-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="43"] /interfaces.php: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:47:30-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="44"] /interfaces.php: ROUTING: configuring inet default gateway on opt7
<13>1 2024-02-17T02:47:30-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="45"] /interfaces.php: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:47:30-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="46"] /interfaces.php: plugins_configure monitor ()
<13>1 2024-02-17T02:47:30-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="47"] /interfaces.php: plugins_configure monitor (execute task : dpinger_configure_do())
<13>1 2024-02-17T02:47:30-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="48"] /interfaces.php: plugins_configure newwanip (,opt7)
<13>1 2024-02-17T02:47:30-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="49"] /interfaces.php: plugins_configure newwanip (execute task : dnsmasq_configure_do())
<13>1 2024-02-17T02:47:30-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="50"] /interfaces.php: plugins_configure newwanip (execute task : ntpd_configure_do())
<13>1 2024-02-17T02:47:35-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="51"] /interfaces.php: plugins_configure newwanip (execute task : opendns_configure_do())
<13>1 2024-02-17T02:47:35-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="52"] /interfaces.php: plugins_configure newwanip (execute task : openssh_configure_do(,opt7))
<13>1 2024-02-17T02:47:35-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="53"] /interfaces.php: plugins_configure newwanip (execute task : unbound_configure_do(,opt7))
<13>1 2024-02-17T02:47:35-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="54"] /interfaces.php: plugins_configure newwanip (execute task : vxlan_configure_do())
<13>1 2024-02-17T02:47:35-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="55"] /interfaces.php: plugins_configure newwanip (execute task : webgui_configure_do(,opt7))
<13>1 2024-02-17T02:47:35-06:00 fw.local.lan opnsense 63740 - [meta sequenceId="56"] /interfaces.php: plugins_configure newwanip (execute task : wireguard_sync(,opt7))
<13>1 2024-02-17T02:48:34-06:00 fw.local.lan dhclient 15701 - [meta sequenceId="57"] dhclient-script: Reason RENEW on em1 executing
<13>1 2024-02-17T02:48:34-06:00 fw.local.lan dhclient 17059 - [meta sequenceId="58"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:49:12-06:00 fw.local.lan opnsense 71229 - [meta sequenceId="59"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:49:12-06:00 fw.local.lan opnsense 71229 - [meta sequenceId="60"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt7
<13>1 2024-02-17T02:49:12-06:00 fw.local.lan opnsense 71229 - [meta sequenceId="61"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:49:12-06:00 fw.local.lan opnsense 71229 - [meta sequenceId="62"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:49:12-06:00 fw.local.lan opnsense 71229 - [meta sequenceId="63"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:49:29-06:00 fw.local.lan configctl 49963 - [meta sequenceId="64"] event @ 1708159769.36 msg: Feb 17 02:49:29 fw.local.lan config[62793]: config-event: new_config /conf/backup/config-1708159769.3562.xml 
<13>1 2024-02-17T02:49:29-06:00 fw.local.lan configctl 49963 - [meta sequenceId="65"] event @ 1708159769.36 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:49:31-06:00 fw.local.lan opnsense 20100 - [meta sequenceId="66"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:49:31-06:00 fw.local.lan opnsense 20100 - [meta sequenceId="67"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt7
<13>1 2024-02-17T02:49:31-06:00 fw.local.lan opnsense 20100 - [meta sequenceId="68"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:49:31-06:00 fw.local.lan opnsense 20100 - [meta sequenceId="69"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:49:31-06:00 fw.local.lan opnsense 20100 - [meta sequenceId="70"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:50:00-06:00 fw.local.lan send_telemetry.py 81780 - [meta sequenceId="71"] telemetry data collected 1 records in 0.03 seconds @2024-02-17 08:49:56.514589
<13>1 2024-02-17T02:50:34-06:00 fw.local.lan configctl 49963 - [meta sequenceId="72"] event @ 1708159834.07 msg: Feb 17 02:50:34 fw.local.lan config[62793]: config-event: new_config /conf/backup/config-1708159834.0496.xml 
<13>1 2024-02-17T02:50:34-06:00 fw.local.lan configctl 49963 - [meta sequenceId="73"] event @ 1708159834.07 exec: system event config_changed response: OK  
<13>1 2024-02-17T02:50:36-06:00 fw.local.lan opnsense 28580 - [meta sequenceId="74"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2024-02-17T02:50:36-06:00 fw.local.lan opnsense 28580 - [meta sequenceId="75"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt7
<13>1 2024-02-17T02:50:36-06:00 fw.local.lan opnsense 28580 - [meta sequenceId="76"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to x.x.x.x
<13>1 2024-02-17T02:50:36-06:00 fw.local.lan opnsense 28580 - [meta sequenceId="77"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1)
<13>1 2024-02-17T02:50:36-06:00 fw.local.lan opnsense 28580 - [meta sequenceId="78"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1))
<13>1 2024-02-17T02:51:00-06:00 fw.local.lan send_telemetry.py 93029 - [meta sequenceId="79"] telemetry data collected 1 records in 0.04 seconds @2024-02-17 08:50:13.675899
<13>1 2024-02-17T02:51:04-06:00 fw.local.lan dhclient 20857 - [meta sequenceId="80"] dhclient-script: Reason RENEW on em1 executing
<13>1 2024-02-17T02:51:04-06:00 fw.local.lan dhclient 22724 - [meta sequenceId="81"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:53:35-06:00 fw.local.lan dhclient 25356 - [meta sequenceId="1"] dhclient-script: Reason RENEW on em1 executing
<13>1 2024-02-17T02:53:35-06:00 fw.local.lan dhclient 26952 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:56:05-06:00 fw.local.lan dhclient 96882 - [meta sequenceId="1"] dhclient-script: Reason RENEW on em1 executing
<13>1 2024-02-17T02:56:05-06:00 fw.local.lan dhclient 98403 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2024-02-17T02:58:35-06:00 fw.local.lan dhclient 90276 - [meta sequenceId="1"] dhclient-script: Reason RENEW on em1 executing
<13>1 2024-02-17T02:58:35-06:00 fw.local.lan dhclient 90741 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf

ilikenwf commented 8 months ago

Those logs feature with and without, you can see where things get reloaded and reset.

ilikenwf commented 8 months ago

I don't see how this could otherwise be config related since the round robin otherwise works if I don't enable monitoring and monitor IPs.

ilikenwf commented 8 months ago

Would it be possible to let us use monit for basic up/down checks, even if latency is not part of it, flagging the interfaces up/down as things change?

Idk...I just want to get this working and can't grasp why it isn't. I'm happy to share more details if they'll help, but I'd need to know what to share.

fichtner commented 8 months ago

Hmm, the issue with starlink appears that it can send you global DNS servers that may overlap with your multi-wan use. Make sure your DNS server configuration either doesn’t allow dialup DNS Overrides or straighten out any overlap which is crucial for multi-wan to work.

ilikenwf commented 8 months ago

I tried previously just having quad9 set for tmo, and their unfiltered 9.9.9.10 set for the starlink WAN. At the end of the day I prefer my own resolver anyway, and that's what clients use, so for the sake of ping I thought that setting and binding without allowing overrides is acceptable practice in such a case?

No actual lookups are really being done when you use dpinger, so I was thinking that was the logic behind even binding dns to the interfaces at all.

Starlink changes the WAN ip somewhat often because of how the service works with sats moving around and all, and then TMO has atrocious latency, and so something of that could factor in too, I suppose.

Are we sure having DHCP and changing IPs on the starlink interface doesn't cause issues? The TMO IP I have is assigned DHCP but static.

For a brief period of time TMO was working with monitoring enabled, but I'd occasionally drop packets when round robin switched me to starlink.

What is weird to me is that in load balance configuration the round robin works even without latencies to measure against, kinda, in that both connections are utilized.

Throughout all this, the firewall itself can reach the internet just fine, the clients can't - I can share my firewall rules if you'd like, but other than the filtering and alias stuff, they're just redirecting specific VLAN and a few odd clients that are specific toward either of the two groups.

Would the API allow me to have some kind of script setup outside the UI that just periodically runs a web request and flags up/down?

ilikenwf commented 7 months ago

@fichtner as it turns out, even touching the autogenerated gateway entry appears to cause this issue, where the gateway is no longer usable until I remove and recreate it with a config backup.

All I did was alter the priority value of the default gateway entry for starlink, and now I no longer am able to use it on the LAN. This makes me feel like some kind of config corruption is happening?

The starlink WAN still works on the router, as in the router can communicate through it, but otherwise the only output I get when I alter them is:

arpresolve: can't allocate llinfo for x.x.x.x on em1 arpresolve: can't allocate llinfo for x.x.x.x on em1 arpresolve: can't allocate llinfo for x.x.x.x on em1

fichtner commented 7 months ago

Can you show the system: configuration: history diff between the working and the non-working state?

ilikenwf commented 7 months ago

I believe this is it - unless I more or less leave gateways empty and allow opnsense to just set those two automatically and use the autoconfigured gateways as is, without them saved to config, this seems to result...

--- /conf/backup/config-1708314128.9724.xml 2024-02-18 21:42:08.982658000 -0600
+++ /conf/backup/config-1708547370.1751.xml 2024-02-21 14:29:30.192113000 -0600
@@ -2795,8 +2795,8 @@
   </widgets>
   <revision>
     <username>root@x.x.x.x</username>
-    <description>/api/unbound/settings/set made changes</description>
-    <time>1708314128.9725</time>
+    <description>/api/routing/settings/setGateway/19766b6f-c4de-4ca9-8f9d-bd4fbf24b261 made changes</description>
+    <time>1708547370.1752</time>
   </revision>
   <OPNsense>
     <proxy version="1.0.6">
@@ -7362,7 +7362,32 @@
         </cronjobs>
       </maintenance>
     </HAProxy>
-    <Gateways version="1.0.0"/>
+    <Gateways version="1.0.0">
+      <gateway_item uuid="19766b6f-c4de-4ca9-8f9d-bd4fbf24b261">
+        <disabled>0</disabled>
+        <name>WAN_STARLINK_DHCP</name>
+        <descr>Interface WAN_STARLINK_DHCP Gateway</descr>
+        <interface>opt6</interface>
+        <ipprotocol>inet</ipprotocol>
+        <gateway>x.x.x.x</gateway>
+        <defaultgw>0</defaultgw>
+        <fargw>0</fargw>
+        <monitor_disable>1</monitor_disable>
+        <monitor_noroute>0</monitor_noroute>
+        <monitor/>
+        <force_down>0</force_down>
+        <priority>254</priority>
+        <weight>1</weight>
+        <latencylow/>
+        <latencyhigh/>
+        <losslow/>
+        <losshigh/>
+        <interval/>
+        <time_period/>
+        <loss_interval/>
+        <data_length/>
+      </gateway_item>
+    </Gateways>
     <Kea>
       <dhcp4 version="1.0.0">
         <general>
@@ -7400,8 +7425,8 @@
     <defaultgw6/>
     <gateway_group>
       <name>HUMANS</name>
-      <item>WAN_TMOBILE_DHCP|1</item>
       <item>WAN_STARLINK_DHCP|1</item>
+      <item>WAN_TMOBILE_DHCP|1</item>
       <trigger>downlosslatency</trigger>
       <poolopts>round-robin</poolopts>
       <descr>HUMANS</descr>

ilikenwf commented 7 months ago

I had a hunch so I did this, and now it works? Cleared the IP and set it as a far and upstream gateway...

ilikenwf commented 7 months ago

May need to blank the IP field by default unless a WAN is a static IP? @fichtner I guess I can try to get out my 2-3 years rusty php skills out and try to do a pull request if you'd like, beyond that my issues are gone... I can close this if you want.

AdSchellevis commented 7 months ago

@ilikenwf looks like I missed this ticket, but https://github.com/opnsense/core/commit/1d229551a5b5cf0e8030cb6c5938a8a8071d90e0 fixes the prefilled dynamic issue. I think it was hot fixed into 24.1.2 yesterday

fichtner commented 7 months ago

I knew we were getting close here. 😊

ilikenwf commented 7 months ago

Thank you both.

opnsense / core

Multi-Wan: Routing Breaks if the autogenerated gateways are edited #7224