"double free or corruption" in community sweep

[jennifer-richards]

After running a pair of peered trust routers for about 20 minutes, this occurred:

tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "redsox.com", "records": [{"record_type": "comm", "type": "apc", "role": "rp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 30}]}}
trps_update_one_peer: adding message to queue.
trps_update_one_peer: update queued successfully.
tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "whitesox.mlb.us", "records": [{"record_type": "comm", "type": "apc", "role": "rp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 30}]}}
trps_update_one_peer: adding message to queue.
tr_trpc_thread: sent 1 messages.
trps_update_one_peer: update queued successfully.
trps_update: rc=0 after attempting update.
tr_trpc_thread: sent 1 messages.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_route_set_triggered: setting route to apc.sports.psec.us/example81.com through trustrouter@apc.sports.psec.us to not triggered
trp_route_set_triggered: setting route to apc.sports.psec.us/reject-me through trustrouter@apc.sports.psec.us to not triggered
trp_route_set_triggered: setting route to apc.sports.psec.us/reject-me.com through trustrouter@apc.sports.psec.us to not triggered
trp_route_set_triggered: setting route to apc.sports.psec.us/whitesox.mlb.us through trustrouter@apc.sports.psec.us to not triggered
trp_route_set_triggered: setting route to apc.sports.psec.us/example12.com through trustrouter@apc.sports.psec.us to not triggered
trp_route_set_triggered: setting route to apc.sports.psec.us/redsox.com through trustrouter@apc.sports.psec.us to not triggered
trp_route_set_triggered: setting route to apc.sports.psec.us/test.org through trustrouter@apc.sports.psec.us to not triggered
trp_route_set_triggered: setting route to apc.sports.psec.us/apc.sports.psec.us through  to not triggered
trp_route_set_triggered: setting route to apc.sports.psec.us/apc.sports.psec.us through trustrouter@apc.sports.psec.us to not triggered
trp_route_set_triggered: setting route to apc.sports.psec.us/mariners.com through trustrouter@apc.sports.psec.us to not triggered
tr_trps_update: update interval=30
tr_trps_sweep: sweeping routes.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
tr_trps_sweep: sweeping communities.
tr_comm_memb_is_expired: (cur->tv_sec>memb->expiry->tv_sec)=(169644 > 169719)=false
tr_comm_memb_is_expired: (cur->tv_sec>memb->expiry->tv_sec)=(169644 > 169719)=false
tr_comm_memb_is_expired: (cur->tv_sec>memb->expiry->tv_sec)=(169644 > 169719)=false
*** Error in `tr/trust_router': double free or corruption (!prev): 0x00005631b6059fa0 ***
Aborted

Possibly related to #63, but I'm not convinced without further investigation.

[jennifer-richards]

I audited the code in the area this seemed to be happening and could not find any obvious causes of a double free. The error message indicates that this is a failed free() call (and not a failed talloc_free()). It is probably through a second freeing of a TR_COMM, TR_IDP_REALM, or TR_RP_REALM, because these contain TR_NAMEs that are freed using free() as part of their talloc destructors. I could not find a clear code path for this.

I am hoping that perhaps #70 has fixed this. This is a bit speculative because I don't have a concrete idea of how that could cause this, but inconsistent treatment of a table that has elements freed from it seems like a plausible culprit.

In any case, after spending quite a while on this, I have not reproduced it. I am closing it for now and we can reopen if we see it again. Hopefully stability testing will find it if it's still there.

[jennifer-richards]

It happened again, in a different place in the logs... Reopening.

tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "example81.com", "records": [{"record_type": "route", "trust_router": "tr.qa.painless-security.com:12308", "metric": 1, "interval": 30}]}}
trps_update_one_peer: adding message to queue.
trps_update_one_peer: update queued successfully.
tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "apc.sports.psec.us", "records": [{"record_type": "comm", "type": "apc", "role": "idp", "apcs": [], "interval": 30}, {"record_type": "comm", "type": "apc", "role": "idp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 30}]}}
trps_update_one_peer: adding message to queue.
tr_trpc_thread: sent 1 messages.
trps_update_one_peer: update queued successfully.
tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode_trp_upd: encoding inforec.
tr_trpc_thread: sent 1 messages.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "apc.sports.psec.us", "records": [{"record_type": "comm", "type": "apc", "role": "rp", "apcs": [], "interval": 30}, {"record_type": "comm", "type": "apc", "role": "rp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 30}]}}
trps_update_one_peer: adding message to queue.
tr_trpc_thread: sent 1 messages.
trps_update_one_peer: update queued successfully.
tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "redsox.com", "records": [{"record_type": "comm", "type": "apc", "role": "idp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 30}]}}
trps_update_one_peer: adding message to queue.
trps_update_one_peer: update queued successfully.
tr_msg_encode_trp_upd: encoding inforec.
tr_trpc_thread: sent 1 messages.tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "whitesox.mlb.us", "records": [{"record_type": "comm", "type": "apc", "role": "idp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 30}]}}

trps_update_one_peer: adding message to queue.
trps_update_one_peer: update queued successfully.
tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "nymets.com", "records": [{"record_type": "comm", "type": "apc", "role": "rp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 30}]}}
trps_update_one_peer: adding message to queue.
tr_trpc_thread: sent 1 messages.
trps_update_one_peer: update queued successfully.
tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "redsox.com", "records": [{"record_type": "comm", "type": "apc", "role": "rp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 30}]}}tr_trpc_thread: sent 1 messages.

trps_update_one_peer: adding message to queue.
trps_update_one_peer: update queued successfully.
tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "whitesox.mlb.us", "records": [{"record_type": "comm", "type": "apc", "role": "rp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 30}]}}
trps_update_one_peer: adding message to queue.
tr_trpc_thread: sent 2 messages.
trps_update_one_peer: update queued successfully.
tr_trpc_thread: notified of message, but queue empty
*** Error in `/home/jlr/Documents/painless-security/src/trust_router/tr/trust_router': double free or corruption (fasttop): 0x00005555557b14c0 ***

Thread 1 "trust_router" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

[jennifer-richards]

Fortunately, this was in the debugger...

(gdb) bt
#0  0x00007ffff63f40bb in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff63f5f5d in __GI_abort () at abort.c:90
#2  0x00007ffff643e28d in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff6565528 "*** Error in `%s': %s: 0x%s ***\n")
    at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007ffff644564a in malloc_printerr (action=<optimized out>, str=0x7ffff6565940 "double free or corruption (fasttop)", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5426
#4  0x00007ffff644773e in _int_free (av=0x7ffff6797c20 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:4175
#5  0x00007ffff644c44e in __GI___libc_free (mem=<optimized out>) at malloc.c:3145
#6  0x00007ffff6e14ef3 in  () at /usr/lib/x86_64-linux-gnu/libtalloc.so.2
#7  0x00007ffff6e14d70 in  () at /usr/lib/x86_64-linux-gnu/libtalloc.so.2
#8  0x00007ffff6e0d828 in _talloc_free () at /usr/lib/x86_64-linux-gnu/libtalloc.so.2
#9  0x00007ffff78dcddb in g_ptr_array_foreach () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#10 0x00007ffff78dce70 in  () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#11 0x00005555555610b7 in trps_update_one_peer (trps=trps@entry=0x5555557989f0, peer=peer@entry=0x5555557ab150, update_type=update_type@entry=TRP_UPDATE_SCHEDULED, realm=realm@entry=0x0, comm=comm@entry=0x0) at trp/trps.c:1759
#12 0x000055555556270b in trps_update (trps=trps@entry=0x5555557989f0, update_type=update_type@entry=TRP_UPDATE_SCHEDULED) at trp/trps.c:1791
#13 0x000055555555b440 in tr_trps_update (listener=<optimized out>, event=<optimized out>, arg=<optimized out>) at tr/tr_trp.c:361
#14 0x00007ffff768b8f5 in  () at /usr/lib/x86_64-linux-gnu/libevent-2.1.so.6
#15 0x00007ffff768c227 in event_base_loop () at /usr/lib/x86_64-linux-gnu/libevent-2.1.so.6
#16 0x0000555555558d6f in main (argc=<optimized out>, argv=<optimized out>) at tr/tr_main.c:309

[jennifer-richards]

The crash is here, on the g_ptr_array_free() call.

cleanup:
  if (updates!=NULL)
    g_ptr_array_free(updates, TRUE); /* frees any TRP_UPD records */
  talloc_free(tmp_ctx);
  return rc;
}

[jennifer-richards]

This runs in the trpc thread. Looking at both logs, in both cases that thread had sent an update just before the crash. In the first one (not in the debugger), it happened that the trps thread was also doing a sweep of the tables, but that is a red herring.

So I'm pretty sure these are both the same bug, despite appearing in different places in the logs.

[jennifer-richards]

One option is to do away with the free function set up for the GPtrArray. This is called during the g_ptr_array_free() because of the TRUE second argument. It is also called when g_ptr_array_remove_index_fast() is used to remove items rejected by the TRP update filters.

That has the minor drawback of hanging on to unneeded memory when items are removed from the GPtrArray. They're only held a short time longer, though, as everything is freed after the update is sent.

I'm hesitant to do this without a clear understanding of why that is fixing the problem (if it works in the first place).

[jennifer-richards]

Another debug log - this with messages printed when freeing TRP_UPDs (these start with >>>>>>). This is in a slightly different place, and unfortunately was not in gdb.

The crash did not happen during freeing of the GPtrArray this time - the debug messages all printed out and the update cycle that caused the crash can be seen starting in the log. That was probably a few second after the previous one had finished.

trps_update_one_peer: update queued successfully.
>>>>>>> freeing update for mariners.com/apc.sports.psec.us
>>>>>>>   record is route
>>>>>>> freeing update for apc.sports.psec.us/apc.sports.psec.us
>>>>>>>   record is route
>>>>>>> freeing update for test.org/apc.sports.psec.us
>>>>>>>   record is route
>>>>>>> freeing update for redsox.com/apc.sports.psec.us
>>>>>>>   record is route
>>>>>>> freeing update for example12.com/apc.sports.psec.us
>>>>>>>   record is route
>>>>>>> freeing update for whitesox.mlb.us/apc.sports.psec.us
>>>>>>>   record is route
>>>>>>> freeing update for reject-me.com/apc.sports.psec.us
>>>>>>>   record is route
>>>>>>> freeing update for reject-me/apc.sports.psec.us
>>>>>>>   record is route
>>>>>>> freeing update for example81.com/apc.sports.psec.us
>>>>>>>   record is route
>>>>>>> freeing update for apc.sports.psec.us/apc.sports.psec.us
>>>>>>>   record is comm
>>>>>>> freeing update for apc.sports.psec.us/apc.sports.psec.us
>>>>>>>   record is comm
>>>>>>> freeing update for redsox.com/apc.sports.psec.us
>>>>>>>   record is comm
>>>>>>> freeing update for whitesox.mlb.us/apc.sports.psec.us
>>>>>>>   record is comm
>>>>>>> freeing update for nymets.com/apc.sports.psec.us
>>>>>>>   record is comm
>>>>>>> freeing update for redsox.com/apc.sports.psec.us
>>>>>>>   record is comm
>>>>>>> freeing update for whitesox.mlb.us/apc.sports.psec.us
>>>>>>>   record is comm
trps_update: rc=0 after attempting update.
tr_trpc_thread: sent 1 messages.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_realm_entries: entered.
trp_route_set_triggered: setting route to apc.sports.psec.us/example81.com through trustrouter@apc.sports.psec.us to not triggered
trp_route_set_triggered: setting route to apc.sports.psec.us/reject-me through trustrouter@apc.sports.psec.us to not triggered
trp_route_set_triggered: setting route to apc.sports.psec.us/reject-me.com through trustrouter@apc.sports.psec.us to not triggered
trp_route_set_triggered: setting route to apc.sports.psec.us/whitesox.mlb.us through trustrouter@apc.sports.psec.us to not triggered
trp_route_set_triggered: setting route to apc.sports.psec.us/example12.com through trustrouter@apc.sports.psec.us to not triggered
trp_route_set_triggered: setting route to apc.sports.psec.us/redsox.com through trustrouter@apc.sports.psec.us to not triggered
trp_route_set_triggered: setting route to apc.sports.psec.us/test.org through trustrouter@apc.sports.psec.us to not triggered
trp_route_set_triggered: setting route to apc.sports.psec.us/apc.sports.psec.us through  to not triggered
trp_route_set_triggered: setting route to apc.sports.psec.us/apc.sports.psec.us through trustrouter@apc.sports.psec.us to not triggered
trp_route_set_triggered: setting route to apc.sports.psec.us/mariners.com through trustrouter@apc.sports.psec.us to not triggered
tr_trps_update: update interval=10
tr_connection_update: checking peer connections.
tr_trps_update: sending scheduled route/community updates.
trps_update_one_peer: preparing scheduled update for tr.qa.painless-security.com:12308
trps_update_one_peer: selecting route updates for tr.qa.painless-security.com:12308.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_selected_entry: looking through route table entries for realm mariners.com.
trp_rtable_get_selected_entry: ii=0.
trps_select_realm_update: tr.qa.painless-security.com:12308 vs trustrouter@apc.sports.psec.us
trp_upd_add_inforec: adding record.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_selected_entry: looking through route table entries for realm apc.sports.psec.us.
trp_rtable_get_selected_entry: ii=1.
trps_select_realm_update: tr.qa.painless-security.com:12308 vs 
trp_upd_add_inforec: adding record.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_selected_entry: looking through route table entries for realm test.org.
trp_rtable_get_selected_entry: ii=0.
trps_select_realm_update: tr.qa.painless-security.com:12308 vs trustrouter@apc.sports.psec.us
trp_upd_add_inforec: adding record.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_selected_entry: looking through route table entries for realm redsox.com.
trp_rtable_get_selected_entry: ii=0.
trps_select_realm_update: tr.qa.painless-security.com:12308 vs trustrouter@apc.sports.psec.us
trp_upd_add_inforec: adding record.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_selected_entry: looking through route table entries for realm example12.com.
trp_rtable_get_selected_entry: ii=0.
trps_select_realm_update: tr.qa.painless-security.com:12308 vs trustrouter@apc.sports.psec.us
trp_upd_add_inforec: adding record.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_selected_entry: looking through route table entries for realm whitesox.mlb.us.
trp_rtable_get_selected_entry: ii=0.
trps_select_realm_update: tr.qa.painless-security.com:12308 vs trustrouter@apc.sports.psec.us
trp_upd_add_inforec: adding record.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_selected_entry: looking through route table entries for realm reject-me.com.
trp_rtable_get_selected_entry: ii=0.
trps_select_realm_update: tr.qa.painless-security.com:12308 vs trustrouter@apc.sports.psec.us
trp_upd_add_inforec: adding record.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_selected_entry: looking through route table entries for realm reject-me.
trp_rtable_get_selected_entry: ii=0.
trps_select_realm_update: tr.qa.painless-security.com:12308 vs trustrouter@apc.sports.psec.us
trp_upd_add_inforec: adding record.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_selected_entry: looking through route table entries for realm example81.com.
trp_rtable_get_selected_entry: ii=0.
trps_select_realm_update: tr.qa.painless-security.com:12308 vs trustrouter@apc.sports.psec.us
trp_upd_add_inforec: adding record.
trps_update_one_peer: selecting community updates for tr.qa.painless-security.com:12308.
trps_select_comm_updates_for_peer: looking through community apc.sports.psec.us
trps_select_comm_updates_for_peer: adding realm apc.sports.psec.us
trp_upd_add_inforec: adding record.
trp_upd_add_inforec: adding record.
trps_select_comm_updates_for_peer: adding realm apc.sports.psec.us
trp_upd_add_inforec: adding record.
trp_upd_add_inforec: adding record.
trps_select_comm_updates_for_peer: adding realm redsox.com
trp_upd_add_inforec: adding record.
trps_select_comm_updates_for_peer: adding realm whitesox.mlb.us
trp_upd_add_inforec: adding record.
trps_select_comm_updates_for_peer: adding realm nymets.com
trp_upd_add_inforec: adding record.
trps_select_comm_updates_for_peer: adding realm redsox.com
trp_upd_add_inforec: adding record.
trps_select_comm_updates_for_peer: adding realm whitesox.mlb.us
trp_upd_add_inforec: adding record.
tr_fspec_matches: Field info_type value "comm" matches "comm" for trp_outbound filter.
tr_fspec_matches: Field comm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "comm" matches "comm" for trp_outbound filter.
tr_fspec_matches: Field comm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "comm" matches "comm" for trp_outbound filter.
tr_fspec_matches: Field comm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "comm" matches "comm" for trp_outbound filter.
tr_fspec_matches: Field comm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "comm" matches "comm" for trp_outbound filter.
tr_fspec_matches: Field comm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "comm" matches "comm" for trp_outbound filter.
tr_fspec_matches: Field comm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "comm" matches "comm" for trp_outbound filter.
tr_fspec_matches: Field comm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "comm" matches "comm" for trp_outbound filter.
tr_fspec_matches: Field comm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "comm" matches "comm" for trp_outbound filter.
tr_fspec_matches: Field comm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" does not match for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" matches "route" for trp_outbound filter.
tr_fspec_matches: Field realm value "example81.com" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" does not match for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" matches "route" for trp_outbound filter.
tr_fspec_matches: Field realm value "reject-me" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" does not match for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" matches "route" for trp_outbound filter.
tr_fspec_matches: Field realm value "reject-me.com" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" does not match for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" matches "route" for trp_outbound filter.
tr_fspec_matches: Field realm value "whitesox.mlb.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" does not match for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" matches "route" for trp_outbound filter.
tr_fspec_matches: Field realm value "example12.com" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" does not match for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" matches "route" for trp_outbound filter.
tr_fspec_matches: Field realm value "redsox.com" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" does not match for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" matches "route" for trp_outbound filter.
tr_fspec_matches: Field realm value "test.org" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" does not match for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" matches "route" for trp_outbound filter.
tr_fspec_matches: Field realm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" does not match for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" matches "route" for trp_outbound filter.
tr_fspec_matches: Field realm value "mariners.com" matches "*" for trp_outbound filter.
trps_update_one_peer: sending 16 update messages.
tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "mariners.com", "records": [{"record_type": "route", "trust_router": "tr.qa.painless-security.com:12308", "metric": 1, "interval": 10}]}}
trps_update_one_peer: adding message to queue.
trps_update_one_peer: update queued successfully.
tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "apc.sports.psec.us", "records": [{"record_type": "route", "trust_router": "tr.indiana.qa.painless-security.com:12308", "metric": 0, "interval": 10}]}}
trps_update_one_peer: adding message to queue.
*** Error in `tr/trust_router': double free or corruption (!prev): 0x000056155772ea40 ***
Aborted

[jennifer-richards]

Finally caught this with some (hopefully) useful debug messages in place.

The log snippet below shows the entire scheduled update process, plus the route/community tables at the start.

apc.sports.psec.us | apc.sports.psec.us | trustrouter@apc.sports.psec.us | 0 | tr.qa.painless-security.com:12308 | tr.qa.painless-security.com | 0 | 0 | 1970-01-02 19:56:29 | 0
apc.sports.psec.us | apc.sports.psec.us |  | 0 | tr.indiana.qa.painless-security.com:12308 |  | 1 | 1 | 1969-12-31 19:00:00 | 0
apc.sports.psec.us | example12.com | trustrouter@apc.sports.psec.us | 0 | tr.qa.painless-security.com:12308 | tr.qa.painless-security.com | 1 | 0 | 1970-01-02 19:56:29 | 0
apc.sports.psec.us | example81.com | trustrouter@apc.sports.psec.us | 0 | tr.qa.painless-security.com:12308 | tr.qa.painless-security.com | 1 | 0 | 1970-01-02 19:56:29 | 0
apc.sports.psec.us | mariners.com | trustrouter@apc.sports.psec.us | 0 | tr.qa.painless-security.com:12308 | tr.qa.painless-security.com | 1 | 0 | 1970-01-02 19:56:29 | 0
apc.sports.psec.us | redsox.com | trustrouter@apc.sports.psec.us | 0 | tr.qa.painless-security.com:12308 | tr.qa.painless-security.com | 1 | 0 | 1970-01-02 19:56:29 | 0
apc.sports.psec.us | reject-me | trustrouter@apc.sports.psec.us | 0 | tr.qa.painless-security.com:12308 | tr.qa.painless-security.com | 1 | 0 | 1970-01-02 19:56:29 | 0
apc.sports.psec.us | reject-me.com | trustrouter@apc.sports.psec.us | 0 | tr.qa.painless-security.com:12308 | tr.qa.painless-security.com | 1 | 0 | 1970-01-02 19:56:29 | 0
apc.sports.psec.us | test.org | trustrouter@apc.sports.psec.us | 0 | tr.qa.painless-security.com:12308 | tr.qa.painless-security.com | 1 | 0 | 1970-01-02 19:56:29 | 0
apc.sports.psec.us | whitesox.mlb.us | trustrouter@apc.sports.psec.us | 0 | tr.qa.painless-security.com:12308 | tr.qa.painless-security.com | 1 | 0 | 1970-01-02 19:56:29 | 0

>> Membership table start <<
* idp apc.sports.psec.us/apc.sports.psec.us
  null origin (0x561e7effaf80) - prov: 
  tr.qa.painless-security.com:12308 (0x561e7f012e10) - prov: tr.qa.painless-security.com:12308

* rp apc.sports.psec.us/apc.sports.psec.us
  null origin (0x561e7effb2d0) - prov: 
  tr.qa.painless-security.com:12308 (0x561e7effc280) - prov: tr.qa.painless-security.com:12308

* idp redsox.com/apc.sports.psec.us
  tr.qa.painless-security.com:12308 (0x561e7eff5800) - prov: tr.qa.painless-security.com:12308

* idp whitesox.mlb.us/apc.sports.psec.us
  tr.qa.painless-security.com:12308 (0x561e7eff58c0) - prov: tr.qa.painless-security.com:12308

* rp nymets.com/apc.sports.psec.us
  tr.qa.painless-security.com:12308 (0x561e7eff5a90) - prov: tr.qa.painless-security.com:12308

* rp redsox.com/apc.sports.psec.us
  tr.qa.painless-security.com:12308 (0x561e7f012990) - prov: tr.qa.painless-security.com:12308

* rp whitesox.mlb.us/apc.sports.psec.us
  tr.qa.painless-security.com:12308 (0x561e7eff5cf0) - prov: tr.qa.painless-security.com:12308

tr_connection_update: checking peer connections.
tr_trps_update: sending scheduled route/community updates.
trps_update_one_peer: preparing scheduled update for tr.qa.painless-security.com:12308
trps_update_one_peer: selecting route updates for tr.qa.painless-security.com:12308.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_selected_entry: looking through route table entries for realm mariners.com.
trp_rtable_get_selected_entry: ii=0.
trps_select_realm_update: tr.qa.painless-security.com:12308 vs trustrouter@apc.sports.psec.us
trp_upd_add_inforec: adding record.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_selected_entry: looking through route table entries for realm apc.sports.psec.us.
trp_rtable_get_selected_entry: ii=1.
trps_select_realm_update: tr.qa.painless-security.com:12308 vs 
trp_upd_add_inforec: adding record.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_selected_entry: looking through route table entries for realm test.org.
trp_rtable_get_selected_entry: ii=0.
trps_select_realm_update: tr.qa.painless-security.com:12308 vs trustrouter@apc.sports.psec.us
trp_upd_add_inforec: adding record.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_selected_entry: looking through route table entries for realm redsox.com.
trp_rtable_get_selected_entry: ii=0.
trps_select_realm_update: tr.qa.painless-security.com:12308 vs trustrouter@apc.sports.psec.us
trp_upd_add_inforec: adding record.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_selected_entry: looking through route table entries for realm example12.com.
trp_rtable_get_selected_entry: ii=0.
trps_select_realm_update: tr.qa.painless-security.com:12308 vs trustrouter@apc.sports.psec.us
trp_upd_add_inforec: adding record.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_selected_entry: looking through route table entries for realm whitesox.mlb.us.
trp_rtable_get_selected_entry: ii=0.
trps_select_realm_update: tr.qa.painless-security.com:12308 vs trustrouter@apc.sports.psec.us
trp_upd_add_inforec: adding record.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_selected_entry: looking through route table entries for realm reject-me.com.
trp_rtable_get_selected_entry: ii=0.
trps_select_realm_update: tr.qa.painless-security.com:12308 vs trustrouter@apc.sports.psec.us
trp_upd_add_inforec: adding record.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_selected_entry: looking through route table entries for realm reject-me.
trp_rtable_get_selected_entry: ii=0.
trps_select_realm_update: tr.qa.painless-security.com:12308 vs trustrouter@apc.sports.psec.us
trp_upd_add_inforec: adding record.
trp_rtable_get_realm_entries: entered.
trp_rtable_get_selected_entry: looking through route table entries for realm example81.com.
trp_rtable_get_selected_entry: ii=0.
trps_select_realm_update: tr.qa.painless-security.com:12308 vs trustrouter@apc.sports.psec.us
trp_upd_add_inforec: adding record.
trps_update_one_peer: selecting community updates for tr.qa.painless-security.com:12308.
trps_select_comm_updates_for_peer: looking through community apc.sports.psec.us
trps_select_comm_updates_for_peer: adding realm apc.sports.psec.us
trp_upd_add_inforec: adding record.
trp_upd_add_inforec: adding record.
trps_select_comm_updates_for_peer: adding realm apc.sports.psec.us
trp_upd_add_inforec: adding record.
trp_upd_add_inforec: adding record.
trps_select_comm_updates_for_peer: adding realm redsox.com
trp_upd_add_inforec: adding record.
trps_select_comm_updates_for_peer: adding realm whitesox.mlb.us
trp_upd_add_inforec: adding record.
trps_select_comm_updates_for_peer: adding realm nymets.com
trp_upd_add_inforec: adding record.
trps_select_comm_updates_for_peer: adding realm redsox.com
trp_upd_add_inforec: adding record.
trps_select_comm_updates_for_peer: adding realm whitesox.mlb.us
trp_upd_add_inforec: adding record.
tr_fspec_matches: Field info_type value "comm" matches "comm" for trp_outbound filter.
tr_fspec_matches: Field comm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "comm" matches "comm" for trp_outbound filter.
tr_fspec_matches: Field comm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "comm" matches "comm" for trp_outbound filter.
tr_fspec_matches: Field comm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "comm" matches "comm" for trp_outbound filter.
tr_fspec_matches: Field comm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "comm" matches "comm" for trp_outbound filter.
tr_fspec_matches: Field comm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "comm" matches "comm" for trp_outbound filter.
tr_fspec_matches: Field comm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "comm" matches "comm" for trp_outbound filter.
tr_fspec_matches: Field comm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "comm" matches "comm" for trp_outbound filter.
tr_fspec_matches: Field comm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "comm" matches "comm" for trp_outbound filter.
tr_fspec_matches: Field comm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" does not match for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" matches "route" for trp_outbound filter.
tr_fspec_matches: Field realm value "example81.com" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" does not match for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" matches "route" for trp_outbound filter.
tr_fspec_matches: Field realm value "reject-me" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" does not match for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" matches "route" for trp_outbound filter.
tr_fspec_matches: Field realm value "reject-me.com" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" does not match for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" matches "route" for trp_outbound filter.
tr_fspec_matches: Field realm value "whitesox.mlb.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" does not match for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" matches "route" for trp_outbound filter.
tr_fspec_matches: Field realm value "example12.com" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" does not match for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" matches "route" for trp_outbound filter.
tr_fspec_matches: Field realm value "redsox.com" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" does not match for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" matches "route" for trp_outbound filter.
tr_fspec_matches: Field realm value "test.org" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" does not match for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" matches "route" for trp_outbound filter.
tr_fspec_matches: Field realm value "apc.sports.psec.us" matches "*" for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" does not match for trp_outbound filter.
tr_fspec_matches: Field info_type value "route" matches "route" for trp_outbound filter.
tr_fspec_matches: Field realm value "mariners.com" matches "*" for trp_outbound filter.
trps_update_one_peer: sending 16 update messages.
tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "mariners.com", "records": [{"record_type": "route", "trust_router": "tr.qa.painless-security.com:12308", "metric": 1, "interval": 10}]}}
trps_update_one_peer: adding message to queue.
full talloc report on 'talloc_new: trp/trps.c:260' (total      0 bytes in   1 blocks)
trps_update_one_peer: update queued successfully.
full talloc report on '{"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "mariners.com", "records": [{"record_type": "route", "trust_router": "tr.qa.painless-security.com:12308", "metric": 1, "interval": 10}]}}' (total    222 bytes in   1 blocks)
tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "apc.sports.psec.us", "records": [{"record_type": "route", "trust_router": "tr.indiana.qa.painless-security.com:12308", "metric": 0, "interval": 10}]}}
trps_update_one_peer: adding message to queue.
tr_trpc_thread: sent 2 messages.
tr_trpc_thread: notified of message, but queue emptyfull talloc report on 'talloc_new: trp/trps.c:260' (total      0 bytes in   1 blocks)

trps_update_one_peer: update queued successfully.
full talloc report on '{"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "apc.sports.psec.us", "records": [{"record_type": "route", "trust_router": "tr.indiana.qa.painless-security.com:12308", "metric": 0, "interval": 10}]}}' (total    236 bytes in   1 blocks)
tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "test.org", "records": [{"record_type": "route", "trust_router": "tr.qa.painless-security.com:12308", "metric": 1, "interval": 10}]}}
trps_update_one_peer: adding message to queue.
full talloc report on 'talloc_new: trp/trps.c:260' (total      0 bytes in   1 blocks)
trps_update_one_peer: update queued successfully.
full talloc report on '{"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "test.org", "records": [{"record_type": "route", "trust_router": "tr.qa.painless-security.com:12308", "metric": 1, "interval": 10}]}}' (total    218 bytes in   1 blocks)
tr_msg_encode_trp_upd: encoding inforec.
tr_trpc_thread: sent 1 messages.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "redsox.com", "records": [{"record_type": "route", "trust_router": "tr.qa.painless-security.com:12308", "metric": 1, "interval": 10}]}}
trps_update_one_peer: adding message to queue.
full talloc report on 'talloc_new: trp/trps.c:260' (total      0 bytes in   1 blocks)
trps_update_one_peer: update queued successfully.
full talloc report on '{"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "redsox.com", "records": [{"record_type": "route", "trust_router": "tr.qa.painless-security.com:12308", "metric": 1, "interval": 10}]}}' (total    220 bytes in   1 blocks)
tr_msg_encode_trp_upd: encoding inforec.
tr_trpc_thread: sent 1 messages.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "example12.com", "records": [{"record_type": "route", "trust_router": "tr.qa.painless-security.com:12308", "metric": 1, "interval": 10}]}}
trps_update_one_peer: adding message to queue.
full talloc report on 'talloc_new: trp/trps.c:260' (total      0 bytes in   1 blocks)
trps_update_one_peer: update queued successfully.
full talloc report on '{"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "example12.com", "records": [{"record_type": "route", "trust_router": "tr.qa.painless-security.com:12308", "metric": 1, "interval": 10}]}}' (total    223 bytes in   1 blocks)
tr_msg_encode_trp_upd: encoding inforec.
tr_trpc_thread: sent 1 messages.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "whitesox.mlb.us", "records": [{"record_type": "route", "trust_router": "tr.qa.painless-security.com:12308", "metric": 1, "interval": 10}]}}
trps_update_one_peer: adding message to queue.
full talloc report on 'talloc_new: trp/trps.c:260' (total      0 bytes in   1 blocks)
trps_update_one_peer: update queued successfully.
full talloc report on '{"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "whitesox.mlb.us", "records": [{"record_type": "route", "trust_router": "tr.qa.painless-security.com:12308", "metric": 1, "interval": 10}]}}' (total    225 bytes in   1 blocks)
tr_msg_encode_trp_upd: encoding inforec.
tr_trpc_thread: sent 1 messages.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "reject-me.com", "records": [{"record_type": "route", "trust_router": "tr.qa.painless-security.com:12308", "metric": 1, "interval": 10}]}}
trps_update_one_peer: adding message to queue.
full talloc report on 'talloc_new: trp/trps.c:260' (total      0 bytes in   1 blocks)
trps_update_one_peer: update queued successfully.
full talloc report on '{"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "reject-me.com", "records": [{"record_type": "route", "trust_router": "tr.qa.painless-security.com:12308", "metric": 1, "interval": 10}]}}' (total    223 bytes in   1 blocks)
tr_trpc_thread: sent 1 messages.
tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "reject-me", "records": [{"record_type": "route", "trust_router": "tr.qa.painless-security.com:12308", "metric": 1, "interval": 10}]}}
trps_update_one_peer: adding message to queue.
full talloc report on 'talloc_new: trp/trps.c:260' (total      0 bytes in   1 blocks)
trps_update_one_peer: update queued successfully.
full talloc report on '{"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "reject-me", "records": [{"record_type": "route", "trust_router": "tr.qa.painless-security.com:12308", "metric": 1, "interval": 10}]}}' (total    219 bytes in   1 blocks)
tr_trpc_thread: sent 1 messages.
tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "example81.com", "records": [{"record_type": "route", "trust_router": "tr.qa.painless-security.com:12308", "metric": 1, "interval": 10}]}}
trps_update_one_peer: adding message to queue.
full talloc report on 'talloc_new: trp/trps.c:260' (total      0 bytes in   1 blocks)
trps_update_one_peer: update queued successfully.
full talloc report on '{"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "example81.com", "records": [{"record_type": "route", "trust_router": "tr.qa.painless-security.com:12308", "metric": 1, "interval": 10}]}}' (total    223 bytes in   1 blocks)
tr_trpc_thread: sent 1 messages.
tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "apc.sports.psec.us", "records": [{"record_type": "comm", "type": "apc", "role": "idp", "apcs": [], "interval": 10}, {"record_type": "comm", "type": "apc", "role": "idp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 10}]}}
trps_update_one_peer: adding message to queue.
full talloc report on 'talloc_new: trp/trps.c:260' (total      0 bytes in   1 blocks)
trps_update_one_peer: update queued successfully.
full talloc report on '{"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "apc.sports.psec.us", "records": [{"record_type": "comm", "type": "apc", "role": "idp", "apcs": [], "interval": 10}, {"record_type": "comm", "type": "apc", "role": "idp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 10}]}}' (total    339 bytes in   1 blocks)
tr_msg_encode_trp_upd: encoding inforec.
tr_msg_encode_trp_upd: encoding inforec.
tr_trpc_thread: sent 1 messages.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "apc.sports.psec.us", "records": [{"record_type": "comm", "type": "apc", "role": "rp", "apcs": [], "interval": 10}, {"record_type": "comm", "type": "apc", "role": "rp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 10}]}}
trps_update_one_peer: adding message to queue.
full talloc report on 'talloc_new: trp/trps.c:260' (total      0 bytes in   1 blocks)
trps_update_one_peer: update queued successfully.
full talloc report on '{"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "apc.sports.psec.us", "records": [{"record_type": "comm", "type": "apc", "role": "rp", "apcs": [], "interval": 10}, {"record_type": "comm", "type": "apc", "role": "rp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 10}]}}' (total    337 bytes in   1 blocks)
tr_msg_encode_trp_upd: encoding inforec.
tr_trpc_thread: sent 1 messages.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "redsox.com", "records": [{"record_type": "comm", "type": "apc", "role": "idp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 10}]}}
trps_update_one_peer: adding message to queue.
full talloc report on 'talloc_new: trp/trps.c:260' (total      0 bytes in   1 blocks)
trps_update_one_peer: update queued successfully.
full talloc report on '{"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "redsox.com", "records": [{"record_type": "comm", "type": "apc", "role": "idp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 10}]}}' (total    248 bytes in   1 blocks)
tr_msg_encode_trp_upd: encoding inforec.
tr_trpc_thread: sent 1 messages.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "whitesox.mlb.us", "records": [{"record_type": "comm", "type": "apc", "role": "idp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 10}]}}
trps_update_one_peer: adding message to queue.
full talloc report on 'talloc_new: trp/trps.c:260' (total      0 bytes in   1 blocks)
trps_update_one_peer: update queued successfully.
full talloc report on '{"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "whitesox.mlb.us", "records": [{"record_type": "comm", "type": "apc", "role": "idp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 10}]}}' (total    253 bytes in   1 blocks)
tr_msg_encode_trp_upd: encoding inforec.
tr_trpc_thread: sent 1 messages.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "nymets.com", "records": [{"record_type": "comm", "type": "apc", "role": "rp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 10}]}}
trps_update_one_peer: adding message to queue.
full talloc report on 'talloc_new: trp/trps.c:260' (total      0 bytes in   1 blocks)
trps_update_one_peer: update queued successfully.
full talloc report on '{"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "nymets.com", "records": [{"record_type": "comm", "type": "apc", "role": "rp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 10}]}}' (total    247 bytes in   1 blocks)
tr_msg_encode_trp_upd: encoding inforec.
tr_trpc_thread: sent 1 messages.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "redsox.com", "records": [{"record_type": "comm", "type": "apc", "role": "rp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 10}]}}
trps_update_one_peer: adding message to queue.
full talloc report on 'talloc_new: trp/trps.c:260' (total      0 bytes in   1 blocks)
trps_update_one_peer: update queued successfully.
full talloc report on '{"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "redsox.com", "records": [{"record_type": "comm", "type": "apc", "role": "rp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 10}]}}' (total    247 bytes in   1 blocks)
tr_msg_encode_trp_upd: encoding inforec.
tr_trpc_thread: sent 1 messages.
tr_msg_encode: outgoing msg={"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "whitesox.mlb.us", "records": [{"record_type": "comm", "type": "apc", "role": "rp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 10}]}}
trps_update_one_peer: adding message to queue.
full talloc report on 'talloc_new: trp/trps.c:260' (total      0 bytes in   1 blocks)
trps_update_one_peer: update queued successfully.
full talloc report on '{"msg_type": "trp_update", "msg_body": {"community": "apc.sports.psec.us", "realm": "whitesox.mlb.us", "records": [{"record_type": "comm", "type": "apc", "role": "rp", "apcs": [], "provenance": ["tr.qa.painless-security.com:12308"], "interval": 10}]}}' (total    252 bytes in   1 blocks)
tr_trpc_thread: sent 1 messages.
>>>>>>> freeing update for mariners.com/apc.sports.psec.us
>>>>>>>   record is route
>>>>>>> freeing update for apc.sports.psec.us/apc.sports.psec.us
>>>>>>>   record is route
*** Error in `tr/trust_router': double free or corruption (!prev): 0x0000561e7f0149f0 ***
Aborted

[jennifer-richards]

Testing with a version that includes various bugs that I fixed while investigating this, I have yet to see any faults. Two peered trust routers stayed up for almost 19 hours without a fault.

I am keeping this open because I have yet to find the origin of the faults or to fix any bugs that I am convinced could have caused the errors. A few of the fixed bugs involved memory leaks and pointer errors, though, so it is at least plausible that the underlying problem has been fixed.

Since these faults do not seem to be occurring, I am going to lower the priority and move this out of the immediate release milestone.

[jennifer-richards]

This is still present, but much less frequent. I've had three machines (two Debian jessie, one Ubuntu 17.10) with about 90 hours of uptime each (total of 250-300 hours so far) and it has occurred once (on the Ubuntu system).

So still need to track this down, but if these numbers are representative and it's happening once every few days to couple weeks I think we can leave it as a not-urgent priority.

But let's not forget about it.