Closed bfaccini closed 1 year ago
BTW, looking at the source code, I wonder if the rxm_conn_close() call before rxm_flush_msg_cq() call in rxm_conn_handle_notify () is necessary since rxm_conn_close() will be later called from rxm_conn_free() :
1160 static int rxm_conn_handle_notify(struct fi_eq_entry *eq_entry)
1161 {
1162 struct rxm_cmap *cmap;
1163 struct rxm_cmap_handle *handle;
1164
1165 FI_INFO(&rxm_prov, FI_LOG_EP_CTRL, "notify event %" PRIu64 "\n",
1166 eq_entry->data);
1167
1168 if ((enum rxm_cmap_signal) eq_entry->data != RXM_CMAP_FREE)
1169 return -FI_EOTHER;
1170
1171 handle = eq_entry->context;
1172 assert(handle->state == RXM_CMAP_SHUTDOWN);
1173 FI_DBG(&rxm_prov, FI_LOG_EP_CTRL, "freeing handle: %p\n", handle);
1174 cmap = handle->cmap;
1175
1176 rxm_conn_close(handle); <<<<<<<<<<< ????
1177
1178 // after closing the connection, we need to flush any dangling references to the
1179 // handle from msg_cq entries that have not been cleaned up yet, otherwise we
1180 // could run into problems during CQ cleanup. these entries will be errored so
1181 // keep reading through EAVAIL.
1182 rxm_flush_msg_cq(cmap->ep);
1183
1184 if (handle->peer) {
1185 dlist_remove(&handle->peer->entry);
1186 free(handle->peer);
1187 handle->peer = NULL;
1188 } else {
1189 cmap->handles_av[handle->fi_addr] = NULL;
1190 }
1191 rxm_conn_free(handle);
1192 return 0;
1193 }
1194
"prov/rxm/src/rxm_conn.c" line 1176 of 1588 --74%-- col 2-9
rxm_conn_close() calls fi_close() on the msg endpoint. That ends up completing any outstanding operations (including posted receives) as canceled. After rxm_conn_close(), rxm_flush_msg_cq() should remove those completions, as indicaed in the comment. So, that part of the flow is needed.
Well, ok, so since you think my assumption was wrong, does anybody from libfabric team have an idea concerning what could wrong to generate this kind of crash/SEGV ?
The CM code in rxm is extremely complex. PR #6833 replaces the implementation and simplifies the code. Those changes aren't quite ready, and I haven't had time to debug the issues there yet. The locking in #6833 is much easier to follow, with proper reference counting on peer addresses. Honestly, I think continuing down that path and getting it in shape to merge is the ideal path.
There were several fixes applied at the tcp layer to handle shutdown better and avoid reporting completions for internal transfers up to the app. Those are already in tree.
The CM code in rxm has been significantly been modified. Does this issue still occur with the latest libfabric main(master) branch?
Do you know which libfabric version will (or already ?) integrate changes from #6833 ?
It is in v1.13.2rc1 and will be in v1.13.2 or v1.14.0 (November release).
This issue is stale because it has been open 360 days with no activity. Remove stale label or comment, otherwise it will be closed in 7 days.
Running DAOS v1.2.0 with libfabric v1.12.0, we have triggered a crash/SEGV with the following back-trace :
the direct cause of the SEGV is because "msg_ep" pointer/field of "struct rxm_conn" is unexpectedly NULL :
when this seems finally correct for a cmap handle in RXM_CMAP_SHUTDOWN state !!...
Looks like PR-5424 may not have handled all cleanup needs.