search

ctsit / redcap_oncore_client

REDCap OnCore Client external module provides integration with Forte Research's OnCore
Other
4 stars 7 forks source link

Oncore client is unreliable when pulling subjects and record auto-numbering is off. #35

Closed pbchase closed 5 years ago

pbchase commented 5 years ago

Oncore client 2.2.0 with RCE 2.3.0 often fails to pull subjects into redcap when autonumbering is off. fails have been seen with 20-ish records pulled at a time.

When record auto-numbering is on, the client seems 100% reliable.

ChemiKyle commented 5 years ago

With autonumbering on and the PrimaryIdentifier mapped to record_id, sometimes trying to pull the same number of subjects (i.e. 25) will fail on one attempt, and retrying it again will work.

ChemiKyle commented 5 years ago

Not reproducible in redcap.test if using the oncore-staging WSDL and SIP Protocol lookup

Switching all OnCore Client options to be the same (aside from assignment of the Institution ID in the Control Center, due to redcap.test not having Shibboleth) the issue is still not reproducible in redcap.test. Both REDCap instances are using v9.1.1

In prod, if autonumbering is not turned on, the pull function is never sometimes not called. When it does work, a record is made and the record_id is autoincremented...

ChemiKyle commented 5 years ago

This error is due to REDCap Entity, specifically this portion of the Entity code that processes bulk operations is occasionally failing:
https://github.com/ctsit/redcap_entity/blob/b070de0e283ddee5b4ffb155fb3cd6c79ae2f9f1/classes/EntityList.php#L705-L707

The function loadInstances is failing during the processing of entity_type oncore_subject_diff:
https://github.com/ctsit/redcap_entity/blob/b070de0e283ddee5b4ffb155fb3cd6c79ae2f9f1/classes/EntityFactory.php#L163-L181

One $id in $ids (that is not consistent among failures) does not exist, causing the entire function to return false. $id is a property of the redcap_entity_<$entity_type> table.

UPDATE:
the id attempted is always 1 less than the lowest id stored in redcap_entity_oncore_subject_diff, possible off by one error or the ids list is being populated before the diff table is rebuilt, but only when autonumbering is off, and only sometimes...

ChemiKyle commented 5 years ago

Uncommonly, after a pull failure, clicking the Pull OnCore Subjects option under External Modules in the sidebar will rerender the page, with double the subjects.

ChemiKyle commented 5 years ago

Simply commenting out the rebuildSubjectsDiffList() solves the problem. But if all records are erased, navigating back to the Pull OnCore Subjects page must be manually refreshed or it uses the cached results missing subjects who had been successfully pulled (the page does say to refresh it, however)

https://github.com/ctsit/redcap_oncore_client/blob/96e634362fc39f8b6577a5ac6461c04c3e2b8b32/classes/entity/list/SubjectsDiffList.php#L24-L26

Moving it after the parent::renderPageBody(); call has the same effect.

I suspect the issue is with function isListUpdated's query which checks for the most recent log events causing rebuilding when not needed. This portion executes prior to page render though, and doesn't seem like it should be changing the list of ids that get passed to Entity for processing, but it does.

https://github.com/ctsit/redcap_oncore_client/blob/96e634362fc39f8b6577a5ac6461c04c3e2b8b32/classes/entity/list/SubjectsDiffList.php#L238-L259

It may be rebuilding through a method I didn't flag after the page renders, as the entity function that checks the subject diffs by their ids (which causes a subject pull to not be performed as all the ids have been shifted up by count($ids)) does not fail after the pull is attempted. Likely here:
https://github.com/ctsit/redcap_oncore_client/blob/96e634362fc39f8b6577a5ac6461c04c3e2b8b32/ExternalModule.php#L443

pbchase commented 5 years ago

fixed by PR #36