We've opted to pursue a strategy of not wrapping the eRSD pull and value set insertions in a transaction, since that would either be a) slow and painful at the concept/value set level, or b) monolithically unwieldy if the entire process were wrapped in a transaction. Instead, what we'll do for each value set we insert is to use a "sanity check" function to make sure that everything got inserted. Given a particular value set, this function will query the eRSD (or parse our pulled extract of it, either way works so long as it's the complete source-of-truth eRSD data) to determine how many concepts / pieces of data should have been inserted into the DB as part of adding this value set. It will also identify which concepts / pieces of data those are, e.g. by name or ID. With this list in hand, the function will query our database after the insert call has been performed to see how many concepts come back. If these numbers agree, and if the concepts are all the same in both cases, we consider the insertion a success and move on to the next valueset. If the numbers disagree or the data don't match, we flag the value set in question for a repeat pull / insert at the end of the DB creation process, reporting to the user that there were problems with it as we try to fix them. This repeat process is why we want the insert to replace entries on conflict, so that a value set that errors doesn't break this process by having multiple copies of the same concept after we attempt to pull it again.
linear[bot]
commented
3 weeks ago
We've opted to pursue a strategy of not wrapping the eRSD pull and value set insertions in a transaction, since that would either be a) slow and painful at the concept/value set level, or b) monolithically unwieldy if the entire process were wrapped in a transaction. Instead, what we'll do for each value set we insert is to use a "sanity check" function to make sure that everything got inserted. Given a particular value set, this function will query the eRSD (or parse our pulled extract of it, either way works so long as it's the complete source-of-truth eRSD data) to determine how many concepts / pieces of data should have been inserted into the DB as part of adding this value set. It will also identify which concepts / pieces of data those are, e.g. by name or ID. With this list in hand, the function will query our database after the insert call has been performed to see how many concepts come back. If these numbers agree, and if the concepts are all the same in both cases, we consider the insertion a success and move on to the next valueset. If the numbers disagree or the data don't match, we flag the value set in question for a repeat pull / insert at the end of the DB creation process, reporting to the user that there were problems with it as we try to fix them. This repeat process is why we want the insert to replace entries on conflict, so that a value set that errors doesn't break this process by having multiple copies of the same concept after we attempt to pull it again.