waldoj
commented
6 years ago My guess is that the time commitment of this is probably substantial, but I'll think out loud here and see where it gets us.
Let's imagine that we get the standard three references for each vendor. And let's imagine that we get (say) six bids. So that's 18 references to be checked. For each reference to be checked, let's say it'll take one person 30 minutes to arrange that interview—that's to coordinate the subject's calendar with OCS's calendar with 18F's calendar. Then the interview itself lasts another 30 minutes, for n people on an interview team. (Let's call n 3 people. We'll imagine that we keep it real small.) And then somebody needs to take, say, 15 minutes to organize the notes from that call. Finally, when all of those calls are finished, somebody needs to summarize those notes in a report to present to the larger eval team. Let's call that 2 hours. And then we all need to meet and talk about what we found. That's perhaps 30 minutes for the x people who comprise the eval team. (Let's call x 8 people.)
So that's 135 cumulative minutes per reference, or 2,430 minutes for all 18 references. Then another 360 minutes to write and present the concluding report. That's 46.5 hours in total, to check references.
(Does my math check out?)
Whether this is a good investment of 46.5 hours spread among ~8 different people, I don't have the procurement expertise to say. But I will say that, with our prior DHSS procurement, nobody raised the need to check references for every (or any) applicant, and if our failure to do so was problematic, I don't think any of us are aware of that.
randyhart
susanjabal
For Section 4.07, do we want to request References and points of contact? If so, will we commit to calling them and using that input in our evaluation? Or will this be met by reviewing the samples and code provided above?