Suggest eliminating requirement of "links to Git repositories" from Evaluation Criteria

[greggersh]

Question/Comment on this U.S. Tax Court RFP

Name and affiliation

Greg Gershman, CEO, Ad Hoc LLC

Section of RFP documents

Evaluation Criteria - Technical Submissions

Question/Comment

The RFQ requires vendors to provide "links to Git repositories," by which it seems to mean references to version control repositories of other similar work to the work defined in the RFQ.

The first suggestion is to change the phrasing from "Git repositories" to "source code or source code repositories." Git is a specific form of version control, but by no means the only tool used for that purpose. There are many other version control tools, and unless the Tax Court is specifically looking to ascertain a vendor's ability and experience working with Git, we suggest broadening the definition to include other version control systems as well.

While Ad Hoc can basically meet this requirement, we find that these specific kinds of requirements are burdensome on vendors, both in determining if repositories meet the exact criteria, and because much of our work, even though with US federal government agencies, is not made publicly available by our customers.

In regards to the difficulty in meeting the exact criteria, Ad Hoc has worked on a project that we believe should be considered adequate, but is difficult for us to determine if it would be accepted by the government as valid. Ad Hoc is contracted by the US Department of Veterans Affairs to build Vets.gov. Our team employs about 20 software engineers, and our program is over three years. Our team, practically, is broken down into smaller product development teams, which typically consist of 3-5 engineers (along with others, for total team sizes around 7-9 personnel). While we feel this meets the government's requirements, if not exactly in terms, but in spirit, we are concerned because of the requirement that projects have "staffing profiles of approximately five (5) to nine (9) Full Time Equivalent (FTE) personnel." Finding something that matches these exact criteria, is made publicly available by the government, can be very challenging for many vendors. In addition, strict requirements that are met in spirit by vendors may create grounds for protest of the final award.

Further, in our experience, the fact that our work for government is made publicly available is not under a vendor's control. We do work for government agencies that we feel is very relevant to this RFQ's requirements, but is kept private by the decision of the agency. Vendors are not able to easily (or at all) provide access to third parties to view or evaluate this source code, even if the third party is another federal agency. Therefore, we do not believe there is anything distinctive gained by the government in ascertaining that a vendor is able to provide these links. In addition, the practice of making code publicly available is rare to non-existent in the commercial sector, in particular applications such as the one that Tax Court is looking to build. This requirement will eliminate many qualified vendors that have very valid and relevant experience.

Therefore, we suggest revising or removing the criteria to provide links to Git (or version control/source code in general) as a part of this RFQ. We believe doing so will result in better competition from a larger pool of qualified vendors, and ultimately better results for the government.

It would help to better understand what Tax Court is looking to determine with the requirement. If there is a desire to modify the criteria to hone in on specific qualities that Tax Court feels would make a successful vendor, we are happy to provide some feedback on how to do that in ways other than requiring links to publicly-available repositories.

[randyhart]

Thank you for raising these points. The Court would like other vendors to share their views on these issues as well.

[cscairns]

I 100% agree with @greggersh. I've raised similar concerns in other forums, and glad to see he's raised them here. Happy to share thoughts on alternative evaluation approaches as well.

[JQuaggles]

Agree with Greg and Chris on this as well and look forward to working with ya'll on alternatives.

[randyhart]

We appreciate the feedback and welcome suggested alternatives from vendors. What we're trying to get to with this evaluation factor is finding demonstrated experience working on user-centered and iterative software development teams. We expect that looking through code repositories will not only show us the quality of the code, but also how members of the development teams work with each other. We are open to other suggested ways that vendors can demonstrate their readiness based on previous work examples. Please provide those suggestions for us to consider as we finalize the RFQ.

[randyhart]

@greggersh @cscairns @JQuaggles ☝️

[bengm]

We similarly looked at this and determined that the current formulation would make it hard for us to be evaluated well, primarily because only a minority of work would have the source code available in some way. I like the concept in general, but few government agencies (or private firms) open source most work that is done.

[JessicaMarine]

@bengm The Court is open to other suggested ways that vendors can demonstrate their readiness based on previous work examples. Please provide any suggestions you may have for us to consider as we finalize the RFQ.

[kaitlin]

@randyhart if you're interested in experience on user-centered projects, GitHub is a poor proxy for that. Most of the UX folks I know don't work within GitHub repos. Instead, I suggest asking for a user research plan and/or script, as well as a user research synthesis artifact from a product.

If you keep the requirement for evaluating source code in a repo, I suggest getting rid of some of the other constraints that make it difficult or impossible to provide a matching repo (period of performance and team size limitations for instance). Also to @greggersh 's point, they're too vague to know if we're in compliance (i.e. Is total team size measured by total number of contributors or the number of contributors for a certain period of time?).

But ideally, instead of the repo requirement, you could have a small code challenge. I think something roughly 1.5-2x the size and scope of the homeworks that Ad Hoc uses to screen engineers (which 18F also uses to screen engineers) would be appropriate. For instance, maybe a small challenge that asks vendors to create a simple interface that allows a petitioner to submit a .pdf or .doc and then stores it in cloud based storage, like S3, and then marks it as "ready for processing" in a database.

[cscairns]

i'd also recommend looking into how al munoz designed the evaluation approach for the first phase of the key initiatives centers of excellence...it was highly talent-focused and made excellent use of orals...was also very rapid...one of the best procurements i've seen run

[pcbock]

We have the same concerns regarding the links to client GitHub repos, and I agree with @kaitlin that a smaller scope coding challenge would be an effective way to evaluate.

[JQuaggles]

Agreed with the above suggestions. As @kaitlin mentioned, a small scope coding challenge (scaled to make sense for the size of the project vs a bigger BPA) would be ideal.

Would be interested to explore @cscairns recommendation if he is open to providing more detail/a link to further collaborate on.

[JessicaMarine]

Thank you for the additional comments and suggestions. The final RFQ will likely include changes that account for this feedback.

We are considering changing some of the more narrow constraints around what is considered similar experience - as noted by @greggersh in the fourth paragraph of the initial issue, it may be difficult for any vendor to find “exact examples” of projects with this staffing profile and duration. Likewise, that aspect of similar experience may not be very important to demonstrating good work.

We are also considering the point made by @kaitlin about Git repos not being a good source of evidence for user-centered design practices. The final RFQ may ask for different artifacts for the design-related aspects of vendors’ similar experience.

The scope of work for the project cited by @cscairns was for management consulting, which is very different than the design/delivery teams contemplated in this RFQ.

Thanks again for all of the feedback!

[greggersh]

@JessicaMarine thanks for the opportunity to provide feedback!

[cscairns]

@JessicaMarine: thanks for the response...understood that the cited evaluation approach was for management consulting, but the fundamental principle can be applied to other types of procurements as well, such as this one, as i've written about elsewhere...either way you go, i'm excited to see what you come up