Quality Assurance

[RachelMuston]

Need support and assistance from IT on:

• how to provide Quality Assurance on the code submitted by web developers.
• alternative hosting arrangements for the website (GC DevEx is currently hosted on a 3rd party server used by the GCTools team at TBS).

[RachelMuston]

From the BCDevExchange folks: The model we are using today leaves the burden of responsibility for quality assurance, security, etc with the government – specifically there is a defined role for code acceptance that is the responsibility of the program area that has conducted the open development transactions.

The developers that are contributing code are doing so under open source license models and they therefore make no warranties or claims about their work from a contracting perspective.

In practice, what we are doing is making sure the government team is constructed so it can competently conduct an open development activity that includes rapid iterations, small code increments, automated tool chains with builtins for security scanning etc.

We regularly have to get creative to assist the government teams with their duties to ensure that the code received is fit for purpose, that it works, that it is good enough quality, and that it is secure/safe to run. Our strategy on this has been to get a lift simply by virtue of the open transparency of the entire development cycle knowing that the people involved are professionals and are motivated to build and uphold their track records and reputations for doing good work.

It’s a work in progress – and our trajectory on this is positive. It is leading us in the direction of more openness, much stronger tool chains, a higher level of automatization, and an increased level of skill and knowledge on the government side of the equation.

[RachelMuston]

From: https://gdstechnology.blog.gov.uk/2017/10/10/open-source-security-meetup-7-things-we-learned-from-the-cross-government-event/

2 new pieces of guidance that have been released by GDS. These detail the subsets of the code departments should keep closed and how to code in the open securely:

1. ‘When code should be open or closed’ https://www.gov.uk/government/publications/open-source-guidance/when-code-should-be-open-or-closed
2. ‘Security considerations when coding in the open’. https://www.gov.uk/government/publications/open-source-guidance/security-considerations-when-coding-in-the-open

[RobJohnston]

I've been thinking about quality assurance since I first heard of GCDevExchange, and I think it can easily be solved with a change in what is being evaluated.

The user documentation in the wiki states,

"We want to make it easier to contribute to government digital products and get paid - and do it in a way that allows you to focus on writing code rather than contract paperwork".

Well, that fails because developers have to now be experts in proposal writing... a type of contract paperwork.

The time spent by GC employees on writing proposals and then scoring them would be better spent on defining the problem well and then evaluating the solutions that are submitted. Let multiple solutions be submitted by a deadline and evaluate them all... you never know who could do it the best. Otherwise, the proposal evaluation criteria is going to quickly morph into a way to arbitrarily limit the responses from developers in order to limit the amount of work that the GC employees have to do.

Require unit tests, installation instructions, state that a standard coding convention must be followed, etc.

So in summary, evaluate the final solutions and not the responses to proposals to get the best quality.

[RachelMuston]

Hi Rob, You raise a good point ...while we aren't expecting the sort of fancy proposals big companies submit for larger gigs, we are asking developers to do some writing which is perhaps not the skill set we want to be assessing.

I can tell you that some of the proposals we received for the first opportunity did show their solution to some extent (not just describe in words how they would do it). But I think there is a balance to be found...is it fair to ask all developers to do the work even though only one can be selected? The idea behind proposals was to only require a small amount of time so that if they weren't selected, not too much is lost.

I certainly wouldn't want GC teams to be writing opportunities in a way that would limit responses from developers. If you think there is anything in particular that would cause that to happen I would be very interested to know what that would be so we can make sure we don't do that.

Re: QA testing....we are currently looking at whether we should require all code procured through GCDev Exchange to pass an automated test tool (like Code Climate). I've put some feelers out to the BCDev Exchange folks to learn more about what they are doing and we are discussing internally with some security folks. Any thoughts?

[RobJohnston]

My thoughts came from previous experience. For example, on a competitive coding site that I've been lurking on for the last few months, they solve the problem of fairness in pay by giving a smaller dollar amount to the second-, and maybe third-, forth- and fifth-placed finishers. And anybody who submits a solution gets to see the winning solution, thereby learning what it takes to win (and how their solution didn't measure-up), so it's not a total loss.

The possibility of limiting responses comes from how I see GoC contracts written. They include things such as asking for a secret clearance to work on a public website. Or a phase such as "5 years experience in the last 7 years writing SQL queries"... isn't 4 years enough to learn SQL? SQL didn't change 7 years ago, so why is experience 8 years ago worthless? I've also seen an ask for more years of experience using something than that thing has existed. Or my favourite from the past for a web developer position, "2 years experience using the IP protocol". Why do I do with that? It makes no sense and has no answer, so I self-exclude. Another is asking for experience using, say for example, AkelPad. If you don't have that, you can't apply, but it's nothing more that an alternative to NotePad. In these cases, everybody knows that they are targeting the opportunity to a one specific person (the incumbent).

I'm not familiar with Code Climate. But at the very lease, require unit tests where possible and specify the coding standards to follow. And get an installation script or instructions as part of it. Also, going back to the competitive coding website that I mentioned, they usually state up-front that they may require "final fixes" before the payout can happen.

[RachelMuston]

Hi @RobJohnston, Could you share the link to the competitive coding site you mention? I'd be interested to check that out.

Agree that those timeframe requirements are often seen in GC contracting. With GCDevEx we are trying to do things differently. We want to cast the net wide. This is why those sorts of requirements did not end up in the first opportunity (instead we asked simply for some examples of previous work - no timelines).

GCDevEx opportunities are processed (for now) as sole source contracts. This means if a GC team already has someone in mind to do a piece of work, there is very little value in them posting it on GCDevEx. They'll still have to do the sole source paperwork AND essentially run a competitive process (via GCDevEx). This extra work only adds value if you do not know a developer with the skill set you require or you are honestly open to hiring anyone based on the strength of their proposal.

[RobJohnston]

@RachelMuston : The link is topcoder.com.

Edit: This is probably a better link to see the current challenges: https://www.topcoder.com/challenges/