What happened at CMSgov/healthcare.gov?

[troyswanson]

Seems like some craziness happened here (CMSgov/healthcare.gov#12). Any news on this?

[benbalter]

Short answer: a merge, a pull, and if I had to guess deleting the .git folder, possibly the repository itself, followed by a force push.[^1] In detail? Read on.

First, some framing: there's healthcare.gov which is an entirely static Jekyll site developed by a small, forward thinking startup and served by Akamai. No doubt there are a few minor optimizations you could make, but in all practicality, you could throw most of the internet at it, as is, and it wouldn't even flinch. That's the part that's been posted on GitHub. At least as far as we know.

Then there's a hole slew of traditional government contractors that built the other components of the Rube Goldberg machine that operationalizes the Affordable Care Act. @philipashlock's got a thoroughly researched writeup on some of the technical considerations that have been floating around. TL;DR; Java, XML, and "enterprise" solutions won out.

So, to your question, why's this great thing that got early accolades sentenced to a Groundhog Day like fate of endless first commits?

In a word: culture. There's two worlds. There's the world where innovative technology thrives. It's open. It's collaborative, it's lock free. As @tmcw put it, it's a world that understands that open source has already won. It values things like interoperability, and elegance, and well, finding the leanest possible solution.

Then there's another world. It's top-down. It's heavy. It's managerial. It's slow to move. It thinks change is toxic. Self preservation is its goal. It values things like outward appearance, and security, and artificial fairness[^2]. :moneybag: is equated with quality. That's the world government has traditionally lived in, and until recently, the two had rarely met.

@cjoh did a great job of explaining just how that culture perpetuates itself, which the system would call a feature, not a bug. Heck, even this week IBM, in arguing it's protest a bid that had been decided in Amazon's favor, cited its decades long history of government contracts as a reason why it should get yet another one.

So you've got this self-perpetuating, closed system that's out of pace with the rest of the internet. You can bring in forward thinking teams like @developmentseed to inject some of the things the cool kids are doing out in Silicon Valley, but at the end of the day even outside innovators need to eventually pass the baton.

This is a long way to say the technology's the easy part. The culture within the government often doesn't appreciate many of the lessons the internet learned a decade or two ago, or if they do, they're generally placed at a lower priority below things like ease of contract administration or jumping through anachronistic administrative hoops.

When you live within the iron triangle, and features and time are set by law, there's only one thing to give. You can either spend the time to transition from centralized management and version control to a radically different, distributed, open system, or you can spend that time trying to overcome all those internal constraints and actually :ship: the thing you've set out to :ship:.

So both hard and soft, bureaucratic and cultural handcuffs. Yes, a big part of it is legal, administrative, or political, but a part of it is also our fault. Part of it is us not being better ambassadors to open source. For allowing the broader community to be seen as liability, not an asset. For not being more vocal for the shared cause. It's a big step to develop out in the open, to live in a glass house, and an even bigger step considering where many government agencies are coming from. Looking through some of the issues, we could do a better job of making them feel a bit more welcome. GitHub should not become the digital equivalent of the West Wing's Big Block of Cheese Day.

I wouldn't go so far as to say openturfing, and there's no reason to suspect anything malicious. The staff at @cmsgov should be applauded for taking the first baby step[^3]. I just hope that we as a community will have the opportunity to help them take the next as well, to help them expose process, not just code.

[^1]: I wasn't directly involved, and this is entirely based on what's been published elsewhere. [^2]: Would anyone argue, despite it's lack of centralized constraints that open source is somehow systematically unfair? [^3]: The source code was published, a community-contributed pull request was accepted, the repository's been renamed to drop "open-source-release" from its name.

/cc @dhcole

[parkr]

[troyswanson]

@benbalter Wow, I wasn't expecting this lengthy of a response. Thanks so much for taking the time.

You bring up a lot of interesting points involving open source ideals vs. code in a corporate, government, or otherwise overly bureaucratic culture. I wonder if there are any tried and true strategies for either convincing the powers that be to think about projects differently, or for folks working on projects to sidle their work into the open source world without necessarily gaining "approval".

Once again, thanks for this fantastic response. I really appreciate it.

[migurski]

Gov.UK has demonstrated that things have to get really bad before anyone is motivated to change them.

I’m not comfortable with some of the “two cultures” language here, by the way. It feels anti-cooperative, and unnecessarily valorizes a startup innovation culture that’s able to move fast because it allows itself to ignore constituencies, er, “pivot”. The GDS didn’t change UK digital culture from the outside, it was a debate at the top that resulted in strong outcomes. “Government is just a word for what we decide to do together,” and Healthcare.gov is already a move in the right direction with its use of tools like Jekyll and small service firms like Development Seed. I think you could say the same thing about the Obama Tech Team by the way. It hired from outside the traditional campaign pool of developers, but was very much of the campaign, a strategic decision made at the top that still had to prepare for the reality of a highly-trafficked, very visible and complex site that needed to work flawlessly on Day One.

[dhcole]

Thanks for compiling this research and your thoughts @benbalter. Also worth considering how much political pressure was brought to bear on this launch: both from people needed it to success and fast and those opposed to its very idea. That's hardly the ideal environment to try to build a big software system, and compounds all the previously mentioned challenges.

Having seen both sides of the government IT process, I agree that there are structural impediments to innovation. I think this report from the Wilson Center has some actionable ideas on how to break through them, as least at smaller scale. I hope it starts getting the attention it deserves.

[migurski]

Novelty, fearlessness, and scope discipline are the key ideas in that report. It's excellent reading, and I wonder whether it would have been smarter to assemble the Healthcare.gov project as an in-house, short-term campaign-style team rather than contract it out. It has at least the same visibility as OFA, and could potentially have attracted similar developers with a taste for wicked problems and the instincts to match. Dave, I'm curious what you think about the developer pool available in DC vs towns like SF or Chicago, and what impact that may have on outcomes. Perhaps there is a PIF model that could be applied here.

[benbalter]

@migurski lots of great stuff in there. To clarify, the question wasn't about healthcare.gov as a website, or how the registration process performed at launch, but rather why subsequent to its release the source code's revision history had been wiped of contributors, which I'd argue stems from an incongruence between the culture that created it, and the culture in which it now finds itself. It's open versus closed, not startup versus established player.

[NoahKunin]

:+1: with everyone else, but one edit to your TLDR re: the exchange.

Subcontractors, subcontractors, subcontractors.

From personal experience in government tech, even if you have a semi-competent prime contractor in "Expertise Area 1", with each additional subcontractor they have to grab in order to produce work in "Expertise Area (n+1)" you will see a radical decrease in performance. I speak from ample personal experience.

For many reasons, the procurement process being one of them, there are now many firms who are only prime contractor "shells" that then subcontract out 95% of the work (or all the non-project management work) to subcontractors.

[dhcole]

Dave, I'm curious what you think about the developer pool available in DC vs towns like SF or Chicago, and what impact that may have on outcomes. Perhaps there is a PIF model that could be applied here.

The PIF problem strikes at the traditional hiring process, which is another really difficult policy issue on par with procurement. Hiring staff can take months and who you end up hiring may actually be out of your control. PIFs allow for fast-tracking specially skilled employees, at the tradeoff of a short tenure. To have a PIF see this through, they'd need at least 4 terms as major development took place over at least 2 years.

Recruiting in DC is really hard. I ran into this when looking for PHP developers back in the dark ages of 4 years ago. Having to do it through a contractor is more difficult. Chances are people will move from SF or NYC to DC for a job as a developer for the "Obama Administration" but not so much for "General Defense Contractor, Inc". Then there's security clearances, FBI background checks, random drug tests.

But there's no reason to think that's how it needs to be done. For instance, our company thrives in DC. We hire a lot of people from outside of DC. Many relocate, and some work remotely. This works well if you set up the right culture and async communication tools. The White House is not allowed to use a chat service because it's been declared that chat messages would qualify as official communication and need to be archived for all time and eventually released publicly, possibly subject to Congressional subpoena. Talk about a chilling effect. There also was no approved web-based video conferencing or screen sharing.

How would you work like that? While many tech companies have used tech to overcome the traditional obstacles to building and scaling a business, like recruiting and location, the government is knotted in regulations that make this very difficult.

I fundamentally agree that building internal teams makes the most sense. But practically, I have not seen it done for these reasons. There's a philosophical question about the role of government departments building vs funding the building of technology, but either way, they need to be able to get the systems necessary to execute their missions, and both tracks have blockers in the hiring and procurement processes. Still some things get done. My outlook has usually been that government tech is on the right track, but will always lag about 5 years behind without some more aggressive shake-ups. It took a whole year for us to bring in an open source CMS and another 6 months to start releasing our own code. Other agencies immediately tried emulating, but it took several months to years for them to get it done. Some still pay way to much for way to little with proprietary systems. The trajectory is right, but it just takes a while. We can speed it up, but it requires some policy changes, as the Wilson Center report highlights.

I appreciate your calling out our work on healthcare.gov as a step in the right direction. We are very proud to be a part of this. That our work stands up to the traffic and performs well is a testament to and made possible by all of the incredible work that people in the open source community have contributed through projects like Jekyll, node.js, backbone.js, bootstrap, and dozens more, as well as the agile development process we use to maintain distributed open source projects.

We were brought on too late to be involved in some of the key decisions about the exchange application's architecture, but we put a ton of time into delivering comprehensive strategy documents on how to build agile, open tech in government. I hope these will open the door for better methods and tools for the next project and be put to good use as this one is revisited.

[waltisfrozen]

"the repository's been renamed to drop "open-source-release" from its name"

Points for honesty, I suppose, but what value does the healthcare.gov repo provide? It's one thing if they want to turn their back on the community and change their minds about being open source, but the content in their repo doesn't even represent the current state of the site. For example, compare robots.txt :

https://github.com/CMSgov/healthcare.gov/blob/master/robots.txt https://www.healthcare.gov/robots.txt

If they don't want to accept pull requests from the community going forward, that's their prerogative. But using Github to host what appears to be a snapshot of the site as it was being developed on July 1st isn't really useful. What are Github users expected to do with this code? Fork it and log issues that have already been fixed in the past three months?

I've seen plenty of articles about hc.gov's technical woes, but I haven't seen any journalists cover this angle. Go to the site's developer page and there are plenty of glaring misstatements :

https://www.healthcare.gov/developers/ "EVERYTHING we do will be published on GitHub: From revising a glossary entry to updating a link in the footer, we're making all of our changes transparent and available to the public."

I think all of your points about how the culture in DC is hard to change are well-taken, but in terms of government transparency and open source, is anyone even listening? If they aren't maintaining their Github repo, then can we honestly expect them to be reading through the issues (much less, see the feedback in this thread)?

[troyswanson]

@waltisfrozen I suspect that this particular issue isn't very important to anyone in Washington, but that doesn't mean anyone in the open source community should feel apathetic about the situation. Exactly the opposite, actually - we should get fired up and figure out a way to effectively demand a higher amount of accountability from our government, especially when it comes to software development and contract procurement. From what I can tell in just my few days of learning about this whole fiasco, there are already a lot of people working on this, but I haven't found a good way for a random dude like myself to get involved.

Something like a million developer march would be pretty sweet to participate in.

[migurski]

@dhcole it’s my understanding that the definition of “short term” used by the PIF program is actually anything less than four years (@feomike is that true?). The fellowship is brief, but the mechanism to make it work developed by Todd Park’s office is good for longer runs. I would love to see per-project task forces outside DC, for example in one of the many fine federal buildings available around the country. Giving your life away to Uncle Sam is not great, but doing a tour of duty for the benefit of a concrete outcome is pretty attractive. Brett Goldstein in Chicago has banged on this particular drum quite a bit. I’m sure many would jump at the chance to built a Healthcare.gov much like you guys did if it was framed as a lean, limited-time, impeccably-scoped project.

VA medical records, anyone?

[NoahKunin]

@dhcole is there any chance the strategy documents your team put together could see wider release, at minimum, in a scrubbed of HHS specific stuff format?

[dhcole]

@NoahKunin not sure that HHS will release them, but I can see about getting out the most interesting parts as blog posts. We've covered some basic topics w/r to building static sites that scale here (the part of the site that didn't flinch during the rollout):

• http://developmentseed.org/blog/2012/07/27/build-cms-free-websites/ (manifesto of sorts)
• http://developmentseed.org/blog/2013/06/25/healthcare-launches-in-the-open/ (lots of links to other posts)

The more interesting content for the exchange site deals with stuff like how to develop in agile sprints, how to ship regularly and user-test in realtime, and how to use github for on-going code review and collaboration, to @benbalter's point about needing cultural change more than tech. Let me see about rewriting those as blog posts (though, much will seem relatively un-groundbreaking to those who already get it).

[dhcole]

Also, we were scheduled to do a web conference, but it got postponed due to the government shutdown. cc @gbinal

[NoahKunin]

The repo has now been taken down in its entirety.

https://github.com/CMSgov/

I'm hearing a couple of things that indicate why this happened, but if anyone can provide additional context, that would be helpful.

[STRML]

I haven't heard anything from anyone as to why this went down. Fortunately there are backups.

https://github.com/blencorp/HealthCare.gov-Open-Source-Release contains some minimal commit history of the jekyll site.

https://github.com/STRML/Healthcare.gov-Marketplace is a fork I'm maintaining of the marketplace code to house community-sourced fixes & issue tracking.

[feomike]

i apologize for the delay here, i took a digital break. @migurski is basically right. the term you can sign up for as a PIF is 1 year, but the clause that is used to do that is something called 'Professional Exchange' whereby the Fed can hire people from the private sector for limited terms without the normal civil service process. these limited terms are predominantly 24 months, with a 1 time extension of another 24 months. so yup, you can get to 4 years. PIF doesn't work that way, as it wants to cycle lots of people in for a tour and get them to do things fast. its a good approach.

[Alex2020]

Good commentary from all parties.

To continue this momentum, consider listening to the upcoming congressional hearing "PPACA Implementation Failures: Didn’t Know or Didn’t Disclose?" which will have testimony from a few healthcare.gov contractors.

Scheduled for Thursday, October 24, 2013 - 9:00am. There should be a live stream here

To give you some context, I think Cheryl Campbell Senior Vice President CGI, has a good testimony about the many moving parts to the project; paraphrased as:

The backend of healthcare.gov must interact in real time with: 1.) systems developed by other contractors - including in the area of enterprise identity management. (see gov contractor Equifax's testimony)

2.) existing federal agency databases from the Internal Revenue Service, the Social Security Administration, and the Department of Homeland Security.

3.) databases from more than 170 insurance carriers qualified to do business in the 36 states. (see gov contractor Optum/QSSI's testimony in regard to their "Data Services Hub" component.)

[troyswanson]

@Alex2020 Thanks so much for posting these links.