Federally funded state and local software

(I’m Robin Carnahan, and I head the State and Local Practice at 18F, an office in the U.S. General Services Administration that provides in-house digital services consulting for the federal government. I’m commenting on behalf of 18F; we’re an open source team and happy to share our thoughts and experiences. This comment represents only the views of 18F, not necessarily those of the GSA or its Chief Information Officer.)

According to the Congressional Budget Office, the federal government spends over $600 billion annually in federal grants to state and local governments. As 18F learned working with the Department of Health and Human Services, the federal government often is responsible for paying for significant portions of state and local IT spending, including the development of applications and platforms for implementation of federal programs in areas such as healthcare, education, transportation, and law enforcement.

Extending OMB’s source code policy to include federally funded software projects developed or procured by state and local government is a logical extension of the federal policy, and would:

greatly improve digital services for more people;
save more money for taxpayers; and
reduce time, costs and risks for government.

State and local governments often face even greater capacity constraints and technology needs than the federal government. Extending the proposed source code policy to code developed on behalf of state and local government using federal money would prevent them from having to reinvent the wheel in every state and community, and open the door to better collaboration across state and local governments. In addition, as federal programs like Medicaid are transforming and modernizing, there is an opportunity to incentivize and encourage use of open source. At a minimum, open source software should be considered a default for federal grantees and federal agencies should be required to justify when open source is not used. A default to open source would minimize the federal government’s risk of paying for the same solution 50 or more times over.

An open source policy would broadly enable states and local governments to collaborate on shared solutions. 18F has directly witnessed some of this dynamic through its work with the GSA’s Digital Analytics Program on analytics.usa.gov. The code behind this project has been used, in some form, by Philadelphia, Boulder, and the State of Tennessee, and teams in the New York and Massachusetts state governments are in the process of adopting it.

Each of these teams has made, or is making, modifications to support their use case. 18F’s public domain dedication on the underlying code has dramatically lowered the cost, time and resources needed for state and local agencies to deploy these open source code solutions. By removing legal uncertainty and the need to ask for permission, full open source release empowers even small, resource constrained state and local government agencies to move fast and make the most of their limited resources while providing better digital services to the public.

Yes, yes, a thousand times yes.

Another reason why this is important is because it will enable inter-state innovation facilitated by code reuse. If Hawaii does something clever, by using open source it becomes possible for Illinois to improve upon that and, in turn, for Florida to use a bit of that code to improve something that might have seemed entirely unrelated.

I have seen, firsthand, the powerful and important work that is being done in open source by seriously resource-constrained state agencies through the nation. Handing them a tool chest of relevant open source government software would facilitate a combination of collaboration and competition that would be a huge win for taxpayers and the public generally.

it will enable inter-state innovation facilitated by code reuse. @waldoj

That is a big one. Enabling peer review and reuse of source code will allow federal, state, and local govt employees to build upon peer best practices and avoid reinventing the wheel. This is especially important for projects that depend on secure data handling from the client-side to internal storage/processing. Agencies should be able to reuse well-documented code blocks instead of introducing risk into their own projects by haphazardly writing new routines.

Glenn Batuyong

Multimedia Designer, Sempra Energy
Former Senior Web Developer, Port of San Diego (@portofsandiego)

This is SO important. I've worked in State and Local governments and we want to work with other local governments to share ideas, processes and technology. The way the system works now when it comes to sharing ideas makes it incredibly difficult to find the right person with the information (often just one or two employees - also very bad) and it leads to a huge loss of time and productivity.

I don't think I'm exaggerating when I say that this could lead to billions of dollars in savings for State and Local governments, increased efficiency and innovation, and better communication among small governments.

I couldn't agree more.

Naturally, there are huge overlaps in the services local governments provide and the tools they need to serve better. At the same time, it's important when selling progressive ideas to have the "social proof" of other governments.

Given both of these things, it should be straightforward to see how open sourced solutions would spread quickly, saving time and money while those solutions would become better for it.

At a minimum, open source software should be considered a default for federal grantees and federal agencies should be required to justify when open source is not used.

Failing to do this virtually ensures that the same players will continue profit from reselling canned solutions to an all but captive audience.

A default to open source would minimize the federal government’s risk of paying for the same solution 50 or more times over.

As a former software developer of a Medicaid Management Information System (MMIS) product this is both very true and very false. Every MMIS system is different. Having worked on multiple state accounts, each one had its own peculiarities and solutions to similar problems (eg. ICD10 was usually implemented in a particular way, but often there were differences in the data model which cascade down into extracts/reporting/etc.). In general, there is a large potential for cross-state code reuse.

However, very often, systems address problems in very different ways because the business rules and workflows are different. The majority of developer time spent in the implementation phase of every MMIS system is requirements gathering to enumerate those state-to-state differences and testing them. Each state has different coverage programs and criteria for provider and recipient eligibility. States often use different COTS packages that need to be integrated differently; not everyone uses k2 bpm or oracle bpel for workflow; many states are not even on the same OS platform (linux, solaris, hpux, z/os).

These kinds of systems (state MMIS) are not drop-in compatible with each other and it would take a huge amount of money (and cross-state compatible legislation) to get them anywhere close, though MITA3 is huge step toward that goal. Just look at the number of companies that have failed to ever implement their own system or continue operating the legacy system after winning a contract.

On the other hand, cross-state licensing would be a huge boon. There are often a lot of hoops to jump through to get the permission to copy features from one system to another because the changes were paid for by each state. Guaranteeing code can be copied will help ensure that the systems are more regular between states both inside a single vendor and across all vendors.

In the specific case of Medicaid, OSS is not a silver bullet. While it should reduce the initial system cost and hopefully the ongoing maintenance cost from federally (CMS) mandated changes, because of the way medicaid is structured it still takes a large amount of resources to adapt and maintain each system state-to-state.

The following is a joint statement on behalf of the California Health and Human Services Agency and California Department of Technology regarding the proposed Federal Open Source Code Policy recently issued by the White House:

"California endorses the Federal government’s position on the use of open source software and feels strongly that it is an excellent investment as a public good.

"California expects to meet its long and medium-term technology objectives through the leveraging of open source software. Open source software will reduce our costs, increase productivity, improve interoperability between our systems and provide a more innovative environment on behalf of our citizenry. This approach will increase government efficiency; ensuring that our applications are more responsive to the needs of the people we serve.

"Cross–jurisdictional collaboration will occur as a result of a more profound and open sharing of technology solutions. Open source is at the heart of democratic ideals, and as a state, we are bound to uphold those principles. Our collective approach to utilizing open source technology will foster development and progress, that can’t be achieved by any other means beyond an open and transparent engagement with the people we serve. This can be facilitated in part by placing our open source technology in the public domain, so that other public entities can repurpose the code and reduce their overall expenses. We expect to provide better inter-agency cooperation and sharing through contributions to common open source projects.

"Implementing our ultimate goal utilizing open source technology will improve our services to the benefit of the public by providing quality goods and services to the people we serve in a responsive, sustainable, and efficient manner."

I echo the support of applying this policy to federally funded state and local software to the extent possible, though given the limited amount of mandate authority OMB has in this regard, I'd like to add 2 points:

Additional language should be added to the Implementation guidance of this policy that requires agencies explicitly describe their approaches to achieving this end in their Agency Policy and report all relevant state and local software projects to the relevant channels listed as Accountability Mechanisms. This will help measure the impact of this policy on federally funded state and local software. Research (by OMB or respective agencies) on what projects have been made open and which have not as a result of the Increasing Access to the Results of Federally Funded Scientific Research memo would help inform policy language needed and helpful metrics to measure to achieve this goal.
I encourage all folks passionate about this in the local space @waldoj & @CaliforniaDepartmentOfTechnology, etc to also additionally advocate for complimentary and leading edge local open source legislation and policies.

On the other hand, cross-state licensing would be a huge boon. There are often a lot of hoops to jump through to get the permission to copy features from one system to another because the changes were paid for by each state. Guaranteeing code can be copied will help ensure that the systems are more regular between states both inside a single vendor and across all vendors.

This is very true. It probably applies to almost every fed/state project. Posting reusable code for comparable modernization projects would be a big step forward. It would be hugely beneficial to states that had not yet started their projects. I think a lot of these state projects are spec'ed and built as though they are the only instance of a particular business process.

Some states are using consortia to gain efficiencies, but moving to open being the default is the way to really unlock value. Dvelopers will write in a more modular way. Having an open source repository that both developers and CUSTOMERS can see as a resource base will be great!

As a civic technologist that has worked on government software at the local level for many years -- both inside and outside government -- I strongly support the need to include the implications, opportunities, and needs of state and local governments.

While as Code for America, one of the nation's leading civic technology organization's we heavily emphasized open source development and software reuse. Open source tools that were built for one city were quickly -- and cheaply -- redeployed in another, and then another; each time gaining features, users, and -- importantly -- a broader community of developers contributing to better software. Given the technical and financial constraints of state and local governments, this kind of collaboration is essential.

Having now worked inside two different governments -- first as Chief Data Officer of Los Angeles, and then now as Chief Innovation Officer of Sacramento -- I can attest to the utility of government-focused open source code. In fact, just this past weekend, I was able to design, build, and deploy a new website for the City of Sacramento in a matter of hours thanks to the open source components built by USDS and 18F. The federal government with its growing and talented technology teams can be and indeed is a force multiplier for state and local innovation. With this policy, per Robin's suggestions, the leadership from the federal government can extend beyond standards in technology and design to policy. This means that even as tech evolves, we can institutionalize a commitment to openness, collaboration, and cooperation, leading to better governance at all levels of government.

Abhi Nemani

Chief Innovation Officer, City of Sacramento
fmr Chief Data Officer, City of Los Angeles
fmr Co-Executive Director, Code for America
Board Member, OpenGov Foundation

As someone building digital services for a mid-sized city, I heartily agree with @abhinemani: open source projects allow us to deliver services that would be out of reach otherwise. Not only is the software itself important, but all of the side benefits:

It creates a market for talented open source development firms we can work with and learn (steal?) from. The world is better for the contributions of companies like datamade, Development Seed, Azavea, and OpenPlans (RIP!). Note all of the recent activity in their GitHub repositories.
It pushes governments to find open source expertise that often lies in smaller companies usually filtered out by procurement.
Tools and platforms emerge around deploying and operating these tools. The software stack becomes more commoditized. Cities and states can use platforms as a service and other modern cloud tools. After lifting the burden of maintaining snowflake datacenters they can narrow in on the complexities of service delivery.

Erik Schwartz, Digital Services Architect (speaking personally) City of Lexington, KY former Code for America fellow

For our cities, it's worth noting that the dream of Civic Commons (#238) is still alive in NYC with Councilmember @benkallos's work http://benkallos.com/event/contracts-committee-hearing-free-and-open-source-act-civic-commons

For federally funded work with states, let's look at the Affordable Care Act:

A few years ago, I started to investigate what seemed to have been an ambitious attempt to use an open source strategy for the federally funded work on the state health exchanges. From the outside, it appears that the open source strategy failed - likely for the reasons @kfogel explains here and here.

I've copied the full post below, but the original with formatting is here.

The Exchanges: Open Source Collaboration Among States and Federal Government?

screen-shot-2013-10-22-at-12 40 34-am1 _A slide from one of the Early Innovator Learning Collaborative webinars_

The last post looked at what may have accounted for the architectural instability of the healthcare.gov infrastructure and tried to learn from it. Recently HHS provided a brief overview of the problems and their efforts to resolve them, but clearly most questions will remain unanswered unless a full postmortem is ever made available. One issue that has received more notice lately is the role of open source in building out the exchanges. Originally this got attention with Alex Howard’s piece in the Atlantic from last June which emphasized the open source approach Development Seed had taken to develop the frontend. With this context many thought of the whole project as open source and were dismayed to discover otherwise. More recently there have been renewed questions about the open source nature of the project because of the removal of the Github repository that housed the code originally created by Development Seed.

It turns out there was a lot of open source thinking from the earliest days of building the exchanges. In fact, I was part of a team asked to help ensure the infrastructure was developed following open source best practices.

Idealistic Youth

In 2010 I helped co-found a project called Civic Commons as a partnership between OpenPlans and Code for America. Civic Commons was meant to provide human capacity and technical resources to help governments collaborate on technology, particularly through the release and reuse of open source software. Civic Commons itself was a collaboration among several non-profits, but initially it was also a close partnership with government. The project was supported by the District of Columbia with then CTO, Bryan Sivak, as one of the city projects in Code for America’s inaugural year. Unfortunately DC’s involvement didn’t survive through the District’s transition to a new administration, but Civic Commons continued on with the Code for America fellows (Jeremy Canfield, Michelle Koeth, Michael Bernstein) and both organizations working together.

In early 2011 the Code for America fellows met with then US CTO Aneesh Chopra and he asked them for assistance on an exciting new project.

“I need your help,” he began before sharing some recent news. He said it had just been announced earlier in the week that, “Seven states and one multi-state consortia will receive in aggregate $250M to build out insurance exchanges – they are required to align to principles, feed into verification hub, and engage in health philanthropy community.” Then he explained how we fit in, “Here is the specific ask, because I am big on open collaboration, I made a requirement that each awardee would have to join an open collaborative, a civic commons, have to join a commons for the reusability and sharing of all IP assets.” Aneesh went on to explain that it didn’t seem appropriate for the federal government to play this kind of role among the states and that he was really looking for an independent entity to help out. He said he was, “Looking for a commons that can act as the convening and tech support arm to the seven awardees – before the states go off and hire [big government contractor] who will set things up to minimize sharing, we want someone to set the rules of the road.”

At the time Civic Commons was just getting started and even though the prospect of a large and important project like this was very attractive it also seemed like it would consume all of our resources. After discussing concerns about our capacity and uncertainty of available funding to help, we decided against being involved. Much of the early work with Civic Commons was focused on more manageable projects among cities, but we did also include open source work at the federal level like the IT Dashboard. I do have a slight sense of regret that we couldn’t be more involved with open sourcing the exchanges, but it seems better that we learned what we learned from smaller experiments.

Lessons Learned

Karl Fogel was the primary shepherd of the open source process with governments at Civic Commons and one of his most notable blog posts detailed how difficult or even futile it was to try to do open source as an afterthought. If you’re not doing open source from the beginning then you’re probably not doing open source. Without the kind of organizational steward Aneesh was looking for, I fear those states might not have ever truly engaged in open source development as originally intended.

The other difficult lesson we learned through experiments getting cities and other governments excited about open source is that there tends to be much more motivation to release than to re-use. Some of this seemed like it may have been motivated by a PR driven sense that giving away your hard work looks good, but reusing others’ work looks lazy. Perhaps we need to do more to praise government when they are smart enough to not reinvent the wheel or pay for it over and over again.

The most encouraging lesson I learned during our time with Civic Commons is that there are some effective models for open source collaboration that involve very little direct coordination. The main model I’m referring to is one based around common standards and modular components. At OpenPlans we saw this with the success of open source projects based on the GTFS transit data standard like OpenTripPlanner or ones based on standardized protocols for real time bus tracking like OneBusAway. I’ve also watched this closely with the open source ecosystem that has developed around the Open311 standard with both open source CRMs and separate client apps like Android and iOS mobile apps that can be shared interchangeably. The full stack of open source tools from the City of Bloomington and the work Code for America has done with cities like Chicago have been great models that demonstrate the opportunities for software reuse when governments have asynchronously agreed on shared requirements by implementing a common standard. The apps developed by Bloomington are now even being used by cities in other countries.

The IT infrastructure for the exchanges was clearly based around common data standards, so you would hope the same opportunity would exist there.

An Open Exchange

The effort Aneesh had referred to did still continue without us and only now have I started to learn about it in detail. The $250 million he had described was more precisely $241 million in federal funding through the Early Innovator grants. These grants were awarded to Kansas, Maryland, Oklahoma, Oregon, New York, Wisconsin, and the University of Massachusetts Medical School representing the New England States Collaborative for Insurance Exchange Systems – a consortium among Connecticut, Maine, Massachusetts, Rhode Island, and Vermont. The grant was in fact as lofty as Aneesh had described to the fellows. The “Funding Opportunity Description” section of the grant states:

The Exchange IT system components (e.g., software, data models, etc.) developed by the awardees under this Cooperative Agreement will be made available to any State (including the District of Columbia) or eligible territory for incorporation into its Exchange. States that are not awarded a Cooperative Agreement through this FOA can also reap early benefits from this process by reusing valuable intellectual property (IP) and other assets capable of lowering Exchange implementation costs with those States awarded a Cooperative Agreement. Specifically, States can share approaches, system components, and other elements to achieve the goal of leveraging the models produced by Early Innovators.

The expected benefits of the Cooperative Agreements would include:

Lower acquisition costs through volume purchasing agreements.

Lower costs through partially shared or leveraged implementations. Organizations will be able to reuse the appropriate residuals and knowledge base from previous implementations.

Improved implementation schedules, increased quality and reduced risks through reuse, peer collaboration and leveraging “lessons learned” across organizational boundaries.

Lower support costs through shared services and reusable non-proprietary add-ons such as standards-based interfaces, management dashboards, and the like.

Improved capacity for program evaluation using data generated by Exchange IT systems.

The grant wasn’t totally firm about open source, but it was still pretty clear. The section titled “Office of Consumer Information and Insurance Oversight: Intellectual Property” included the following:

The system design and software would be developed in a manner very similar to an open source model.

State grantees under this cooperative agreement shall not enter in to any contracts supporting the Exchange systems where Federal grant funds are used for the acquisition or purchase of software licenses and ownership of the licenses are not held or retained by either the State or Federal government.

It’s not totally clear what came of this. The last evidence I’ve seen of the work that came out of these grants is from a Powerpoint deck from August 2012. The following month a report was published by the National Academy of Social Insurance that provided some analysis of the effort. The part about code reuse (referred to as Tier 2) is not encouraging.

Tier 2: Sharing IT code, libraries, COTS software configurations, and packages of technical components that require the recipient to integrate and update them for their state specific needs.

Tier 2 reusability has been less common, although a number of states are discussing and exploring the reuse of code and other technical deliverables. One of the Tier 2 areas likely to be reused most involves states using similar COTS products for their efforts. COTS solutions, by their very nature, have the potential to be reused by multiple states. Software vendors will generally update and improve their products as they get implemented and as new or updated federal guidance becomes available. For instance, our interviews indicate that three of the states using the same COTS product for their portal have been meeting to discuss their development efforts with this product. Another option, given that both CMS and vendors are still developing MAGI business rules, is that states could potentially reuse these rules to reduce costs and time. CMS has estimated that costs and development could be reduced by up to 85 percent 32 when states reuse business rules when compared to custom development

When you’re simply talking about using the same piece of commercial software among multiple parties, you’re far from realizing the opportunity of open source. That said, the work developed by these states was really meant to be the foundation for reuse by others, so perhaps that was just the beginning. We do in fact have good precedent for recent open source efforts in the healthcare space. Just take a look at CONNECT, OSEHRA, or Blue Button.

Maybe there actually has been real re-use among the state exchanges, but so far I haven’t been able to find much evidence of that or any signs of open source code in public as was originally intended. Then again, there’s still time for some states to open up their own exchanges, so maybe we’ll see that over time. Right now the attention isn’t on the states so much anyway, but it turns out the Federally Facilitated Marketplace was supposed to be open source as well.

I’m not just talking about the open source work by Development Seed that recently disappeared from the Centers for Medicare & Medicaid Services’ Github page, I’m also talking about the so called, “Data Hub” that has been a critical component of the infrastructure – both for the federal exchange and for the states. The contractor that developed the Data Hub explains it like this:

Simply put, the Data Services Hub will transfer data. It will facilitate the process of verifying applicant information data, which is used by health insurance marketplaces to determine eligibility for qualified health plans and insurance programs, as well as for Medicaid and CHIP. The Hub’s function will be to route queries and responses between a given marketplace and various data sources. The Data Services Hub itself will not determine consumer eligibility, nor will it determine which health plans are available in the marketplaces.

So the Data Hub isn’t everything, but it’s clearly one of the most critical pieces of system. As the piece of integration that ties everything together, you might even call it the linchpin. What appears to be the original RFP for this critical piece of infrastructure makes it pretty clear that this was meant to be open source:

3.5.1 Other Assumptions The Affordable Care Act requires the Federal government to provide technical support to States with Exchange grants. To the extent that tasks included in this scope of work could support State grantees in the development of Exchanges under these grants, the Contractor shall assume that data provided by the Federal government or developed in response to this scope of work and their deliverables and other assets associated with this scope of work will be shared in the open collaborative that is under way between States, CMS and other Federal agencies. This open collaborative is described in IT guidance 1.0. See http://www.cms.gov/Medicaid-Information-Technology-MIT/Downloads/exchangemedicaiditguidance.pdf.

This collaboration occurs between State agencies, CMS and other Federal agencies to ensure effective and efficient data and information sharing between state health coverage programs and sources of authoritative data for such elements as income, citizenship, and immigration status, and to support the effective and efficient operation of Exchanges. Under this collaboration, CMS communicates and provides access to certain IT and business service capabilities or components developed and maintained at the Federal level as they become available, recognizing that they may be modified as new information and policy are developed. CMS expects that in this collaborative atmosphere, the solutions will emerge from the efforts of Contractors, business partners and government projects funded at both the State and federal levels. Because of demanding timelines for development, testing, deployment, and operation of IT systems and business services for the Exchanges and Medicaid agencies, CMS uses this collaboration to support and identify promising solutions early in their life cycle. Through this approach CMS is also trying to ensure that State development approaches are sufficiently flexible to integrate new IT and business services components as they become available.

The Contractor’s IT code, data and other information developed under this scope of work shall be open source, and made publicly available as directed and approved by the COTR.

The development of products and the provision of services provided under this scope of work as directed by the COTR are funded by the Federal government. State Exchanges must be self-funded following 2014. Products and services provided to a State by the Contractor under contract with a State will not be funded by the Federal government.

So far I haven’t been able to find any public code that looks like it would be the open source release of the Data Hub, but I remain optimistic.

Open source software is not a silver bullet, but it does a lot to encourage higher quality code and when coordinated properly it can do a great deal to maximize the use of tax dollars. Beyond the transparency of the code itself, it also helps provide transparency in the procurement process because it makes it easier to audit claims about what a company’s software is capable of and what software a company has already produced. It also tends to select for a culture and an acumen of software engineers that is genuinely driven to work with others to push the capability and impact of technology forward.

I hope we can stay committed to our obligations to maximize the tax dollars and ingenuity of the American people and stay true to the original vision of the exchanges as open source infrastructure for the 21st century.

How can OSS-friendly Federal agencies encourage OSS in states? Agencies that fund activities in the states could be helped by OMB guidance that encourages Federal agencies to provide funding in ways that reward states for posting code and for reusing code developed elsewhere. There is a very big inertia, knowledge gap, and fear problem that has to be overcome. Sweeteners might help that process along. The savings are obvious, but this is not enough.

Medicaid is mentioned above. However there are many other analogous fed/state programs characterized by similar business processes being implemented differently by 53 separate shops.

Wow the state, is really cracking down isn't it? It seems like things are about to happen.

WhiteHouse / source-code-policy