Software should be "Open by default" not 20%

[jackiekazil]

The Presidential Innovation Fellows Foundation (PIFF) has been following this conversation very closely. As the current member-elected President, I wanted to share that an overwhelming majority of respondents to an internal dialog stand by the following statement:

The PIF Foundation membership strongly believes that it is in the best interest of increased government efficiency and taxpayer savings for the OMB to set a policy of 'open by default' for custom software developed by the Federal Government rather than setting a goal of at least 20% OSS as originally proposed in the source code policy: https://sourcecode.cio.gov/ .

This statement was supported by 34 of our members, opposed by 1, with one abstention.

More about PIFF: The PIFF has approximately 120 members. The PIFF is a membership corporation registered in Washington DC with 501c3 Status. Our bylaws restrict our membership to Presidential Innovation Fellows serving for at least 6 months. Fellows are industry-leading innovators chosen via a very competitive process. Many Fellows have gone on to retain important positions in the Federal government, and others have returned to private industry.

Sincerely, Jackie Kazil Member-elected President Presidential Innovation Fellows Foundation

[jateeter]

This proposal will make a great difference with respect to the long term viability of the overall proposal. Forcing individuals or project managers to select an arbitrary 20% of a code base to be published in open source would, I my opinion, be counter productive to the goals of the initiative.

John Teeter

• PIF Round 2
• Green Button (My Data) Initiative
• Open Source Leader of the Green Button (OpenESPI) stack.

[Neetuj]

impressive!

[eads]

I used to work at a national science lab as a software developer. The policy in the national labs is "open by default."

I saw firsthand that this policy was a boon to collaboration, sharing, software quality, and long-term sustainability. I believe it should be adopted throughout the government.

[Momiperalta]

Outstanding, pushing the limits, a great example for other countries to follow , +1!

[jkokenge]

Yay!!

[pahlkadot]

Code for America supports open source as the default. We and 18F recently worked with the State of California on an RFP which used the following language:

The Child Welfare System will use open standards, commodity open source components and tools when they're available. All new source code will be made open and reusable and published with an appropriate license. Where there aren't open standards or open source components, we'll consider proprietary standards against the tradeoff of a long-term open, maintainable system that the state can direct and control. We intend to release the code created under this effort as an open source product.

This sets a higher goal than 20% and this community should think about this as precedence for federal, state and local policies.

[stephani]

This is excellent! Woot woot

[pgibler]

Well done, "open sourcing" 20% of a working system is not open sourcing the system.

[andyoram]

I could understand a directive to organizations who have not operated in a free software/open source context before, allowing them to try one major project in an open source manner to learn the practices and challenges. This would be preparatory to an open-by-default strategy, as people on this thread have recommended. But 20% is totally arbitrary and provides no help for agencies who need to learn open source practices.

[yarbelk]

I believe that making souece code FOSS licenced by default will reduce the barrier to entry. Adding an arbitrary 20% target adds an additional managerial burden that would be better spent on only identifying the core, truly sensitive parts of a code base.

[dkaoster]

Agreed that "open by default" should be the policy. 20% is arbitrary and seems like it could become yet another quota to meet.

Another thing to consider in both "open by default" and "at least 20%" is the guidelines for which it is acceptable for a piece of software to be not FOSS. The process for determining what is released as open and what is not should be unambiguous.

[benbalter]

As one of the above-signed members of the inaugural class of Presidential Innovation Fellows, I would like to further expand upon why government software should be open by default, rather than the government only making a small subset of taxpayer-funded and taxpayer-owned code available to the taxpayers who made its development possible and for whom the software is intended to benefit:

Microeconomic motivations

Open source is in your best interest, whether you're an individual, a corporation, a small business, a non-profit, or a government agency:

• Shift developers from low-value work to high-value work - We like to say in open source that all the easy problems have already been solved. Blogging, content management, and operating systems are all problems with established (and mainstream) open source solutions, to name a few. While your developers could spend their time reinventing wheels that the open source community have already perfected, it's far preferable to use the worlds best wheel, especially when that wheel comes at no cost to you. This frees developers up to work on yet-unsolved challenges, the types of challenges that are unique to and add value to your organization's mission. Why not stand on the shoulders of technology giants?

• Lower total cost of ownership (TCO) - Using open source software yields a lower total cost of ownership when compared to closed source and proprietary alternatives. Adopting open source software generally has a lower up-front cost (because the software often comes at no cost or relatively low cost), and shifts the cost center from licensing (an operating expense) to customization and implementation (a capital expense). Additional costs like training, maintenance, and support are sunk costs. You're going to be paying for both regardless of if the software is open or closed source, the cost often being baked into the license in the case of commercial, off-the-shelf software (COTS). What makes open source unique is that you're not paying for the right to use the underlying intellectual property.

  Cost	Open source	Proprietary
  Licensing	:x:	:heavy_check_mark:
  Implementation	:heavy_check_mark:	:heavy_check_mark:
  Maintenance	:heavy_check_mark:	:heavy_check_mark:
  Support	:heavy_check_mark:	:heavy_check_mark:

• Given enough eyeballs, all bugs are shallow (Linus's Law) - Empirically, open source tends to produce better quality software than its proprietary or alternative counterparts. When you write closed source software, the only developers that can potentially detect, diagnose, triage, and resolve software bugs are those that happen to be employed by the company that publishes the software (or more likely, the arms-length contractor they pay to build it). Open source provides three advantages: first, you have the opportunity to tap the knowledge of the world's best developers, not just those on one organization's payroll. Second, the number of potentially contributing developers and thus the potential knowledge pool is orders of magnitude larger. Finally, open source software gets adapted to a variety of use cases, not just the one the publisher originally intended, surfacing bugs and edge cases much more rapidly than traditional, predictive QA processes.
• Modern software development practices - Open source software is more than simply "published" code. You'd be hard-pressed to find an open source project that follows outdated, waterfall, or rigid command-and-control development philosophies, still common in many large, bureaucratic organizations. By virtue of being distributed and unbridled by policy or technical debt, open source projects all but necessitate modern software development workflows. These workflows are electronic (meaning process is naturally captured and exposed), asynchronous (meaning decisions are time and location agnostic), and lock-free (meaning contributors can rapidly experiment without prior approval). These three workflow characteristics means more rapid development cycles and more frequent releases without sacrificing quality.
• Open source is the future - You'd be hard pressed to find a startup today worth it's venture capital funding not based, at least in part (if not largely) on open source. Open source isn't a fad, or a bunch of hippies out in California passing around tie-dye laptops like they would illicit substances. Open source is how modern organizations, and increasingly more traditional organizations build software. It's becoming exceedingly challenging to make the argument that five-or-ten years from now the technology landscape is going to be less collaborative and more closed. Heck, even Microsoft, traditionally one of the most stark opponents to open source, has open sourced their primary development framework (along with Apple and Swift). Today, all the largest names in technology, from IBM, to SAP, to Adobe actively participate in the open source community. It's slightly inaccurate to say that "open source is the future". Open source has already won.
• Patch on your own schedule - So long as software is written by humans, bugs and security vulnerabilities are inevitable. When a vulnerability is discovered in a proprietary software project, you have to wait for the workday to begin in the company's timezone, for meetings to be held, tasks deligated, and code written, all before G&A teams like legal, sales, and marketing have to coordinate how to get the fix into the next regularly-scheduled release. For proprietary software, bugs and vulnerabilities affect the bottom line, and thus there's a disincentive to make their details widely publicized. With open source, not only can leaner, more agile, non-profit-oriented organization move faster, since you have access to the source code, you can often apply fixes, both large and small, at your own convenience, not at the convenience of the publishing organization's release cycle.
• Upstream improvements - If you consume open source software, it's in your best interest to contribute back. Contributions can be in the form of reporting bugs, or even submitting proposed fixes. Since software is written by humans, it's highly unlikely to be perfect, and even if so, likely doesn't satisfy every use case. Rather than forking the project and implementing changes into your own version (closed source), submitting bug reports and improvements upstream (open source) allows you to more easily continue to benefit from the subsequent fixes and improvements submitted by others.

GitHub Cofounder Tom Preston-Werner (@mojombo) lays out some additional arguments in his post Open Source (Almost) Everything:

• Force multiplier - Open source is a force multiplier for your developers. This happens in three ways: First, when communities form around shared challenges, the diversity of ideas that naturally emerges surfaces better solutions than if the marketplace of ideas was limited to just your organization, meaning developers are working smarter. Second, exposing the problem space to other interested organizations provides additional human capital to tackle your challenge, meaning the solution has more developer hours thrown at it, at no additional cost to you. Finally, "more users means more use cases being explored, which means more robust code."
• Modular - Open source projects tend to be more modularly architected, improving both the flexibility, and the robustness of the code. When you're building software for a single use case, you're able to take some technical short cuts. The problem comes when you'd like to use that software in a different use case, or when your requirements change. Open source, by its nature, is built for a variety of use cases, environments, and users. This means more options (rather than hard coding defaults for a particular use), and a tendency to encourage more modularity (rather than assuing a one-size-fits-all featureset), resulting in greater flexibility and lower customization costs in the long run. Put another way, open source necessitates cleaner, more maintainable code. "Even internal code should pretend to be open source code".
• Reduce duplication of effort - You should focus on your core competency. What makes you unique or gives you an advantage? Everything else is the work everyone else is also doing, or put another way, the work you need to do so that you can do the work that you want to do. It doesn't have to be that way. Open source reduces duplication of efforts, both within an organization and across organizations, by allowing for individual components to be shared. Coca-Cola's secret sauce is the formula for its syrup, not its ability publish blog posts or post press releases. Using an open source CMS, or sharing their built-in-house blog components with the world, doesn't make Pepsi taste any better. "Less duplication means more work towards things that matter."
• Great advertising - Maintainers of successful open source projects are often seen as industry leaders, providing themselves with the ability to shape the conversation around a particular software problem and associating their brand with the preferred solution. 37Signals is known for creating Ruby on Rails. GitHub is known for creating Hubot ("Within two days it had 500 watchers on GitHub and 409 upvotes on Hacker News. This translates into goodwill for GitHub and more superfans than ever before").
• Attract talent - Developers want to work on yet-unsolved problems. Open source allows you to showcase to the developer community, the interesting challenges you face, and how you think about solving them. Open source developers can casually contribute to projects, to learn how you work, and what it's like developing software for a particular set of challenges. If they like what they see, there's a much better chance that they'll apply for a job, than if your organization was a black box when it comes to what it's like to work there. "Smart developers like to hang out with smart code."
• Best technical interview possible - Technical interviews traditionally involve working on a simulated problem that can be tackled in a set amount of time with little additional context. Such simulations, by definition, aren't real world use cases, nor do they show what working with an applicant would be like. Open source provides visibility into both how a candidate solves problems, and how they work with others. You can hire much more confidently if, for the past six months, the candidate has been contributing to the project you want them work on, and you like their work. "[T]he best technical interview possible is the one you don't have to do because the candidate is already kicking ass on one of your open source projects."

Macroeconomic motivations

By combining and augmenting parallel or related efforts, open source makes a society more efficient at producing higher quality software:

• Efficiency - Many of the microeconomic arguments above, in the aggregate, have a macroeconomic impact. When firms work more efficiently, the economy produces more (and better) software, software that can in turn, improve lives. If you believe that all the easy problems have already been solved, then on a macroeconomic level, open source allows you to move firms from lower-level work to higher level, yet-unsolved challenges. The results of this higher level work tends to be disruptive, rather than iterative, creating the churn that invigorates the economy. If every technology company must devote a few years of R & D solving the same 5–10 problems, that delays (or at the very least shifts capital from) the company's ability to be productive and produce technology that's valuable to society, not to mention, it raises the barrier for new market entrants to offer alternatives to established firms.
• Stand on the shoulders of giants - Today, innovative technology, the type of technology that improves quality of life, isn't created in a vacuum. Even closed source technology, from the cell phone in your pocket to the car you drive to the Fortune 500 company that produced them, rely heavily on open source (don't believe me? Take a look at your phone's "settings -> about" page). Just as algebra and trigonometry being unencumbered by private sector copyright gave way to everything from calculus to quantum physics, open source allows developers to take already-solved problems as a given, to rely on the knowledge of experts beyond their own domain of expertise, and unlocks their potential to create new inventions, otherwise not possible.
• Fuel the marketplace of ideas - Software is nothing more than technical knowledge. There was once a time when alchemists would withhold the results of their work, claiming their discoveries as proprietary. Each had to learn the hard way the result on the human body of drinking lead. When those alchemists began sharing their work, we began calling them scientists, and the scientific revolution was born. The same is true of mathematics, literature, and computer software. Two developers working individually may come up with two solutions to a given problem, but through dialog and collaboration, each bringing their own knowledge and experience, may discover three, five, or ten solutions to the same, and society is better as a result.

Moral motivations

The formal name for open source is free/libre open source software. As such, open source motivations have a strong moral component:

• Free as in speech, not as in beer - Open source software is not without cost. When open source software is called "free", it is a reference to the rights the software consumers receive, not the cost they must pay. Adobe's Flash player, for example, is free software in the economic sense, but is still at the core of proprietary (non-free) software (and formats). Specifically, free software refers to four core freedoms: the freedom to run the software, the freedom to study and modify the software, the freedom to redistribute the software, and the freedom to distribute your modifications.

  As the Free Software Movement argues, non-free software risks software that can control the user (with the publisher controlling the software). This creates the potential for the software becoming an instrument of unjust power. Today, it's not uncommon for proprietary software to spy on users (e.g., phoning home), to restrict them (e.g., DRM), to censor them (e.g., corporate firewalls), or to take advantage of them (e.g., unskippable commercials). This becomes even more important as the internet of things emerges, and risks turning the world into "the internet of telemarketers" or "the internet of snoopers". Free software places power back in the hands of users and ensures users control the software they use, not the other way around.

• Obligation to give back - Open source is the give-a-penny-take-a-penny jar of software. If you consume open source, be it a server, a desktop publishing application, or a software library, you have an obligation to give back to the community. After all, without the contribution of others, the micro- and macroeconomic motivations would no longer hold true, and open source as we know it would cease to exist. This is the Golden Rule, or in traditional philosophical terms, a categorial imperitive.
• Governments should give what they've developed to the people who pay for their development - If the development costs are paid by a government, then there's an additional argument for giving back. Governments take money (taxes) to perform services for their populace. For example, the US federal government was established through its Constitution by we the people to perform a number of important tasks. If the populace (we the people) are paying for the development of software, it stands to reason that we should receive what we paid to develop, including the software we paid to develop.
• Teaching the next generation - Many of the industry's most prominent engineers today cut their teeth by learning from open source. When software's underlying code is made available for inspection, consumers can learn how their favorite software works and computer science courses can analyze how the industry's cutting edge technology is built. This goes a long way to train the next generation of software engineers (who without open source would be left guessing at the inner-workings of prior iterations).

Transparency motivations

Open source allows for greater transparency of process, whether that's the transparency necessary to check a government action or the quasi-governmental function of a private company's software.

• Showing your work - As the government increasingly relies on technology to regulate industries and deliver citizen services, being able to see the underlying algorithms and processes are essential to checking the government's work. If a closed-source software package is used to calculate my taxes or allocate broadcast frequencies, how do I know that the process, our process, is fair and accurate? Whereas human processes can't be copyrighted, when close source, such processes become a black box, minimizing the potential for a citizen counterbalance.
• Positions of public trust - As the software produced by private corporations are increasingly placed in positions of public trust, the transparency obligation extends into the private sector as well. Did the voting machine accurately count my vote? When given a lose-lose choice, does the self-driving car conform to community norms? For example, if a closed-source software package is used to compare DNA at a crime scene, unlike a medical expert who can be cross examined, that proprietary algorithm is shielded from scrutiny by copyright (or patent) law. Open source, at least partial open source of particular components, will become essential as private companies automate quasi-government functions.

Participatory motivations

Open source affords software stakeholders, both technical and non-technical, the opportunity to shape any given software development project:

• Direct democracy - True direct democracy (where every citizen votes on every issue) isn't tenable given the size of most democracies. Nor have citizen been able to participate directly in most issues, due to technical constraints. Open source changes that by allowing software stakeholders to participate directly in the software development process (as software is increasingly relied on to codify regulatory and service-delivery preferences and norms). Think the process of buying health care is confusing? Open an issue. Believe the government should use more open source, submit a pull request. The White House is already doing this for many IT-centric policies like the Digital Services Playbook, HTTPS, and open data policies.
• Customer feedback - Open source empowers consumers to have a combined, powerful voice in the private sector development process. Think of it like Yelp for software. Without Yelp, a restaurants is free to upset a single customer. At most, that dissatisfied patrons could dissuadee 5–10 other potential patrons. In a post-Yelp world, customers read the reviews of strangers before they choose where to go. Open source amplifies the voice of software consumers within the consumer-publisher relationship. Not to mention, the feedback the private company receives can go towards better informing product decisions and improve the overall product. Open source gives companies a direct line to their most passionate customers.

Personal motivations

If you're a developer (or an aspiring developers), open source can provide an easy (and free) onramp to and path for personal growth within the software development community:

• Learn to code - Open source is a great way to learn how to code. Want to know how your favorite website works? Hitting "view source" in your web browser can go a long way to point you in the right direction. You can read the documentation of the software that powers it, and potentially even stand up your own clone. Want to learn more? Join the local meetup for the framework or language. Better still, submit a pull request to the project to fix a small bug or add a new feature. While there are certainly time and opportunity costs involved, all of this comes without any direct cost to the aspiring developer, at least not in terms of software, and is infinitely more inclusive than doing the same within a proprietary community, especially as an outsider. There's a reason nearly every coding school today outside of traditional academia teaches open source.
• It's fun - According to Wikipedia, open source is a hobby. In fact, open source has traditionally had the reputation of being the product of hobbyists (although I'd argue that's less true today). Open source is fun. If you're a hacker it provides an endless set of ever-changing set of Rubix cubes for you to solve on weekends. Just as puzzles (both crossword and jigsaw) provide bite-sized intellectual escapes, the order and symmetry of open source can often be a rock garden of code (especially for those for which football is an embarrassing non-starter).

Additionally, I would also like to include @david-a-wheeler's canonical paper, "Why Open Source Software", by reference as part of my comment.

Open source isn't the next big thing. Open source is already here.

[benbalter]

Also to note, it is unclear, what percentage of new government software projects, today, are open source. Anecdotally, I'd suspect that of new projects, as many as 10-20% may already be open source. Thus, a 20% open source mandate (potentially only a 10% increase over the status quo) may not provide an accurate experiment, if it does not sufficiently differ from the control. While, as stated above, I believe the policy should be "open by default", both politically, and in terms of adequate testing, requiring that 50% of all software be open source would provide a more accurate A/B test to gauge the impact of the experiment on government IT and the broader software development ecosystem.

[benbalter]

I'd like to suggest one additional reason why the policy should mandate that code be "open by default", rather than creating a three-year, 20% pilot to evaluate its effects of such a policy. In September 2014, the president, at the United Nations, expanded his commitments under the Second Open Government National Action plan to include the following:

Adopt an open source software policy. Using and contributing back to open source software can fuel innovation, lower costs, and benefit the public. No later than December 31, 2015, the Administration will work through the Federal agencies to develop an open source software policy that, together with the Digital Services Playbook, will support improved access to custom software code developed for the Federal government.

(Emphasis original)

That comittment, along with the statement that "open source software can fuel innovation, lower costs, and benefit the public" is repeated in the very first line of this policy. There are two reasons why the policy, as currently stated, fails to live up to the president's commitment:

1. The president's commitment was to improve access to software developed for the Federal government (which would include contractor-developed software), not just that software which is developed by the federal government (a small subset of the government's total software budget). Improving access only to software developed by the Federal government, while creating a pilot to evaluate the potential effects of improving access to software developed for the government, fails to substantively live up to the president's National Action Plan commitments.
2. The first sentence of the president's commitment stated the benefits of open source software development as a foregone conclusion. It committed to creating a policy that would improve access to government software. A open source code policy that creates a pilot program to evaluate the effects of open source software development is a means of evaluating a statement made within the commitment, not a means of operationalizing the commitment itself. Further, since the goal of the pilot program is to "issue a subsequent policy to continue, modify, eliminate, or expand the pilot program", the policy as written, may not, ultimately, expand access to software developed for the Federal government, and once again, fail to substantively live up to the president's National Action Plan commitments.

[CGIFederalInc]

Open by default has its advantages as noted by many comments above. However, there are important considerations project managers must appreciate when promoting the policy. A few of those are:

• Development costs may elevate with OSS governance overheads
• Contracting and funding models will have to accommodate labor associated with responding to issues raised by the community
• All custom code will not be suitable for reuse, and may not attract a community of developers
• To jump-start the community process, policy must require publishing all project related artifacts, not just source code

[johnmod3]

beyond just open by default, this policy needs to be looked at as a way to simplify delivery for the contractor AND the contracts officer and government technical folk. Make it easy and everyone wins

[IanLee1521]

I'm Ian, a Computer Engineer at Lawrence Livermore National Laboratory, working on consolidating and showcasing the open source development performed by our staff. This comment represents my views, not necessarily those of LLNL.

From @benbalter comment: https://github.com/WhiteHouse/source-code-policy/issues/90#issuecomment-208013645

Also to note, it is unclear, what percentage of new government software projects, today, are open source. Anecdotally, I'd suspect that of new projects, as many as 10-20% may already be open source. Thus, a 20% open source mandate (potentially only a 10% increase over the status quo) may not provide an accurate experiment, if it does not sufficiently differ from the control.

Another issue with the 20% rhetoric which I was going to open as a separate issue before I found #47, #106 and item # 14 of https://github.com/WhiteHouse/source-code-policy/issues/152#issuecomment-208538722, is how to define this 20%? Even if a metric can be decided upon (# of projects and SLOC are two mentioned in #47), how would these be measured / reported / verified? And how does one define a "project" or even a line of code? Etc, etc...

Therefore I believe that a greater push towards "open by default" language is a good one. That said, I do believe that if the policy does move in that direction there should be clarification given to what is excluded (there will always be some things excluded, such as for national security or internal infrastructure (think passwords, database connections, etc) reasons) and who would review such exclusions. Something similar to the 18F Open Source Policy Exceptions would be a good first cut.

[david-a-wheeler]

This policy should say open source software by default. I think also including 20% is a good idea, but it would be far better to do both. It is important to define how to measure 20%. I think that the best metrics is dollars. Dollars are easy to measure, and easy to add up.

[connorshea]

@CGIFederalInc if tax dollars sourced from the American public are spent on software contracted by and used by the government, why wouldn't it be open by default? Photographs taken by Presidential photographers are public domain, photographs created by NASA are public domain, and there's a huge amount of information available under the public domain through US Government Works.

I understand that this would add overhead to projects, but it would also likely improve much of the software used by the government. If security vulnerabilities are discovered, reported, and fixed by open source contributors, then they are well worth the tax dollars. If usability issues are easier to report, or even for external parties to fix, that's a great use of tax dollars as well!

If a project doesn't have a community grow up around it, there is minimal governance overhead to worry about. Government open source projects would be able to reuse the same rules across most, if not all, projects.

That said, thank you for offering a counter-argument, open source is certainly not a perfect solution, but I do believe it's far better than the current situation.

[pjdufour]

(I am Patrick Dufour, a Humanitarian Information Specialist & Data Engineer with about 5 years experience working with government contracting and open source software. My comments only represent my views.)

In regards to comments made by @CGIFederalInc, I have a few comments.

Within the policy under consideration, I do agree that "OSS Governance overhead" is worth explictly mentioning since government managers may bring it up. However, it should be mentioned as a cost inherent in following modern best practices and the policy should also mention that it shall not be construed as a valid reason for not open sourcing code.

Additionally, if following modern devops workflows (Ansible, Packer, etc.), the publishing of project-related artifacts should be a trivial task and not require extra undue burden.

In regards to "open by default", I have a few comments.

There have been tangible limitations to the effectiveness of the "open by default" principle on open data. "Open by default" provides some negative incentives for discovering all the silos (leads to more work). However, the benefits of "open by default" have been incalculable. For open data, what's most important is not releasing ALL the data, but releasing the highest-value datasets. The White House "open by default" directive provides significant support to the pro-release side in internal deliberations about a dataset. Alltogether, "open by default" provides the right incentives.

For open source software, what should be most important is releasing the best reusable code. A 20% mandate will necessarily incentivize centralization of open source code management to reduce the reporting burden on IT management (e.g., 1 GitHub organization). It will also provide negative incentives for modularizing code.

On the other hand, "Open by default" increases support for a decentralized approach, which appreciates that many agencies and offices are already sharing opensource code. "Open by default" will incentivize broad engagement with open source communities at a developer-level, which is more inline with the spirit of the policy. "20%" provides no similar positive incentives.

Open by default is far superior to 20%.