What metrics should be used to determine the impact and effectiveness of the pilot proposed in this draft policy, and of an open source policy more generally?

[CenterForGovernmentInteroperability]

Interoperability should be one of the primary metrics.

Open source removes constraints that prevent interoperability. Programmers are free to create APIs and other connectivity that brings enterprise-wide integration.

Interoperability is often the main driver for government solutions in areas such as cross-agency coordination, business process reengineering, innovation and general efficiency. Many of government's largest problems are related to business software systems that are not integrated. Without continual interoperability updates, you cannot re-engineer your business processes; your IT systems hold your business process improvements hostage. The most important areas for interoperability planning are in national security and emergency preparedness.

Alex Glaros

[mgifford]

Open standards were in many ways the community with which open source emerged. The success of open source has likewise influenced open data & open government.

It really doesn't make sense to build open source projects that aren't geared for interoperability, but if they are engaging with a larger open source software community, then this is something that can probably be assumed (or built).

[CGIFederalInc]

Here's a quick list of metrics we believe will support OMB's OSS pilot program and long-term effectiveness of the OSS policy

Metrics to evaluate pilot program effectiveness:

• Number of OSS projects that have gathered a community, and number of community members with each project.
• Number of issues and comments submitted to the issue tracker for each OSS project
• Number of OSS projects have been forked, number of pull requests submitted, and accepted
• Percentage of reused OSS in net new projects (ideally based on metrics like function points)

Metrics to evaluate long-term effectiveness of the OSS policy:

• Percentage of OSS v/s custom code in mature projects (ideally based on metrics like function points)
• Cost savings (expected v/s actual) is NOT a good metric
• Number of projects that remain modern as per the current industry reference, not requiring a rewrite for modernization
• Custom code written for mature projects remains business specific and is not reusable
• Productivity gain from use of Application Lifecycle Management (ALM) and DevOps processes for new projects (use standard ALM metrics for reference)

[johnmod3]

maybe a Gov-Prize to the Agnecy who releases the most lines of code?