Lessons Learned from the Defense Intelligence Information Environment on OSS

Copying and pasting the contents of the document for those who can’t open .xlsx files. The views expressed herein are those of the original poster, not my own. All emphasis matches the original. I've added "[sic]" where a reasonable reader might otherwise think I'd erred in copying.

Item Reviewed: Draft (Mar 2016) Federal Source Code Policy – Achieving Efficiency, Transparency, and Innovation Through Reusable and Open Source Software

Comments From: Defense Intelligence Information Enterprise (DI2E) Program Management Office (PMO)
Date of Review: March 2016

Cmt #	Name/Contact Info	Page/Line	Comment w/Rationale
1	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	General	The DI2E program was established to align with and promulgate many of the tenets in this proposed policy. We have generated a body of knowledge, lessons learned and other information that we think could be helpful to this effort, and hope to facilitate collaboration and establish a mutual beneficial working relationship with the organization ‘to be’ the federal government’s steward of the Code Repository.
2	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	General	Not being familiar with how Policy documents are written for Federal Agencies and not knowing if this follows a template or not, this comment is written from ignorance, and because this reviewer found the format very hard to follow in its narrative-like form. It would help to have a format that includes sections as follows (bold for emphasis of added sections): Background; Authority/Authorities; Goals/Objectives; Roles and Responsibilities; Scope and Applicability; Policy Provisions, where the content of Sections 3-5 would be addressed under this heading; Appendices.
3	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	General	Within the DOD\IC arena, a policy as significant as this is would have a more formal Program Management Plan (PMP) and Program Master Schedule (PMS). Later comments provided will identify things deemed critical to the success of this effort, and some of these would be items captured in a PMS or on a Program Roadmap. Other things would include critical events (and dependencies) like the policy approval date, due dates for Agency CIO ‘deliverables’ (as identified in implementation section’), the Pilot’s key milestones, and target for full implementation, etc. such that the impacted Agencies can also understand when they will have to adjust their contracts and processes to be in compliance. This needs to be provided as a companion ‘artifact’ to the policy itself; it also needs to be a living document.
4	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	General	We have significant “Lessons Learned (LL)” from the startup, operation and decommission of the ICs Enterprise Registry and Repository (ER2) over the period 2005-2016. ER2s original purpose was to make software and Services-Oriented Architecture [SOA] run-time Services available for reuse across the community (COTS/GOTS/FOSS/GOSS); a mission similar to the proposed implementation plan for this effort with specific emphasis on OSS. As we learned, successful establishment of a R2 is NOT an insubstantial effort is critical to the success/failure of the effort, from ‘tool’ selection to governance and strategic communication/change management. From our LL, establishing a common platform that is easy to use and meets the core objectives is difficult but essential, and we would recommend you use an OSS product for this at the outset. It is unclear from draft review materials if a tool has been identified; however, if this work is TBD or not sufficiently mature, we think it is imperative that a significant level of maturity must be demonstrated to a core group of stakeholders BEFORE this policy is released. (We encourage you to use Agile SW development as well.) Additionally, your governance approach and implementation plan should also be developed and matured enough to release in parallel with the policy. DI2E SMEs can share knowledge on why these points are important and how you might be able to leverage other existing services (to include DI2E resources) to help you. Another organization such as the Program Manager, Information Sharing Environment (PM-ISE) may be very useful to you in this effort as well. PM-ISE is already a champion for information (and resource) sharing across DOD/IC and Fed/Civ/State/Local/Tribal governments.
5	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P3, Para 1, line 57 & App A: Definitions, p13, line 427	As it relates to the footnote, and definition for covered Agencies, the Federal Information Security Management Act (FISMA) of 2002 was superseded by FISMA 2014. (PL 113-383) Recommend you update reference. Also, recommend you add another Appendix to document that includes all the reference citations in the document with URLs to actual documents.
6	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	3, para 1, lines 53-70	Believe this section should include objectives related to: establishing governance mechanisms to manage the pilot and subsequent implementation effort establish program authority for the acquisition, program management, operation and maintenance, oversight, etc. of the program effort (because what is described in this policy document is a program that needs to be managed.) To this point, it is unclear if there is any specific organization responsible for the initial actions called for in the document; is it the White House, is it OMB, is it DHS (like FISMA)? This needs to be specifically cited. If that authority is transitional, and it will be passed off to another organization following the pilot, this needs to be identified also. This is where governance comes in to play. For example, later in policy document all ‘federal agency CIOs’ are supposed to execute actions. Ideally, this would not be done in a vacuum; that is, some kind of governing body should be coordinating the activity of the agency CIOs, maybe helping create templates, standard processes or artifacts so as to do common actions in a common way, etc. As written, the policy leaves everyone on their own – this will not be effective and whoever is in charge will be ‘herding cats’. #4 needs to be much more specific or be more precise. Instructions provided are vague, the sites pointed to (e.g. project open source) are ‘coming soon’, and, as previously noted, some key activities such as having an established code registry\repository with defined content metadata schema and search tools, governance processes, and other resources need to be in place when policy is signed.
7	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	General and p3, Para 2, line 76	In keeping with cmt #1, we believe this policy is essential to changing behaviors in order to achieve the stated goals of efficiency, transparency, and innovation; thus, we think it is imperative that this policy ALSO apply to National Security Systems just as the requirements for Clinger-Cohen, FISMA, and other OMB direction relative to the management of IT systems and investments do; that is, where certain exceptions are made for NSS IT systems. We would recommend it be handled in a manner similar to FISMA 2014 (delegation of authority to DOD/IC CIOs for implementation and compliance). The intent already aligns with a key DOD/IC goal and objective, and use/reuse of this policy by DOD/IC to drive a similar approach/implementation (using DI2E or other mechanisms) should help drive similar change into the DOD and IC communities which are struggling with the same issues identified in the Exec Summary/preamble of this document. Benefits would be even greater if cross-community sharing is enabled (that is, sharing across the entire federal government), except where policy precludes this. The NSS community is already using more commercial and OSS SW today, so a reuse/sharing policy such as this makes sense and only codifies expectations. (Also, just as exceptions are granted in this policy for some items, the same could be done for NSS (e.g. special compartmented SW).
8	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P3, para 2, line 77	Would change USC 3542 to USC 3552 to be more current and align better with a more recent IT regulation (e.g. FISMA 2014 uses this callout for NSS definition).
9	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P3, para 2, line 94	See comment 7. This statement advising CIOs and CAOs to ‘immediately being working together to implement this guidance’ is too vague, and non-specific, especially lacking any overarching Plan, Milestones, and/or real understanding of what they need to do and when they need to do it. Also, policy and guidance are two different things – policy is direction, guidance infers it is not necessarily direction – use of terms is important. Recommend removing this statement here, and later including a specific notional Plan of Action and Milestones (POA&M) that can be provided to the Agency’s CIOs (to coordinate with CAOs and other internal stakeholders) that aligns to a Program Master Schedule as described in cmt 6. The POA&M can use a “start date + x days approach” for notional schedule planning.
10	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P4, para 3, line 106	Reco the Figure in Appendix B (or any update) be placed within the text body to add better context to the process description that follows. Within the DOD/IC community, in general, there is a program or systems-engineering or software engineering lifecycle phase, which might begin with Concept Exploration, progress to Design/Development, Operations and Sustainment, and Retirement. Is there n Agency SW Development Life-cycle Model? If so, would recommend that it be introduced and/or referenced, and it be clearly stated where in the life-cycle the Alternatives Analysis (Step 1) be done.
11	[sic]	P4, para 3, line 106 & p15, Appendix B	This Figure is incredibly oversimplified, and could very likely result in the opposite effect; that is, a quick decision path for custom SW development, esp lacking an established Registry/Repository. Within DOD/IC space, a more detailed Business Case Analysis, which would include an Analysis of Alternatives, is performed. As written, Steps 1 and 2 are confusing and appear to be somewhat blended together as terms related to SW reuse and types of SW are blended throughout both paras. Further confusing matters is having Step 2 labeled as “COTS Solutions” when the emphasis of policy is on OSS; this seems contradictory. This section needs considerable clarification; e.g., perhaps step 1 is: 1. Consider all OSS alternatives – FOSS or GOSS – considering the operational and mission objectives; total LCC; price for performance; value to govt & citizens; security, privacy and interoperability; and cloud solutions first. This would include consulting the (eventual) R&R where they would all be identified, right? Then it would make sense for Step 2 to be specific to COTS, BUT it should also include GOTS (Government Off the Shelf) SW as well. Then only after all reusable options have been exhausted and/or ruled out as cost-prohibitive or non-value add, or not meeting mission, then allow for custom code development, but require that code to be developed IAW this policy. One other point – Licensing cost and considerations are important when looking at COTS/GOTS so some discussion about possibly using or leveraging enterprise licensing agreements should be considered or leveraging the host organizations agreement if possible.
12	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P4, para 3, Line 116	There is an error in the footnote callout. The url cited did not resolve; the one that worked for me was: https://www.whitehouse.gov/omb/memoranda_m97-02.html
13	K. Holladay/DI2E PMO,br>Kimberly.holladay@di2e.net<brD. Treat/DI2E PMO, david.treat@di2e.net, 719-964-1612	P5, para 3, bullet 3, line 136+	Add a sentence after the first sentence which states: “The custom-developed source code must be developed in a government-owned development environment using government-owned development tools.” An example environment can be found at www.di2e.net (DOD Intelligence Information Enterprise (DI2E))
14	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P5, para 4(1), line 157	Again, LL within DOD/IC community – when, how and how often delivery is made is key. Once may not be enough! So, it is important to ensure that deliveries are made when first put into production, on any major upgrade, and at the end of the contract (contract transition) at a minimum. Presumably, the delivery repository would be (eventually) the Registry/Repository you stand up, or has there been consideration of a Federated approach? (In other words the R2 architecture is not really identified here, and seems like it may still be in development. LL is federated approach with centralized content management is a best practice approach, if possible, but requires strong governance and adherence to standards.)
15	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P5, para 4(2), line 162	The FAR and DFARS define 8 different categories of data rights for tech data and computer SW currently. This para suggests that only one of these 8 will be allowed in the future for all Fed Govt contracts. Will there be corresponding action to modify the FAR/DFARS clauses to align with this policy? If not, how will Acquisition Officers be able to get approval from Agency lawyers and Contracting Officers on putting these clauses in contracts if FAR/DFARS are conflicting?
16	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P5, para 4, line 168	Project Open Source Link was ‘coming soon’ when one goes to the link? As this is the primary resource supporting this policy document, a ‘wireframe’ or some level of detail on what will be provided and accessible via this page should be provided to the community as part of this review process. To points made earlier, all of the governance activity should be transparent and made available to all via the site (perhaps some discussions could be handled in Exec Sessions). What is the plan, timeline for this site?
17	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P6, para 5	This whole section is informational in nature –not policy language. It belongs in the ‘front matter’ or as suggested in another comment recommending some document reorganization, put in Background.
18	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P7	Footnote 30…consider referencing the DOD Modular Open Systems Architecture (MOSA) Handbook as well as this document: https://acc.dau.mil/osaguidebook
19	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P7	Footnote 33 – error in URL…link is NOT an https but is an http
20	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P8, para 5.1	The Pilot Program explanation and approach is very vague. Also, would think that it might be just as valuable to know/understand how many agencies are finding reusable SW in the Registry/Repository to be stood up as it is important to ensure that 20% of new custom code is OSS. Isn’t a key goal to incentivize reuse of other agencies code, or to use already available FOSS\GOSS? In other words, Metrics collected should align to the objectives in section 1 (in their end state). Also, if you expect the covered Agencies to meet metrics on an annual basis, but you are giving OMB effectively 3 months after the policy is signed to come up with the metrics the agencies are going to be measured by, you are setting your pilot up for failure. Why? Metrics are accomplished as a function of process (whether manual or automated), not to mention that this one identified metric involves changing contracting practices, not a simple thing to change. Thus, it is critical that the metrics should be developed and communicated as part of the policy, and that the stakeholders agree that the metrics are something that they can collect, especially within the timeframe specified. Finally, it is important that there is a common methodology for collecting the metrics from the covered agencies, and an understanding of how the metrics will be assessed and used. For example, if an agency has 0% but that is because they had no new custom developed code, but reused 80% of OSS code or GOTS already available – isn’t this a win? The pilot protocol would suggest otherwise; that is, that it is bad to be at 0%. It is important to understand the behavior you are trying to encourage and incentivize and ensure that the metrics program aligns to this; this proposal does not, and the metric serves no real purpose/value.
21	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P8, para 5.2	Totally support the concepts and principles within this paragraph. Based on LL, much of this is tied back into governance, which as noted is a critical component to the success of this effort and is a thread that needs to be consistent throughout the policy. Consistent with other comments, it is important to clearly articulate if these “principles” are intended to be policy mandates (that is, “must do” vice “it’s optional”), and some are clearly reliant on a Program Office to provide the support (as suggested in Para 1-Objectives, item 4) to actually make the action workable. Again, the DOD and IC have much experience in these areas already and LL (some of which you have cited) but an organization has to take the lead for corralling the collective efforts called for in this policy. At a minimum, this means the PMO needs to establish the common Reusable SW Registry/Repository (R2) (to incl FOSS/GOSS/COTS/GOTS and new DEV done as OSS) whether that Common SW R2 is Centralized or Federated; Develop a Common Metadata Schema for the SW Content Records for all categories/types of SW and specify minimum mandated fields to be completed by lifecycle phase by the developer/developing program PM; ensuring that the metadata content is DISCOVERABLE consistent with common industry (IT) content discovery and retrieval standards; require persistent maintenance of the record over the lifecycle of the SW; capture user metrics (even though open to user community, it is still important to know if it is being used to validate investment) ; setting a minimum standard of common development tools (i.e. common development tool environment – DI2E has done this and can offer advice). As Federal Agencies are supposed to be compliant with the Federal Enterprise Architecture Framework (FEAF), and possibly others (like NGA – Geospatial is also compliant with the ICs Joint Architecture Reference Model (JARM) ) identifying a SW reusable component or system to its architecture ‘fit’ may also be of value, but that requires significant maturity and would likely be an improvement for much later in this program’s lifecycle. However, compliance with IT standards may be something that is very important for reusability and this may be something important to capture even as a more static architecture view stored as a document with the SW asset record. How are Federal IT Standards specified?
22	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P8, para 5.2a, line 262	It is not enough to say go find a community (Community of Interest (COI) or Community of Practice (COP)) to leverage…from a governance perspective, this information needs to be published/made available; i.e., users should not have to go searching for it. Thus, you should require COIs/COPs to register their ‘fact of’ existence in a common location (whether that is part of the SW R2, Part of a Dev Tool Environment offering [e.g. Confluence wiki]) as par of this policy, and make it the responsibility of the Agency CIOs to gather the data and maintain it.
23	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P9, para 5.2b, line 267	Comment 210 stressed the point about establishing either a common or federated approach to a common Dev Tools Environment. This is important especially if you want to ensure interoperability, reusability and compatibility of data across tools. For example, JIRA is a very commonly used tool for developers doing agile, and for bug tracking, etc. Some other developers use RedMine. They can share data, but it is not necessarily easy or pretty. Also, if you are engaging the public, you want to ensure the tool you choose doesn’t require a computer science degree to use (that is, use the ‘KISS’ principle in tool selection).
24	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P9, para 5.2c, line 274	Delete this paragraph. It is more confusing than it is directive, and seems to describe a rare instance based on what the policy is trying to achieve.
25	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P9, para 5.2d, line 274	Two thoughts on this para: The purpose of OTD (addressed in b) is specifically this – that is engaging the using community to make the product better by providing input on code during development, prioritizing features, forking – if appropriate, etc. So would merge elements of this para with para b. Stakeholder Feedback is part of what is missing in talking about User Engagement. That is inherently tied back to Governance as well. What are you going to do to get input from the community on the reusable apps in the common or federated respositories? Will you employ a ranking/rating system like Amazon? Have people write reviews? What will you do with metrics? Etc. Recognize who your stakeholder’s are, and what their needs are and address a strategic communications plan/approach to address this.
265 [sic]	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P9, para 5.2e, line 285	Similar to comment made about user engagement, Code contributions is the essence of Open Development. This paragraph just adds additional narrative but does not compel any separate policy action. Consider bundling it into a single paragraph with a declarative statement that states the policy action that is: ‘the covered agency will use Open Technology Development principles which include user engagement, code contributions, etc. …when developing their agency unique OSS newly developed SW code.”
27	L. Nasta/DI2E PMO,br>laurie.nasta@di2e.net,<br703-449-3336	P9, para 5.2f, line 292+	The overarching comment directed against Para 5.2 overall noted the importance of whatever organization will be the PMO for this effort setting up a Common SW R2, and identifying a common metadata schema with common mandated fields over the asset’s lifecycle – this includes DOCUMENTATION. In other words, the metadata ‘field’ in some cases may be resolved to a document or a url where a document can be accessed (particularly if using dev tools) such as release notes, or a user manual, or configuration scripts, etc. All of this should be associated to the Asset record, and should be part of what is stored and maintained with the asset.

WhiteHouse / source-code-policy

Lessons Learned from the Defense Intelligence Information Environment on OSS #96