FINOS DevOps Mutualization Project Agenda - July 30th 2020 @ 6pm BST / 1pm ET

[mcleo-d]

Description

The FINOS DevOps Mutualization Project is scheduled to meet on July 30th 2020 @ 6pm BST / 1pm ET.

DevOps Mutualization aims to solve common engineering problems by providing a continuous compliance and assurance approach to DevOps for financial services and fintech.

The group last met on 18th June with the meeting minutes found here https://github.com/finos/community/issues/44#issuecomment-648791809

Agenda

• [x] Convene, roll call, welcome new people
• [x] Approve previous meeting minutes - https://github.com/finos/community/issues/44#issuecomment-648791809
• [x] Discussion topics listed below ...
• [x] FINOS Question : Should we consider DevOps Mutualization a Special Interest Group now SIGs have been introduced to FINOS? #61
• [x] AOB, Q&A & Adjourn

Discussion Topics

• Topic 1 : Structuring conversations around SDLC and sharing what's being done to tackle the problem #55
  • Facilitated by Peter Thomas, Deutsche Bank
• Topic 2 : Iterating the different types of evidence that need to be produced as part of GitHub Issue #55
  • Facilitated by Stefanos Piperoglou, Citi
• Topic 3 : Understanding available tooling and what changes are required as part of DevOps Mutualization consensus #62
  • Facilitated by George Kichukov, Citi. Jamie Jones, GitHub. Anders Wallgren, CloudBees. Timothy Johnson, CloudBees
• Topic 4 : How to provide a mechanism for continuous conversation as Slack is not regulated and isn't the right place for team members to communicate outside the foundation
  • Facilitated by Gus Paul, Morgan Stanley, Karel Deman, Scott Logic

Please leave your feedback on the agenda above in the comments below #52

We look forward to your ideas and contribution.

The FINOS Team.

[mcleo-d]

As DevOps Mutualization continues to establish itself, it would be great for participants to read the following FINOS Meeting Procedure that includes running special interests groups. This will help with setting agendas and taking minutes moving forward.

• https://github.com/finos/community/blob/0fb1a6cf9b22bf687808e2ebab9426dcf071a1e2/governance/Meeting-Procedures.md

[jbjonesjr]

One of the outstanding questions I had heard on the last call was for:

Iterate the list of the different type of evidence, what risk we think it's trying to address, and where the source fo that evidence typically comes from.

as a way to identify the first movements on this group. Would like to check to see if any of the banks made progress on that. This should map to our Topic 2 from above

From my side, George from Citi asked about Tekton support with Actions. That is something we're looking into, but no immediate roadmap feedback (note, our roadmap is now public at https://github.com/github/roadmap ) on it, however we are embarking on many of these same themes (better k8s support, scaling, etc) later this year.

[peterrhysthomas]

Here is the list we came up with in point 1's discussion:

• Record of code review/4-eyes check
• Record of test execution
• Record of test result acceptance/sign off
• Record of code/image/vulnerability scanning
• Environment deployment history/promotion
• Record of ITIL related control points
• Conformance to other control points – CSO or Architecture compliance
• Code/config changes - commit id/PR/etc
• Traceability to requirements/jira issues, etc
• Test plan
• Change classification - material/minor - impact and testing scope, risk, etc
• Change/risk assessment - maybe some automated risk assessment, blast radius assessment

[mcleo-d]

All - Thank you for attending the second FINOS DevOps Mutualization formation meeting last Thursday. It was great having so many FINOS members join the call and input into the project.

Please find a list of the attendees below as I complete the meeting minutes and add them to the issue. Also, please let me know your GitHub ID in the comments if it's missing from the list.

DevOps Mutualization Meeting Attendees

Date and Time : Thursday 30th July @ 1pm ET / 6pm BST

Name	Firm	GitHub ID
Andrew Aitken	Wipro
Eric Tice	Wipro
Murali Kaundinya	Wells Fargo
Rajeev Agrawal	Wells Fargo
Karel Deman	Scott Logic	@kdeman
Amol Shukla	Morgan Stanley
Dov Katz	Morgan Stanley	@DovOps
Gus Paul	Morgan Stanley
Jamie Jones	GitHub	@jbjonesjr
James McLeod	FINOS	@mcleo-d
Maurizio Pillitu	FINOS	@maoo
Rob Underwood	FINOS	@brooklynrob
Tosha Ellison	FINOS	@toshaellison
Peter Thomas	Deutsche Bank	@peterrhysthomas
Shay Naeh	Cloudify
Anders Wallgren	CloudBees
Tim Johnson	CloudBees	@tcraigjohnson
George Kichukov	Citi
Paul Groves	Citi	@grovesy
Stefanos Piperoglou	Citi	@citistefanos
Tyler Bell	Capital One
Lee Faus	Armory	@leefaus

[mcleo-d]

Please find below meeting minutes from the DevOps Mutualization Formation Meeting that took place on Thursday 30th July @ 1pm ET / 6pm BST.

DevOps Mutualization Meeting Minutes

Date and Time : Thursday 30th July @ 1pm ET / 6pm BST

• The following question was asked to the group, "Should DevOps Mutualization become a Special Interest Group now SIG's have been introduced to FINOS?"
  • The attendees were asked to comment, 👍 or +1 in #61.
• It was explained the output of the previous meeting have become the discussion topics for this meeting.
• The following topic was introduced to the group, "Structuring conversations around SDLC and sharing what's being done to tackle the problem #55".
• Software Supply Chain with Grafeas and Kritis introduced was shown via a slide.
• The question was asked, "How do we create a taxonomy around the metadata to gather a software supply chain?"
• It was explained that Grafias stores information and validity tests whilst Kritish enforces policy for the data stores in the Grafias database.
• It was asked whether a straw man of common pieces of metadata should be created that could include the aspects below ...
  • Record of code review/4-eyes check
  • Record of test execution
  • Record of test result acceptance/sign off
  • Record of code/image/vulnerability scanning
  • Environment deployment history/promotion
  • Record of ITIL related control points
  • Conformance to other control points – CSO or Architecture compliance
  • Code/config changes - commit id/PR/etc
  • Traceability to requirements/jira issues, etc
  • Test plan
  • Change classification - material/minor - impact and testing scope, risk, etc
  • Change/risk assessment - maybe some automated risk assessment, blast radius assessment"
• The group was then asked, "Do these common factors sound reasonable and what else can we add to the list?"
• It was agreed all points strike as necessary and certain teams like to gather evidence surrounding the changes that get added to code.
• It was also pointed out that change types and requirements are missing from the list.
• A question of the "artefact types" expected to be collected was asked and also items like "technical design documents"?
• The group fed back collecting information makes a lot of sense.
• It was agreed that we need to collect the information, change types and audit trail related to all software changes.
• The group was asked, "Is all the information actually needed as not every commit is a perfect business requirement and edge cases are difficult to track?"
• It was explained the change type does often influence how software changes move through the release pipeline.
• It was explained there is work being done on how to identify material change as audit trails should change to something more reliable than human, risk based, self categorisation.
• The approach should be more artefact related in order to reduce the risk surrounding making a change.
• The question "Is there some form of automated risk assessment that improves the speed of deployment whilst reducing risk to live systems?" was asked to the group.
• It was added that people want to know the blast radius of change in any given environment and architecture often influences the blast radius. Tightly coupled architectures have a greater blast radius to microservices.
• It was also added, items not thought through whilst coding are the areas that hit teams. Hitting deadlines with last minute changes are also the types of behaviour that increases risk of breakage.
• The group fed back that ticking boxes manually on change requests is a horrible way to determine success as too much paperwork with standard responses. The intent behind the change process is good, but the mechanism is wrong.
• The group was asked, "Do we also need to consider what is a release and what's the difference compared to a commit?"
• It was noted the team should capture the "What" around the topics and and focus on the "How" at a later date whilst implementing the change.
• Group members agree the topic is close to their heart and an initial set of taxonomy is useful as we won't need to reinvent.
• The question "Is there a good way to store and share the information?" was asked to the group.
• A clarification of sharing artefacts or sharing policies was asked as risk and policy teams have implemented homegrown systems across 20 years to collect evidence.
• These teams are now moving towards industry solutions like JIRA and Service Now where attachments can be added to immutable stores.
• The question, "What are the things, how are we tackling, should we share notes and get regulators involved in a more automated way?" was asked to inform a standardised way of presenting the information across banks so regulators can understand.
• It was added that we should go above what the regulator wants and build a secure pipeline that reduces lead times from a security point of view as security requirements are more strict as they want demonstrable proof.
• The group agreed we should show the regulators what good looks like.
• It was explained that many silos have been removed through DevOps but there are now lots of data silos. Getting visibility across data is important to consider.
• Orchestration tools have been created but gluing them together is the interesting part. Tools out of the box also don't give metrics, so they need to be glued together to obtain.
• It was added that self service is important so things can be setup in a secure and right way by the team. Self service in cloud is important, especially considering banking is wedded to approval mechanisms and raising multiple tickets for orders.
• It was suggested an orchestration engine should be open sourced so external parties can integrate their tools. This way we reach a financial services unified voice. Open sourcing and comparing notes is a valuable conversation.
• Also, self services gives the change frequency, regulatory compliance and security needed whilst being super important
• The group was asked, is there value having a whirlwind tour of people who represent their current pipeline orchestration over a call? The group agree there is value with many volunteers.
• The group was also asked to add vulnerability scanning tools to the discussion topic list as group representatives don't want to roll their own solutions.
• The group fed back that we might find many members are following the same principles of DevOps orchestration with some differences. If one bank leads, we can say whether we're the same or different.
• The group would like to answer the question of where we've been forced to build bespoke DevOps orchestration solutions and therefore duplicating effort.
• The priority are those who have built bespoke on premises solutions with the vendor response coming later. JPMC and Goldman Sachs will be invited to the next call.
• It was added that it would be good to know what vendors are never going to be able to answer, such as, will this change affect a desk in Tokyo?
• The question of how the group communicates will be taken into the next meeting as a topic of discussion with the question "What are people comfortable sharing in public?" asked in advance.
• The question about DevOps Mutualization becoming a special interest group will be kept open for people to feedback and vote on #61.

Group Questions and Discussions

The following are DevOps Mutualization group questions that are open to asynchronous feedback and discussion.

• (#61) Should we consider DevOps Mutualization a Special Interest Group now SIGs have been introduced to FINOS?
• (#55) Structuring conversations around SDLC and Iterating the different types of evidence that needs to be produced
• (#62) How are banks orchestrating DevOps and the 'Glue' utilised to create existing self-service workflows?
• (#63) Whilst building effective group communication, what are people comfortable sharing in public?

[mcleo-d]

@tcraigjohnson asks here ... https://github.com/finos/community/issues/44#issuecomment-668159551

On last week's call, I believe it was Stefanos was talking about the burden of Change Management. They have an astounding number of manual approvals in their process that sound like they are little more than someone ticking a box because Change Management requires it. In a segment that has to move fast, that's a lot of administrative burden.

Here's my questions for the forum:

How do you change Change Management? What evidence and automation would the CMB need to actually improve and streamline the process? Are you changing people to change the process or do you show how to change the process to change the people?

[mcleo-d]

@peterrhysthomas - I have added these points to the meeting minutes and have also added to #55 where the subject matter discussion should continue 👍

Here is the list we came up with in point 1's discussion:

• Record of code review/4-eyes check
• Record of test execution
• Record of test result acceptance/sign off
• Record of code/image/vulnerability scanning
• Environment deployment history/promotion
• Record of ITIL related control points
• Conformance to other control points – CSO or Architecture compliance
• Code/config changes - commit id/PR/etc
• Traceability to requirements/jira issues, etc
• Test plan
• Change classification - material/minor - impact and testing scope, risk, etc
• Change/risk assessment - maybe some automated risk assessment, blast radius assessment

[guspaul]

@tcraigjohnson asks here ... #44 (comment)

How do you change Change Management? What evidence and automation would the CMB need to actually improve and streamline the process? Are you changing people to change the process or do you show how to change the process to change the people?

We are starting to go through this right now, and I am very keen to swap notes on what others are doing. We are not even using Service Now to capture change requests, which I think makes us an outlier, but even if we were, I feel like we would still have to cover other assessments to automate the approvals.

Things like

• Is every change to software reviewed in a pull request.
• What level of testing has been done
• What level of security scanning has been done.
• Is the deployment/rollback fully scripted/automated
• What is the potential blast radius of an outage?
• Are there any currently open incidents on the system being changed?
• What is the historical risk profile (we are starting a pilot with http://www.numerify.com/) to help us with this.

I think some of those can be yes/no answers, but others might be more graduated, which would lend itself to an assessment rather than fully automating the approvals. So you would automatically grade the release ( a bit like this site does for security of websites https://securityheaders.com/ ) which would let you better highlight the releases that need more oversight. You could then move the boundary up and down for which releases got auto approved depending on current context.

[mcleo-d]

@DovOps and Group Members - On 10th September at 11am ET / 5pm BST we have the opportunity to present a 3 minute DevOps Mutualization update on the FINOS All Community Call as a FINOS Focus Project. See issue #53

Please use 👍 to register your interest to present and list your thoughts on the topics below ... 🚀

[mindthegab]

Can this be closed @mcleo-d ?