Contributing to the FINOS DevOps Mutualization Project

[mcleo-d]

Description

DevOps Mutualization aims to solve common engineering problems by providing a continuous compliance and assurance approach to DevOps and we’re asking for your input and contribution prior to initiating the project.

Within this GitHub Issue provide your thoughts and ideas that align with ...

• Current or planned areas of development in DevOps targeted at accelerating engineering in financial services whilst removing duplication of effort.
• Existing or “Work in Progress” DevOps tool enhancements related to regulatory reporting or other regulatory requirements that your firm might consider open sourcing.
• Experience of DevOps where existing tools or services providers are falling short in the financial services industry and will benefit from collaborated requirements.
• Areas most prone to DevOps reporting and infrastructure integration risk and hence candidates for improvement.
• Least automated or optimized areas of financial services DevOps engineering.
• 1-3 specific ideas of where collaboration makes sense, the more granular the better.

We look forward to your ideas, feedback and contribution.

The FINOS Team.

[mcleo-d]

The following question was raised over email to be raised in the agenda of the first project meeting ...

• Does FINOS expect all of the existing solutions to be open source by definition?

[DovOps]

Some thoughts I had on areas we can collaborate:

• We can share approaches to continuous gathering of evidence that a certain regulatory-policy around SDLC was followed. (how have our processes reflected our own interpretation of policy). How are we capturing what policy requires us to. What systems have we built/configured/set up to store that evidence, and what products (commercial and open source) have we found helpful. As we look towards public cloud, and/or a bigger role of SaaS development toolchains, how does the approach we have taken in the past scale, or need to adjust? Should we look to create a collective voice towards regulators around continuous automated deployments to production and collaborate with them on how changes in this space should reflect in policies/regulations?

• What controls have we set up on tools made available to developers? Have we had to write administration / provisioning tools to ensure these controls are always established?

• How are we driving development best practices, particularly around DevOps adoption practices? For example, at Morgan Stanley we have 'badges' which display on all of our git repos, pulling metrics from the telemetry gathered across all of the development toolchain. This has driven up adoption of some best practices quite sharply as developers see the feedback, explanations, and documentation right alongside their existing tools. This may be a opportunity for us Open-Source and share the approach.

[kdeman]

Some thoughts I had are the following:

Tooling: Open-Source & commercial: Why not also cover guidance and advisory non-open source tooling/frameworks? Quite often commercial ALM platforms are in place, alongside commercial core app platforms, and connecting them to work in a DevOps way is not always well understood. Examples include Jira, Azure DevOps or Xebialabs on one hand (CI/CD orch tooling) and for instance Murex, Calypso, FIS or Finastra (financial technical solutions/platforms).

People & Process: I would like to make sure the People and Process side of DevOps are not overlooked and equally represented. In my experience tooling is not the main aspect of DevOps, however cultural behavioural change and (people) capability in the broader process context definitely is. ‘a fool with a tool is still a fool’ kind of philosophy. “DevOps is not just a technology imperative but also an organizational imperative”.

Reporting: I would like to also make sure there is a good focus on robust analytics and reporting as main feedback loop mechanism and basis for evidence based decision making. This drives not only higher levels of experimentation and pivoting where needed, but also continuously enhances the above mentioned topics of People and Process and making sure the right DevOps engineering techniques yield the right benefits (and having empirical evidence they do/or do not make things better/aligned with DevOps way of working).

Advisory/Selling: I have always found it very hard to “convince” certain organisations on the benefits and the art of the possible of DevOps way of working, especially in finance organisations. The difficulty is multi-facetted – incumbents’ “not invented here syndrome”, “we like the way things are now” and fear of job loss/”where do I fit in all this” are rife. So a focus on creating advisory/guidance on this “convincing” would be great, targeted at high-mid management.

Partnering with industry leaders: is anyone looking at partnership with industry leading organisations – for example Scrum.org or DevOps institute? This will aid adoption but also allow to tap into the communities of experts that is behind these organisations to drive/build the initiative.

[natishalom]

Agility vs Control

• How can you govern DevOps processes without conflicting with the development agility?

Consistency vs Portability What are the differences between the two and which of them is more achievable to minimize locking?

[mcleo-d]

Thank you @DovOps, @kdeman and @natishalom for your response to the DevOps Mutualization ideas and feedback.

Are you happy if I share the screen and ask you to represent your own feedback on Thursday's call?

James.

[mcleo-d]

@citistefanos and @grovesy - Thanks for joining the first DevOps Mutualization project meeting yesterday. It was great having your experience and opinions on the call. Thank you 👏👏 💯💯💯

[mcleo-d]

All - Thank you for attending the first FINOS DevOps Mutualization formation meeting last Thursday. It was great having so many FINOS members on the call inputting into such an exciting project.

Please find a list of the attendees below as I complete the meeting minutes and add them to the issue. Also, please let me know your GitHub ID in the comments if it's missing from the list.

DevOps Mutualization Formation Meeting Attendees

Date and Time : Thursday 18th June @ 1pm ET / 6pm BST

Name	Firm	GitHub ID
James McLeod	FINOS	@mcleo-d
Alexandra Stratigos	FINOS	@alstratigos
Tosha Ellison	FINOS	@toshaellison
Amol Shukla	Morgan Stanley
Colin Eberhardt	Scott Logic	@ColinEberhardt
Conor O'Neill	NearForm
David Gould	UBS
Duncan Lowie	Credit Suisse
George Kichukov	Citi
Gus Paul	Morgan Stanley
Jamie Jones	GitHub	@jbjonesjr
Karel Deman	Scott Logic	@kdeman
Nati Shalom	Cloudify	@natishalom
Peter Thomas	Deutsche Bank	@peterrhysthomas
Paul Groves	Citi	@grovesy
Robb Keayes	Nomura
Rob Underwood	FINOS	@brooklynrob
Stefanos Piperoglou	Citi	@citistefanos
Tim Johnson	CloudBees	@tcraigjohnson
Marcelo Toro	Morgan Stanley
Vitor Monteiro	GitHub	@bitoiu
Dov Katz	Morgan Stanley	@DovOps

[tcraigjohnson]

Thanks for letting me attend the call. We will also have senior technical leaders on future calls.

I wanted to follow up on something David Gould mentioned late in the session about not being able to deliver a secure pipeline/release process from multiple vendors. What did you mean by that and what prevents you from doing that?

[mcleo-d]

Please find below meeting minutes from the DevOps Mutualization Formation Meeting that took place on Thursday 18th June @ 1pm ET / 6pm BST.

DevOps Mutualization Formation Meeting Minutes

Date and Time : Thursday 18th June @ 1pm ET / 6pm BST

• The meeting is opened by suggesting DevOps is not a source of competitive advantage within finance.

• The purpose of the project is to navigate the vendor ecosystem whilst sharing notes to the group based on areas of common interest and approaches.

• A further objective is to create a common engineering voice to the tech industry and regulator by sharing what works and what doesn't work.

• A recommended best practice for developers is to provide regulatory evidence. This is achieved with approaches like hosting massive data warehouses of telemetry and labelling which is fed back through Chrome extensions. This brings DevOps practices into Git repos.

• Such ideas and approaches have non-competitive advantages and should be shared. Capital One could also leverage and report through systems like Hygieia.

• This would contribute to high quality practices to enable fast deployments to production.

• It was stated it's often the case regulation stipulates software is tested by a QA function that isn't the original development team. This implies manual testing and not delivery pipelines.

• This led to questions like ...

  • How do we test software that utilises persistent, raw fixed sockets rather that HTTP connections?
  • How do we deploy software that uses CI/CD and zero downtime when utilising fixed socket connections?
  • How do we drive alerting that tells a support team that something is wrong using regular trading usage patterns?

• Within finance our problems are not solved by the open source community as they are niche and not big scale systems. It's up to open source project like DevOps Mutualization to provide answers.

• The group suggests we should provide guidance and advisory to non-open source tooling and frameworks as they also exist in the DevOps ecosystem, especially with vendors that integrate core Asset Liability Management (ALM) frameworks.

• It was also noted scrum and agile works well with tooling as cultural changes occur. This should also be recognised and addressed through robust analytics and reporting facilities for the DevOps feedback loop.

• This project also has the chance to convince organisations about the art of the possible as people have fear of change and it can be difficult to convince people in a reassuring way.

• It was suggested that partnering with industry leaders like DevOps Institute could be an advantage as this will widen the project to other large communities.

• The group fed back that process is more important than the tools when it comes to handling regulators in a common industry matter. Questions like, "What are our standard processes, like change control, and how DevOps meets the expectation of that process?"

• It was communicated that automation doesn't change the process, but improves the flow through the process. If the process needs to change, a lot of manual hand overs can be removed from the original process that improves manual audit trails. This then delegates the process to automated tooling.

• It was strongly recommended that aligning to industry defined measures helps with internal adoption, as you're able to say, "we didn't make this up". The argument, "the guys across the street are doing this", does hold a lot of weight.

• It was fed back that machines do things more consistently than people. If you need consistency, then it's best to get a machine to do this.

• With others asking, "Why isn't everyone doing this?", we answer the question about needing a unified industry voice as it's best to set the standards as an industry before someone else sets the standards for us without understanding the problems we need to solve.

• There was a strong backing for setting the standards and working with the tooling vendors by the group that led to a question being raised around GitHub Actions and why GitHub did not adopt the Tekton approach.

• A question about the types of evidence that's required before you can go from code to deploy was also asked to the group.

• Answers like "this is the metadata that's needed, these are the tools that can send to a data lake" need to be provided. Currently teams are having to stitch these answers together rather than having a standard.

• It was asked whether there is a lack of clarity from the regulator on what the standard should be. Is there a gap between theory and practice?

• There was recognition of a potential lack of clarity from regulators down to individual developers with regulator requirements often quoted without providing clarity by the people requesting.

• It was suggested this is an opportunity for FINOS to go to the regulators and join the regulators together with standards provided by the banks. FINOS could provide recommended standards of audit and evidence to the regulators

• It was also noted FINOS wants to be the place where banks and regulators come together to define the required standards.

• There was guidance to the group that RegTech and SDLC regulation are related but addressed differently. They should be treated differently at this point in time by this group. DevOps Mutualization should be separate and pure as defines how things should be done on the ground.

• The group also recognised that as we move towards a standard, there is a chance some tools will be locked in, so we should be consistent with how data from those tools is handled if they are unable to change the way they're built and integrated.

• GitHub responded to the group by saying they are happy to hear the feedback from the group and happy to see progress with these types of activity alongside projects like Cloud Service Certification.

• GitHub suggested next steps could be to identify a couple of regulations, control or processes that are needed to move forward. Especially if there are low hanging fruit?

• GitHub asked for suggestions and needs from people and members involved in the project.

• In response to GitHub, it was strongly suggested financial players need evidence the SDLC is entirely passed and audited. This is core to providing information to regulators.

• It was voiced that SDLC is the most consistent place to start and will contain the basic things that need to be tracked across the whole project group who have built solutions themselves that are combinations of different systems.

• It was communicated the industry doesn't have a data model where you could apply this type of system.

• The data model defines stages in a build, then around signing and storage, then policies can be created based on the success. Integration can happen however needed by banks and teams.

• It was seconded that SDLC is most common across all regulators.

• It was suggested the regulators are not often tech savvy and they have not seen a git repo or a pull request cycle. However, once people are educated and process is shown to be implicitly embedded within current tooling, it is accepted.

• It was then noted that not all tools are built thinking you need to track development activity and log this externally as evidence.

• It was fed back that teams often retrospectively apply auditing to loosely coupled tool chains and the group is not convinced you can purchase a completely compliant tool chain from the market.

The suggested next actions include ...

1. Structuring a conversations around SDLC and sharing what's being done to tackle the problem
2. Iterating the different types of evidence that needs to be produced as part of a GitHub Issue.
3. Creating a viable meeting to continue the conversation started in the project.
4. Tooling - understanding the available tooling and what changes are required to use them to see if there are common changes
5. FINOS will provide the mechanism for continuous conversation as Slack is not regulated and isn't a good place for team members to communicate outside the foundation.

Please feel free to comment below ...

[davidgouldubs]

@tcraigjohnson I was just making the observation that a lot of the popular CI/CD build chain tools on the market have not been designed with compliance at the forefront of their minds, but rather the expedient delivery of code. Trying to retrospectively make these tools operate in a compliant manner is difficult. An issue exacerbated when you put these tools into a loosely coupled tool chain. An example; we've been looking at digitally fingerprinting artifacts with something like grapheas so we can prove the code that's being deployed into production (or has been deployed) is what was intended by the developer and hasn't been tampered with (traceability). We've found it very difficult to make this fingerprinting work across multiple tools in the tool chain without "gaps" where it could be compromised. This leads me to wonder if the answer is now a single tool, from a single vendor for the whole CI/CD pipeline e.g. GitHub Actions, GitLab, etc. where compliance is built in by design, or at least easier to fit retrospectively.

[mindthegab]

@leefaus here's the nascent effort to discuss devOps mutualization. @mcleo-d from our team is leading the charge, feel free to connect directly with him to join.

[leefaus]

@mcleo-d @mindthegab I'm excited to provide any assistance I can IRT DevOps inside the Finance Industry. Armory is currently having some very interesting conversations with some large banks around Policy Driven Deployments. This allows operations teams to create self-service capabilities for their AppDev teams in non-production environments while creating the safety needed for teams to deploy to production once a change review has been completed in an automated fashion. Our focus has been to spend time with companies around the people and process of DevOps to allow for collaboration and transparency rather than the technical bits of building out pipelines or automating tickets.

[mcleo-d]

@leefaus - It's great to be introduced to you. We have our next group meeting on July 30th 2020 @ 6pm BST / 1pm ET with the agenda published on issue #52. I'll contact you via email to pass across additional details.

[mcleo-d]

Issue closed as next meeting occurred here 👉 #52

[tcraigjohnson]

On last week's call, I believe it was Stefanos was talking about the burden of Change Management. They have an astounding number of manual approvals in their process that sound like they are little more than someone ticking a box because Change Management requires it. In a segment that has to move fast, that's a lot of administrative burden.

Here's my questions for the forum:

How do you change Change Management? What evidence and automation would the CMB need to actually improve and streamline the process? Are you changing people to change the process or do you show how to change the process to change the people?

[mcleo-d]

@tcraigjohnson - I have pasted your question under the minutes of our last call here ... https://github.com/finos/community/issues/52#issuecomment-669345026

[tcraigjohnson]

Excellent. Thanks.

On Wed, Aug 5, 2020 at 11:01 AM James McLeod notifications@github.com wrote:

@tcraigjohnson https://github.com/tcraigjohnson - I have pasted your question under the minutes of our last call here ... #52 (comment) https://github.com/finos/community/issues/52#issuecomment-669345026

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/finos/community/issues/44#issuecomment-669345470, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOYFLFKDI7I3I7SAJJUDQSLR7GNAVANCNFSM4NN7PZWA .

[tcraigjohnson]

I am collecting examples of audit requests/demands from various regulatory agencies that the group has to deal with. For example, here is a summary of a Targeted Examination letter from FINRA. https://www.finra.org/rules-guidance/guidance/targeted-exam-letter/high-frequency-trading

Can the group provide links to other examples they regularly encounter? Thanks.