search

canada-ca / architecture

Repository for the storage of architectural modeling
MIT License
44 stars 18 forks source link

Automate Creation of EA Artifacts from GC policy sources #7

Open samperd opened 5 years ago

samperd commented 5 years ago

User Story

As a GC EA Architect I want to generate EA artifacts automatically from authoritative sources So that EA components can be automatically generated, and updated instead of each EA Architect needing to re-model these components

EA Artifact Sources

Background

*It should be possible to generate a new baseline GC EA architecture at the business and motivation level on a daily basis. This is one proposal how

Therefore, it should be possible to generate a new baseline GC EA architecture at the business and motivation level on a daily basis.

Initial Implementation Details

Future upgrades

Future work

SteveSavage commented 5 years ago

Hi Dave, FYI, John and I took a look at auto converting the XML versions in to Archimate elements.
Unfortunately they use XML to denote the structure of the document, not to describe the content. Because I have to read the document to actually create meaningful elements anyway, I use a feature in Sparx EA that lets me import the entire document as a document, and then select text in the document to create elements. The document is updated with hyperlinks to the newly created elements. I think this is a Sparx EA specific feature, but greatly simplifies working with these documents.

I can open the document, read it, select a hyperlinked statement, which loads the linked element (typically a requirement), then I can use Sparx EA traceability window to see all the elements I created that "realize" that requirement.

Once I have everything in I have a script in Sparx EA that loops through all the created elements and updates the notes to provide full context for the statement.

E.g. for the requirement element with the name:
4.4.1.2. Adopting, as applicable, enterprise solutions within their respective departments.

The full context to make it readable (in the notes) is: Directive on the integrated management of service, information, information technology, and cyber security

  1. Requirements 4.4 Leveraging technology in the digital era 4.4.1 Departmental CIO is responsible for: 4.4.1.2. Adopting, as applicable, enterprise solutions within their respective departments.

I could query the DB to get this hierarchy dynamically, but to simplify my reporting templates I add this static information to the notes.

samperd commented 5 years ago

I noticed that there was a fair bit of formatting in there too. I am not yet convinced the XML to Archimate approach is not possible, or could not someday be possible, however I think it will require some collaboration across a few different groups.

I will maintain this pipe dream of using open sources and open data to auto generate an EA model, however I get the Sparx is serving you well.

I hope that some day we have an EA ecosystem that is actually compliant to some of our policy, for example I hope to see Archi become the tools of choice and Archimate model exchange become the default excahnge format and git become the repository of choice. Other options present large barriers to entry, however I understand this is our current siloed world.

I encourage you to share all of your hard work of dissecting documents with the canada-ca/architecture repository so others can re-use these components.

Perhaps we can move closer towards automation as time progresses. I think the perfect use case are all those policies you mentioned that will change. So I assume this means work on your part to update even your scripts.

Cheers

SteveSavage commented 5 years ago

I think an automatic conversion will be possible in the future, if they add contextual information to the policy XML. The best way would be to create the model first to capture everything, then create the document based on the model vs. the other way around.... I've said the same thing for regulations.

Will definitely continue to share my work. Both through my blog and through whatever official repositories the group decides. The Archimate Exchange Format seems to be our best option at the moment for sharing information, but this doens't let us share not Archimate models. So I also plan to share Sparx EA XMI files and generate CSV files: three, one for elements, one for connections, one for diagrams. We may have to use our collection of tools to publish the information in a set of native formats to optimize reuse. E.g. Sparx, Qualiware, etc.

For Sparx, personally I wish we had an enterprise version of it's Reusable Assess Service that I could just link to. https://sparxsystems.com/enterprise_architect_user_guide/14.0/model_repository/reuseable_asset_service.html

For Change management, I don't update. I set the status of the old policy instrument to "deprecated then replaced/retired" and note what it has been replaced by if replaced.

That way we can keep track of anything linked to the old policies that will need to be reviewed for changes. This will also let me explicitly cross link new and old policies to show and comment on specific changes. I plan to write some blog posts about all of this as I have time.

SteveSavage commented 5 years ago

I've been thinking about this some more, and I think I can use Sparx's API to automate this work a bit more.

Pass 1: Import

    • API downloads the XML version for each Policy instrument
    • API saves each XML locally
    • API for each XML, parses the XML
    • API creates a package for the XML named by the Policy Instrument, with a ID of GCPI-{documentID from XML), auto generates and Alias based on the caps (this will need manual tweaking I'm sure)
    • API creates sub-packages based on the chapter titles and appendix titles
    • API creates sub-packages based on the headings within chapters
    • API creates Archimate Requirement elements based on paragraph nodes, these "paragraph" requirements are created within the appropriate chapter, appending or heading package
    • API creates Archimate Requirement elements based on some list nodes (those that are preceded by a paragraph, this filters out lists of links etc.), these requirements elements are created as children of the preceding "paragraph" requirement.
    • API creates Archimate Meaning elements based on dt and dl notes

Pass 2 - Enumerate The API attempts to assign unique numbers to all requirements based on the heiarchy, e.g. 6.1.2 Authorized individuals are informed of the following:

Becomes 6.1.2 Authorized individuals are informed of the following: 6.1.2.1 Expectations for acceptable use of Government of Canada electronic networks and devices per Appendices B and C;

After Phase 1 and 2 are complete for all Policy instruments.

Pass 3: Identify Actors / Roles

  1. The API looks for the phrase "are responsible for"
  2. The API generates a master set of elements for of potential Roles/Actors, default as Role.
  3. A human reviews the list and the elements and converts elements to Actors if needed, or deletes
  4. The API analyzing the requirements from each Policy Instrument, if the Policy Instrument has a requirement that references the Role/Actor followed by the phrase "are responsible for", a copy of the Role/Actor element is made in the same package as the policy instrument. The local copy of the Role/Actor element "realizes" the requirement. The local copy of the Role/Actor element is "generalized" by the master copy of the element.

I'll probably be able to do something similar for Business Objects, and perhaps Business Functions etc. But those might have to be manually created "master" Elements.

Dave, I know you and some others are Phython wizards, perhaps once I get the initial structure in place you can use some of it's AI type features to do some more analysis?

orthosys commented 4 years ago

Hi. We have developed a python-based engine that does exactly that : extracting data from multiple sources and integrating the model update into the EA repository

SteveSavage commented 4 years ago

I've pretty much parked this project due to a career change and language training, I'd love to see an example of what you've managed to extract if you can make it available as an XMI or Archimate Exchange file.