search

cyberark / conjur-oss-suite-release

Under development - Latest stable releases of the Conjur OSS suite
Apache License 2.0
7 stars 2 forks source link

Spike on auto-generating an initial release changelog #2

Closed izgeri closed 4 years ago

izgeri commented 4 years ago

Phase One: Initial Release

Given a list of GitHub repositories and corresponding tagged versions (you can use the list from here to start) And a VERSION file Then a CHANGELOG can be auto-generated that includes links to all of the GitHub releases for the set of tagged versions

Mainly with this spike we are determining the initial structure of the suite release repo. Please refer to the draft design doc for more info about the basics of what will be required here.

sgnn7 commented 4 years ago

@doodlesbykumbi Circled around the following as a spike direction:

sgnn7 commented 4 years ago

CC: @hughsaunders let us know if you have any comments on this ^

izgeri commented 4 years ago

Is parse-a-changelog a piece of this? We built that to enforce the keepachangelog format consistent - I wonder if as part of this we'll want to advise all repos to have this check in their pipelines

sgnn7 commented 4 years ago

@izgeri yeah, currently it's the only consistent part of majority of our repos and will probably involve making sure they all follow that format.

doodlesbykumbi commented 4 years ago

Some thoughts re:

Trimming the changelog to only include a particular version.

Given

It might be necessary to filter entries from the CHANGELOG. Here are some options:

  1. Build a markdown parser that goes in an extracts CHANGELOGs on a per version basis.
  2. Revamp the approach to generating CHANGELOG so that the source of truth is something easier to parse like JSON/YAML. This will be used to add the entries to CHANGELOG.md. This is similar to how reno does it.
  3. Use the release description in Github releases. This will likely need some parsing of markdown since the release description could contain more than the CHANGELOG for that particular release. It also means coupling of the concept of Github releases and possible options in the suite, which isn't necessarily a bad thing.
izgeri commented 4 years ago

@sgnn7 @doodlesbykumbi can you please share here an update on how you will define that this spike is "done" and whether/what follow-on spikes will be needed? thank you!!

sgnn7 commented 4 years ago

So from my perspective the spike should end up with a viable initial PoC:

Version X.Y.Z

Added:

Changed

Removed

...

sgnn7 commented 4 years ago

Items left to be done:

Questions left to consider:

CC: @izgeri @hughsaunders @diverdane @BradleyBoutcher for feedback/thoughts/opinions

izgeri commented 4 years ago

Fetching changelogs not just for version x of component but also include any previous one going back to the last release

an important question, but I think this can be left to the next spike #4

Figuring out how to version this "last release". Do we do it in Git with tags? Do we maintain branches of a major suite release? Do we have folders with all the release's suite definitions? Do we use GitHub releases for this?

I was thinking the OSS suite release repo would use git tags / github releases. What are the benefits of maintaining branches? In my proposal, we only bump OSS suite major version when Conjur OSS major version is bumped - so can this question be deferred? What do you mean by "release suite definitions"?

Do we add the capability to diff suite versions more than one version apart somehow (eg. generate changelog from suite v0.0.1 to v.1.2.3)?

If you use git tags and keep a CHANGELOG.md in the repo (only roughly in keepachangelog format - each version's entries would be the auto-gen'd change log for the version) then you could use git tag diff to see what changed between two versions

How do we include release notes and upgrade instructions/changes for each component in the changelog since it's outside of the keepachangelog spec?

My proposal is that we get this from the github releases, and we require all repos to provide gh releases with Notable changes, Known issues, and Upgrade instructions headers. But if you're pulling changelog from CHANGELOG.md and not github releases, it's TBD how you can pull this info out.

What will be the suite's "release" pipeline? Will it result in a GH release? A generated archive that we then make a release out of?

a good question. I do think it should result in a gh release, but I'm not sure about automating it because we don't do that in other repos (though we could). we could have the master branch Jenkins pipeline save artifacts when the commit corresponds to a tag.

izgeri commented 4 years ago

Noting here as I've done on slack - I'm not sure we can have the final release notes organized by change type (added, changed, fixed, etc) - my sense from feedback is that users will want this organized by component

I also think that organizing by component may make it easier to compile suite release notes from component release notes, since we're just concatenating with headers at that point. That would enable us to require that individual components format their github release notes like this and then the suite release notes might look like:

[1.1.0] 2020-03-29

TOC

[INSERT TOC HERE]

CHANGE LOG

Changes to cyberark/conjur since last release

cyberark/conjur@v1.3.1

[INSERT RELEASE NOTES FROM THIS VERSION]

cyberark/conjur@v1.3.0

[INSERT RELEASE NOTES FROM THIS VERSION]

Changes to cyberark/secretless-broker since last release

cyberark/secretless-broker@v1.5.1

[INSERT RELEASE NOTES FROM THIS VERSION] ...

Note also from @alexkalish:

users will likely only care about a subset of components, so I think they might want to consume changes by component. Also, Rails does it by component and I’ve always found it easy to consume.

Also, frankly, some components are more important than others. Changes to Conjur are a bigger deal than say, the Java SDK.

I don't think it's a requirement of this spike that the end result has the release notes organized by component, but my sense is that this is the direction we should head. That being said, I'd be glad to hear counterarguments / reasons why we should organize it as proposed here - please feel free to share if you'd like.

sgnn7 commented 4 years ago

What are the benefits of maintaining branches?

They can maintain a history of changes (if needed) to a specific OSS release. We may sometimes have problems/errors/typos in a suite release that may need updates and a static tag will not be conducive for maintenance.

In my proposal, we only bump OSS suite major version when Conjur OSS major version is bumped - so can this question be deferred?

I somewhat doubt it. Suite release may not correlate to Conjur release versioning even if the suite release is triggered by a bump in conjur version.

What do you mean by "release suite definitions"?

Something like:

/
- release/
  - v1.0.0/
    - repositories.yml
  - v1.1.0/
    - repositories.yml

or

/
- release/
  - conjur-suite_1.0.0.yml
  - conjur-suite_1.1.0.yml
...

If you use git tags and keep a CHANGELOG.md in the repo (only roughly in keepachangelog format - each version's entries would be the auto-gen'd change log for the version) then you could use git tag diff to see what changed between two versions

A git diff will be gnarly I think. Sometimes git will consider the additions as changes to the original release CHANGELOG and would make the output unreadable but this is just a theory.

My proposal is that we get this from the github releases, and we require all repos to provide gh releases with Notable changes, Known issues, and Upgrade instructions headers. But if you're pulling changelog from CHANGELOG.md and not github releases, it's TBD how you can pull this info out.

The extraction isn't a big question but more about how do we include it in the resulting CHANGELOG. Do we even use CHANGELOG for it or maybe we use a different output file?

we could have the master branch Jenkins pipeline save artifacts when the commit corresponds to a tag.

I'd consider this as "not feasible" though - for now this is just the changelog but as we include the actual artifacts from components, this may be untenable and also has an additional issue of not being publicly visible (maybe that's ok?).

my sense from feedback is that users will want this organized by component

While I do think this makes sense for some things, as a "full suite" changelog, this would make more sense to me grouped as the type of change since the end user persona is interested what changed between the suite versions, not individual components. I think additionally, the volume of text would be harder to parse if divided by component individually with also a (somewhat marginal) problem that the resulting changelog would not fit a keepachangelog format too. I feel like this is one of those "you should see how both options look" things because it's hard to see the difference without having them side-by-side.

doodlesbykumbi commented 4 years ago

... resulting changelog would not fit a keepachangelog format too

Good point! Something I hadn't considered before. I agree with your general argument for not going with per component CHANGELOGs.

izgeri commented 4 years ago

@doodlesbykumbi @sgnn7 how do you feel about the changelog in gh being in the per-type format but the published release notes on doc.conjur.org being organized per component?

diverdane commented 4 years ago

What we could do for CHANGELOG.md is something similar to what Kubernetes does. Look here: Kubernetes Repo They maintain a running master CHANGELOG.md file that has links to individual CHANGELOG.x.y.md files. What we should do differently than what they do:

I lean towards preferring to group all of the Added's together, and all of the Changed's together, etc, so that viewers have a quick summary of all the things added in the suite, all things changed in the suite, etc... but I see advantages to grouping by component, so no strong preference. (Viewers could always go to the component repo to see just that component's Added/Changed/Removed).

As for GH releases for our OSS suite, I think we should create releases for our suite that indirectly link to GH releases for our individual repos/components. Each repo would be responsible for maintaining GH releases with upgrade/migration/binaries info for its releases, and maybe a pointer to the CHANGELOG.x.y.md file in that repo?

For your questions:

Fetching changelogs not just for version x of component but also include any previous one going back to the last release

Agree with @izgeri , maybe a future nice-to-have?

Figuring out how to version this "last release". Do we do it in Git with tags? Do we maintain branches of a major suite release? Do we have folders with all the release's suite definitions? Do we use GitHub releases for this?

I think it's a combination of CHANGELOGS + GH releases (with tags, since that's required for a GH release, IIRC). CHANGELOGS would be a compilation/aggregation of CHANGELOGS from our project repos, and maybe the suite GH release would have links to component GH releases.

Do we keep the generated CHANGELOGs for a release in the repo itself? Do we just make a GH release and leave it there?

My preference would be the K8s approach, with an overall CHANGELOG.md and versioned CHANGELOG.x.y.md's in the repo.

Do we add the capability to diff suite versions more than one version apart somehow (eg. generate changelog from suite v0.0.1 to v.1.2.3)?

I think having CHANGELOG deltas from one version to the next would be sufficient, IMO.

How do we include release notes and upgrade instructions/changes for each component in the changelog since it's outside of the keepachangelog spec?

Have a suite GH release that points to individual component GH releases?

What will be the suite's "release" pipeline? Will it result in a GH release? A generated archive that we then make a release out of?

A suite GH release, updated CHANGELOG.md, and CHANGELOG.x.y.md files?

alexkalish commented 4 years ago

Wow, such good discussion here! I don't have strong feelings about most of the questions, but would say it's important to have an aggregate CHANGELOG committed, as well as present in the GH release. Also, I would vote for all versions of the change log using the same structure.

As I think I said in Slack, I would also suggest looking at Rails: https://github.com/rails/rails. They have a similar set of challenges: multiple individually versioned components shipped/versioned together. They have the benefit of all/most code in one repo, but segregate them into separate directories with their own changelogs and then aggregate them for the release.

Would love to see a final write up after most questions have been answered. Thanks!

izgeri commented 4 years ago

Re: CHANGELOG format

it's still an open question how to integrate notable updates, upgrade instructions, and known issues into release notes

the suite release CHANGELOG is auto-generated from the CHANGELOGs of individual components it was proposed that the additional version info (notable updates, upgrade instructions, and known issues) was put in gh releases - so our suite release automation would additionally have to parse gh release descriptions to get this info