Document a recommended release process for our SOAR apps

[edthedev]

Context

Our preference is to have an audit log of each release, and to allow members of incident response and engineering to trigger those releases directly. Now that the technical nuts and bolts of our release process are ironed out, I would like us to revisit the actual mechanism for triggering a release.

I'm very okay with just porting the process we used on the command line automation; and I am also open to other ideas.

Key outcomes are:

• engineering and incident response team members have access to perform a release or rollback
• the who, when, and what release details are logged somewhere that engineers, developers and incident responders all have access to review

Tasks

• [x] discuss
• [x] create a pull request with updated functionality in one app (we did the Midpoint SOAR app)
• [x] draft an updated release guide in this repository wiki space (or just copy over the relevant details from the previous release and rollback guide) - See https://github.com/techservicesillinois/splunk-soar-template/wiki/Release-and-Rollback-Procedures
• [x] document release process
• [x] document rollback process (using the drop-down)
• [x] document verification process (observing the build day/time in the code file)

[edthedev]

Takeaways from discussion this afternoon:

• Currently tagging pushes to production.
• SecDev manager needs production release to require at least one action by an engineer.
• Rollback is simply changing the dropdown box in SOAR back to the previous version.
• We are unsure why deployments do not land in 'unconfigured apps'? We would probably prefer for them to arrive there, allowing engineers to upgrade via the SOAR interface
• For our fully open products, getting our app published into the SOAR marketplace would address the above concerns.
• We do not anticipate our Midpoint connector being accepted since our Midpoint password scramble API is not open source
• In a pinch, we could continue to use the GitHub 'Releases' functionality. The need to release in GitHub and Roll back in SOAR would cause confusion. Rollback via GitHub releases would also likely work.

[edthedev]

I sent our question about deploying without enabling immediately to our colleague at Splunk, and I posted it to the Splunk soar_app_dev Slack channel.

[edthedev]

Per discussion this morning:

• For most apps, the Splunk app store (shared GitHub repo) is probably our preferred method, long term.
• The Midpoint app can't go in the app store, since the API we use upstream is closed-source, but it's not a huge deal for it to be the one-off.

Plan

• For the Midpoint app, we can re-use the release process from SecOps tools.
  • Verification becomes a guided process in SOAR (instructions and screenshots)
  • Rollback is updated to simply be switching the dropdown back.
• For other apps, we work toward publishing in the SOAR store.
  • Install is through the normal SOAR app update process.
  • Rollback is the same.
  • Use the Midpoint pattern in null router and TDX until published to Splunk SOAR.

[edthedev]

Update: We updated our SOAR midpoint app with the new pattern.

Here's the deployments:

Does not deploy:

• Pushing a new commit to a branch does not deploy.
• Merging to main does not deploy.
• Pushing a tag to main does not deploy.

Does deploy:

• Pushing a commit to an open pull request deploys a test copy of our app to our SOAR cloud instance
• Releasing via the GitHub 'release' mechanism deploys a production copy of our app to our SOAR cloud instance.

Here's the CI/CD snippet:

    - name: Build Package (Prod)
      if: github.event_name == 'release'
      run: |
        make build
    - name: Build Package (Test)
      if: github.event_name == 'pull_request'
      run: |
        make build-test
    - name: Store build as an artifact
      uses: actions/upload-artifact@v3
      with:
        name: deployed tar file
        path: app.tar
        retention-days: 7
    - name: Post to Splunk SOAR API
      if: github.event_name == 'release' || github.event_name == 'pull_request'
      run: |
        make deploy
      env:
        SOAR_TOKEN: ${{ secrets.SOAR_TOKEN }}
        SOAR_HOSTNAME: ${{ secrets.SOAR_HOSTNAME }}

[edthedev]

Done: https://github.com/techservicesillinois/splunk-soar-template/wiki/Release-and-Rollback-Procedures