IBM i: actually using automated pipelines, builds and git #66

Open worksofliam opened 9 months ago

My team has been working hard on preparing the tools needed to make deploying to IBM i easier. Whether you want to build apps in a dev, test, or production environments, the tools have never been more accessible.

We're going to cover two technologies today:

Source Orbit, and
IBM i CI

These two tools on their own are useful and have their individual use cases, but separately they are a power house for IBM i development. As your teams are moving RPG, CL, DDS, SQL, etc, source code into git, you can extend your development processes with these two tools.

The best part? They are both open-source.

We want to enable our customers to use modern automation tools like GitLab CICD, GitHub Actions, Azure Pipelines, or any other tech - even Jenkins. Also, yes, this can tie into Merlin since you already have a Jenkins instance!

Source Orbit

Repo: https://github.com/IBM/sourceorbit

Source Orbit somehow puts all the complexity of dependency management into one small command line tool. Ever wanted to understand how your objects talk to each other, or what the affected objects might be if you change a single program or table? Source Orbit to the rescue. SO has the ability to help your teams understand not what changes in your source, but how your entire application dependencies look. Perhaps you've been working in a new branch (in git) and want to understand what changing a program might do - well, SO can automatically generate impact reports and share it via email, or perhaps even attach it to a pull request in GitHub or GitLab.

Source Orbit is unique in that it generates all this information from source and doesn't look at any existing objects to create this data. Source Orbit does not understand field level dependencies - only object level.

The cool part about understanding object relationships is that with this information, Source Orbit is able to generate many types of files to be consumed. For example, if you wanted to build your own reports, you might generate a relationship map as JSON to be used in another environment. But, if you wanted to automate the builds of your application, you might generate a makefile or Rules.mk (for ibmi-bob).

So, now we can generate build information - how can we actually deploy it?

Bonus

The Source Orbit CLI actually has built in tools to help migrate your source code into a git-styled project. That means:

using the correct file extensions, and
using posix style include and copy statements in your RPG

You will also be able to do this directly in VS Code as part of the IBM Project Explorer very soon!

Check out the docs for migrating with SO or IBM Project Explorer.

IBM i CI

Repo: https://github.com/IBM/ibmi-ci

In 2023 (and soon 2024) most large businesses are using some kind of enterprise source control management system as part of their lifecycle. Some companies use GitHub, some GitLab, there are tons of options for source code management. The key thing is that most of these services also have integrated CICD environments to execute build pipelines when an action, like a push or PR creation, happens. GitHub has GitHub Actions, Azure has Azure DevOps Pipelines, GitLab has GitLab CICD, and of course there is trusty ol' Jenkins!

This is where IBM i CI comes in. It has always been possible to use SSH from those environments to connect to IBM i, but generally the build scripts have been tedious to say the least. IBM i CI is a command line tool that simplifies connecting to, deploying and building on IBM i. Instead of 3 or more SSH commands, we can use one: ici.

ici allows you specify multiple steps to take when connected to an IBM i to deploy code from your branch, or execute commands, or deploy files, or pull artifacts back down from the system - all in one command.

Combining them together

This is where the blog starts to get exciting. It's really quite simple. Imagine we are creating a new pull-request: we can generate the build scripts to only rebuild the changes using Source Orbit, then we use IBM i CI to deploy the branch plus new build scripts to an IBM i, and then execute the build - all in one pipeline! A sample pipeline to build a branch might look like this:

- name: Generate makefile
  run: so -bf make -l `git diff --name-only origin/main origin/${GITHUB_HEAD_REF}`

- name: Deploy to IBM i
  run: | 
    ici \
      --cmd "mkdir -p './builds/ics_${GITHUB_HEAD_REF}'" \
      --rcwd "./builds/ics_${GITHUB_HEAD_REF}" \
      --push "." \
      --cmd "/QOpenSys/pkgs/bin/gmake LIBL='CMPSYS' BIN_LIB=$(so -bl ${GITHUB_HEAD_REF})"
  env:
    IBMI_HOST: ${{ secrets.IBMI_HOST }}
    IBMI_USER: ${{ secrets.IBMI_USER }}
    IBMI_PASSWORD: ${{ secrets.IBMI_PASSWORD }}
    IBMI_SSH_PORT: ${{ secrets.IBMI_SSH_PORT }}

Let's double the coolness; let's say the PR was tested and was merged into main - well then we should rebuild the changes sources/objects into main automatically using the same steps as above.

Another cool thing about these two types of tools being separate is that you are in full control of how objects are built, where they are built, where the source code gets deployed to, and much more. You have full control over your environment when you use these granular tools.

More

There is more to come. We're in the business of perfecting the tools to make sure they're the best they can be for our IBM i users. We want to enable people to integrate modern DevOps tools with IBM i with ease.

2024 update!!!

Here is an awesome video showing these tools being automated!

This sounds incredibly promising and i'm impressed by the speed you and IBM have put this together. Hope i can check it out very soon; can't wait until Jan 31st https://members.common.org/CommonSite/Events/Event_Display.aspx?EventKey=W23s where i think/hope you'll be showing it, too. But i really want to put a PoC asap, as these gems might have a big impact on decisions my employer is currently making.

Sounds exciting Liam! We're looking at getting into using GIT. It's hard to figure all this out when you've been on the IBM i since forever doing it the old way. If IBM and you are building all this cool stuff, I don't know what, if any, 3rd party tools are necessary. Does this mean we can do everything we need to and don't need anything else from the likes of Arcad, Eradani, Midrange Dynamics, etc.?

@worksofliam Do you ever sleep? LOL Keep up the great work

worksofliam / blog