Integration with CTS

[ed-holland]

Switching the conversation here to not confuse Lars to much. Hope you have time to discuss this topic to come to a working solution.

There will be no 100% solution i agree with that. I'm still new with AbapGit/AbapLint but have 25+ years in the Abap world. Still i would like to streamline our process for Abap Development. Currently we have no real history of what was done for what issue. Also our Abap developer team is small (4 devs) which should make things easier. What makes thing more complicated for us is the need to support older SAP release. We are basing our transports on 700 but they support up to 1909.

To avoid changing SAP owned code, one can only move control to a higher level (like AbapGit). Additionally, the few Badi's can be utilized. AbapGit is currently to more Tool than process (as far as i could figure out so far).

My current setup is very similar to what you have in your picture (GitLab-CE, AbapGit, AbapLint, AbapLint Abap-Backend).

If you are interested to discuss this further, we can do it here or via other channels. my company email is ed.wenmakers@redwood.com (also Teams). German is also no issue if it is easier for you.

[ed-holland]

Our current process is as follows (without AbapGit) and is mainly manual:

1. Jira ticket comes in which requires Abap Development
2. Transport Request is created
3. Change Request is created for the Developer wo is going to work on it
4. Development is done using ADT
5. Change Request is released
6. Testing is done. If failed, then goto 3
7. Transport Request is released
8. Exported Transport is automatically installed on 3 different target SAP releases
9. If import fails then goto 2

[fabianlupa]

If you are interested to discuss this further, we can do it here or via other channels.

I think it's fine discussing it in the open as others may join in.

Regarding the process you listed I have a few questions:

• What is a "Change Request" in this context? Do you mean a transport task as a child of a transport request or are you referring to SolMan/ChaRM change requests?
• The testing is done only on the development system? Are there no transports to integration systems before the change is completed?

Regardless of the details I think your options are as follows:

1. Create or adapt solutions to integrate CTS with Git-based processes, i. e. pull requests and branching, utilizing and possibly changing abapGit and related software around it (for example this repository).
2. Use existing solutions that might make your change management and CI processes more automated that do not rely on abapGit -> using Solution Manager + (Central) ATC
3. Use existing solutions that use git but not abapGit -> gCTS (?)
4. Use third party solutions (if I recall correctly ActiveControl was supposed to get some kind of abapGit integration)

Which basically is the same range as always between "Do it yourself the way you want it without support and with lots of work" and "Use the existing tested proprietary thing with vendor support that mostly does what you want with maybe less work".

[larshp]

I would like to write down some options for abapGit, just need to focus some time, right now I'm deep into some abaplint stuff

need more spare time

[ed-holland]

To the Questions:

In General, i'm referring to classical TMS/CTS (no Charm/Projects if i can avoid it).

• When you create a new "Workbench Request" while in ADT, it creates a "Transport Request" and inside a "Change Request" which will contain the changes you make in ADT. SAP has been inconsistent with their naming over the years...
• We do our QA in the target Release systems (700, 702, 740, 1610). ATC we only use in the development system.

The problem with using solutions from SAP is that the new one's require high releases to work from (Central ATC 750, gCTS 1909). Solman is dreadfull to setup and maintain (we have joint development in this area with SAP). Transporting from these newer releases back to 700 is problematic. For this reason #2 and #3 is not good for us.

I was not aware of ActiveControl. Any expeciances here?

My preference would be #1 as a lot of good things are already available.

[fabianlupa]

Ah okay, we are talking about classic tasks then (I've never seen them called change request before).

I have no experience with ActiveControl, only know about it because of an Inside Track.

If you want to do 1., as you said the tools are already available. The endeavor is to link it all together in a way that is intuitive for developers (and comprehensible for internal and external IT audits). I kind of stopped there because my team was reduced from 2 people to just me, which reduces the complexity by several magnitudes. I am still using GitLab with branches per transport and merge requests with automatic CI (abaplint + ABAP Unit using sapcli) and nightly CI runs on QAS systems (abaplint + ABAP Unit + ATC). Though the link between TMS and GitLab is still manual (branch needs to be manually named with the transport number for CI to recognize it and in abapGit you need to manually only stage the objects that are included in the transport; also the assumption is that you release the transport after merging the merge request and transport it to QAS as the master branch will assume that the QAS system already has all the changes that were merged into master).

[ed-holland]

That is already a great start... lately abap teams don't tend to be big any how but that allows us to "Keep it Simple". The central start point should be AbapGit as we have more control here.

What about these changes to remove some manual activities:

1. Extend the Configuration UI with CTS settings
   • Checkbox: Create Transport Request when creating a Branch
   • Checkbox: Release Change Request when commiting to Branch
   • Checkbox: Release Transport Request when staging a Branch
2. Extend the "Create Branch" functionality to create a "Transport Request" first and use it's name to create the branch. Descriptions can be added at they might be important for references like a JIRA Ticket. The UI could also contain users input for which Tasks ("Change Requests") are created, defaulting to the user that creates the branch
3. Extend the "Commit" functionality to also releases the Task (first non released Task of the user in the transport request=branch)
4. Extend the "Stage" functionality to also release the Transport Request so that test imports are done automatically in QA systems

Some enforcements to avoid problems should be enabled (how?):

• Object changed in another branch/transport request
• No direct changes in Master

[ed-holland]

@flaiker could you share your pipelines with us?

[fabianlupa]

@flaiker could you share your pipelines with us?

Sure (probably tomorrow).

In my concept I had the following assumptions (also see #1):

• Transport = Branch
• Task -> not mapped
• Commits are just normal git commits to update the state of the transport to the branch and trigger CI. I also like to document changes using commit messages.
• master branch = state of the QAS system (with delay as merged branches need to have their transports imported)
• feature branch = state of master + changes in a transport
• Parallel feature branches would have to be rebased once another one is merged

I create the transport first in ADT as I am doing the first change and then later create the branch in abapGit and commit, instead of the other way around. I would probably still do this if abapGit had some kind of workflow for it, partly because refreshing the main repo I work on in abapGit takes ~30 seconds.

[ed-holland]

Looks like we are quite close in the way of process.

• feature branch = state of master + changes in a transport What is the use of the feature branch?

What i did not mention is the releases that we create. They are currently done via CTS (automated) but i would like to move the process to AbapGit/Pipeline. They would become tagged Branches.

[fabianlupa]

The feature branch is the branch created for a transport to be able to do code review and CI in GitLab using merge requests.

Yes, releases is something I wanted to do too but haven't had the time for (create a release using tags in GitLab of the current state in master, this triggers the CI pipeline for tags, this calls a webservice on the QAS system to generate a TOC of the whole package, TOC is returned by the webservice and attached to the release in GitLab and published on an artifact server). This is not a must have thing for me as the target production system is in the same transport route (after QAS) in an inhouse development scenario but would be nice for external use cases or sandboxes.

This is the pipeline for a nightly build triggered by cron in GitLab (the trigger stage is just for user docs generated by GitLab pages to be run independent of the build pipeline):

Pipeline for normal, non-nightly, master-builds (any build steps using the ABAP system are turned off as there is a delay between releasing a merge request and importing the transport into QAS):

Pipeline for merge requests:

Status on the merge request (parsed junit output of abaplint and sapcli):

I can also share the .gitlab-ci.yml files but will have to anonymize them quite a lot.

There are some caveats I ran into designing these pipelines:

• For security reasons the docker containers running a build step cannot have any internet connectivity and must have IP whitelists to only be able to communicate with the QAS and DEV SAP systems and the GitLab server (sorry I don't trust NPM not pulling some dependency which just zips all proprietary code in the build folder and uploads it to some server...). Therefore all build dependencies are build / tested / checked locally and beforehand and published as a trusted docker image in the GitLab docker container registry.
• I use a hand built patch file for sapcli as some things need to be adjusted for it to work on 7.40.
• abaplint needs a configuration file with the rules to be checked. I don't want to maintain and upgrade the same configuration file accross a dozen repositories. Therefore I put it in the docker image for abaplint (two versions actually, one per namespace as the rule configuration is partly namespace specific, one base file and either the z-namespace or /.../-namespace file are merged using jq). In the repos themselves there is another abaplint configuration just containing the repository specific rule exceptions, which are also merged with the merged base file to generate the final config file used in the build step.
• For sapcli the configuration is outsourced into a separate repository to be able to include it and inherit build steps from it without defining tons of variables everywhere.
• I haven't quite found a solution for the ATC checks. These are currently just allowed to fail. I don't have someone else to use the exemptions workflow and also exemptions would be maintainted in the system while abaplint exceptions would be maintained in the repo which doesn't make much sense. Also the exemptions would have to be transported or maintained in a central ATC instance which I don't have.
• The pipeline for a merge request / a branch should only check the state of the repository in that branch. The system dependant build steps will use the state in the DEV system which might deviate if you have paralllel development. That's why the ABAP Unit step only checks the unit tests in objects that are contained in the transport. For ATC there is currently not way to only check objects in a transport using sapcli therefore it's just turned off. abaplint is of course using the correct basis for its checks.
• Cross-repository-transports: You might have transports which have objects that belong to different repositories. You could prevent this by implementing the BAdI or using different transport layers (I have not). These will lead to you having two merge requests in two different repositories and only if both are merged the transport should be released. Some of the build steps will report wrong results though (the ABAP Unit check should only report the tests belonging to the repo and not all of them for example). You run into this quite frequently if you have a common-basis-objects repository / package.

[larshp]

abaplint exceptions: I'm trying to add configuration options so exemptions style things are not needed, but it is difficult and will always be work in progress

[larshp]

@flaiker I don't remember, do you do product development on a SAP system, or ongoing changes to a productive system running business processes? Or release/milestone based?

[fabianlupa]

I have an unusual mixed scenario: Normal three system customer landscape with ongoing changes to the software used in production but this software is developed as an add-on and supposed to be deliverable to external parties. No milestones or release cycles.

Edit: not using the AAK