AIX playbook should not put `/opt/freeware/bin` at start of /etc/environment

[sxa]

Ref: https://github.com/AdoptOpenJDK/openjdk-infrastructure/blob/master/ansible/playbooks/aix.yml#L735

This can break certain default AIX startup scripts which require the AIX grep command.

We need to determine the impact to the adoptopenjdk scripts of changing this.

@Haroon-Khel Possibly somethign we can do while switching to the invididual Role-based playbooks?

[aixtools]

The tactic I have seen most, over the years, is to either add PATH at the beginning of a script (as opposed to making something a system default), OR having variables defined to execute a specific command, e.g. GREP=/opt/freeware/bin/grep and then calling it using ${GREP}.

And after glancing at the playbook - I see you are 'installing', i.e., attempting to install, a dated version of openSSL (1.0.2.1601) while the version 1.0.2.2100 is already installed.

FYI:

p.s. - while I do not know what version yum.sh is demanding version 1601 may, or may not be sufficient.

p.p.s.: $ openssl version OpenSSL 1.0.2u 20 Dec 2019

[sxa]

@aixtools I think we need to be clear on why that PATH statement was there in the first case really (Doesn't appear to be commented in the playbook...) I suspect we should undo it on one of the machines and see what breaks :-)

[Haroon-Khel]

@sxa Do we know which AIX start up scripts fail as a result of this? And is the issue here that the PATH variable is simply at the start of /etc/environment, or that its there at all?

[aixtools]

One thing it breaks is rc.nfs. I wanted to do some AIX maintenaince (using NFS) - and fell over this. It takes too much time to find and document all the things that break because the syntax of 'coreutils' and/or 'binutils' is different than the AIX versions.

So, it is not that it is in the PATH, but that it is first. Note: if a nodejs command is requiring a coreutils/binutils command syntax - and it is not providing the PATH - then that will break.

In short, the 'bad' is that the systems are unmanageable. Updates cannot be applied (at least not on the system I first saw this, and it had not changed the PATH - it had overwritten /usr/bin/grep.

Hope this helps, Michael (aka aixtools)

On 28/09/2020 16:33, Haroon Khel wrote:

@sxa https://github.com/sxa Do we know which AIX start up scripts fail as a result of this? And is the issue here that the PATH variable is simply at the start of |/etc/environment|, or that its there at all?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/1544#issuecomment-700046572, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACSZR5OVWW3WFLRIOQYSGN3SICNDTANCNFSM4RNFA6DQ.

[aixtools]

Smarter now: I see where it happens in aix.yml

      ######################################
      # Add bash to available login shells #
      ######################################
        - name: Add bash to available login shells
          replace:
            path: /etc/security/login.cfg
            regexp: 'shells = '
            replace: 'shells = /bin/bash,'
          tags: login_shell

        - name: Add bash to available login shells
          blockinfile:
            dest: /etc/shells
            block: |
              /bin/bash
          tags: login_shell

        - name: Set variables for global environment
          blockinfile:
            dest: /etc/environment
            block: |
              AIXTHREAD_HRT=true
              PKG_CONFIG_PATH=/opt/freeware/lib64/pkgconfig:/opt/freeware/lib/pkgconfig
              PERL5LIB=/opt/freemarker/lib/perl5
          tags: login_shell

        - name: Add freeware and xlc to PATH for global environment
          replace:
            path: /etc/environment
            regexp: 'PATH=/usr/bin'
            replace: 'PATH=/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin'
          tags: login_shell

As I mention in #1581 there are some assumptions here.

• Classically, tools such as compilers - and java - are appended to the PATH, not pre-pended.
• This code does not check if the admins had already placed xlC (must it be C++, or does xlc (no C++) suffice?
• This code combines several, I would consider not related, under the heading "bash is included in login_shells". Again, it does not check if bash is already included, it just prepends it.

[Haroon-Khel]

Related pr https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/2057

[Haroon-Khel]

ref https://github.com/adoptium/infrastructure/pull/2288

[sxa]

Related pr #2057

Superceded by https://github.com/adoptium/infrastructure/pull/2288

[aixtools]

OK - currently, they are almost all the same - with:

==============
root@nim.bak:[/home/root]dsh-adopt.ksh "echo \$PATH"
adopt01:
/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java6/jre/bin:/usr/java6/bin
==============
adopt02:
/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java6/jre/bin:/usr/java6/bin
==============
adopt03:
/usr/bin:/opt/IBM/xlC/13.1.3/bin:/opt/freeware/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin:/opt/bin
==============
adopt04:
/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin
==============
adopt05:
/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java6/jre/bin:/usr/java6/bin
==============
adopt06:
/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java6/jre/bin:/usr/java6/bin:/opt/bin
==============
adopt07:
/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java6/jre/bin:/usr/java6/bin:/opt/bin
==============
adopt08:
/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin:/opt/bin

However, going to change the default to have xlC at the end, and modify jenkins user so that PATH is modified during ssh login.

[sxa]

Quick note on this - if /usr/bin is put ahead, then the GNU make is not picked up correctly with the current setup, so that would need to be resolved before we can put this change in.

[aixtools]

Maybe adding an alias for make will be a quick fix (jenklins .profile), or we 'mangle' the current /usr/bin/make to be a symbolic link to GNU make.

[aixtools]

I am thinking there are two ways to handle this:

• PREFIX $PATH on jenkins user with /opt/freeware/bin
• Add an alias in /etc/profile for make so that it uses /opt/freeware/bin/make rather than /usr/bin/make.

[sxa]

I think most of the invocations of make already have some switches to invoke something else on Solaris. It's quite likely we could do the same on AIX ... If you search the temurin-build and aqa-tests repositories for gmake you can probably find most (maybe all?) of the places it would need to be overridden to /opt/freeware/bin/make on AIX.

[aixtools]

ok. when I am not enjoying a sunny day. ie, let the rain come (the yard is begging for rain!)

[aixtools]

I think most of the invocations of make already have some switches to invoke something else on Solaris. It's quite likely we could do the same on AIX ... If you search the temurin-build and aqa-tests repositories for gmake you can probably find most (maybe all?) of the places it would need to be overridden to /opt/freeware/bin/make on AIX.

Based on this: created the PR: https://github.com/adoptium/temurin-build/pull/3441

This looks to be all that is needed.

And then we can remove the addition of /opt/freeware/bin from PATH.

[aixtools]

As https://github.com/adoptium/temurin-build/pull/3441 has been merged, want to work on a PR cleaning up PATH, but waiting for #3139 to be merged. I hate to rebase stuff.

[aixtools]

I've been playing around with options - and, imho - the best path will include:

• PR to remove the assignment of /opt/freeware/bin (and xlc13 paths)

• manual fix of the PATH variable on the current systems - rather than use playbook editing to remove it.

• Once done, basically, the playbook takes a handsoff attitude to global PATH (in /etc/environment and/or /etc/profile). If an admin wants something different - that is their call. And if it breaks java build or runtime - that is their responsibility.

• So, discussion first (on above points) - and then I'll take the required steps (including making sure the PATH is set via jenkins parameters/profile).

[aixtools]

As a reminder: the current setup breaks the PATH for system administrators - and amazingly, despite the recent refresh of all systems using the playbooks - not all systems display the same behavior:

root@osunim:[/root]dsh-adopt "df -g /tmp"
adopt01:
df: invalid option -- 'g'
Try 'df --help' for more information.
==============
adopt02:
df: invalid option -- 'g'
Try 'df --help' for more information.
==============
adopt03:
df: invalid option -- 'g'
Try 'df --help' for more information.
==============
adopt04:
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd3           4.00      3.40   16%      252     1% /tmp
==============
adopt05:
df: invalid option -- 'g'
Try 'df --help' for more information.
==============
adopt06:
df: invalid option -- 'g'
Try 'df --help' for more information.
==============
adopt07:
df: invalid option -- 'g'
Try 'df --help' for more information.
==============
adopt08:
df: invalid option -- 'g'
Try 'df --help' for more information.
==============
adopt10:
df: invalid option -- 'g'
Try 'df --help' for more information.
==============

[sxa]

Thanks for pushing this forward. So I guess adopt04 (test-osuosl-aix72-ppc64-2) is in the state you want at the moment, so we should run a build of a couple of versions (probably 8 and 20) to ensure that jenkins and the build scripts work ok there and then roll this out.

[aixtools]

adopt04 comes close - but I would ALSO remove the xlc13 compiler related path variable. adopt03 has duplicity issues.

root@osunim:[/root]dsh-adopt /usr/bin/egrep ^PATH /etc/environment
adopt01:
PATH=/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin:/opt/bin
==============
adopt02:
PATH=/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin:/opt/bin
==============
adopt03:
PATH=/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/opt/IBM/xlC/13.1.3/bin:/opt/freeware/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin:/opt/bin
==============
adopt04:
PATH=/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin
==============
adopt05:
PATH=/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin:/opt/bin
==============
adopt06:
PATH=/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin:/opt/bin
==============
adopt07:
PATH=/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin:/opt/bin
==============
adopt08:
PATH=/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin:/opt/bin
==============
adopt10:
PATH=/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin:/opt/bin
==============

[aixtools]

Thanks for pushing this forward. So I guess adopt04 (test-osuosl-aix72-ppc64-2) is in the state you want at the moment, so we should run a build of a couple of versions (probably 8 and 20) to ensure that jenkins and the build scripts work ok there and then roll this out.

And - how did that work out?

[aixtools]

• I'm guessing it works out (on adopt04) - since it has not been changed.
• Looking at two systems - build-1 and build-2 - I see that build-1 has the PATH set as a parameter for jenkins, but build-2 does not - so noone has been updating the jenkins settings (and as long as the global PATH prefixes GNU applications (ie, /opt/freeware/bin) we can never know if the jenkins parameters are doing the job.
• That said - on adopt04 jenkins it is set: https://ci.adoptium.net/computer/test%2Dosuosl%2Daix72%2Dppc64%2D2/configure - so I hope that is conclusive enough to make this acceptable.
• This cannot, really, be resolved via a PR. ie, once accepted that /opt/bin/freeware should not be in the PATH - a PR change can be done (I recommend a new issue to remove a lot of noise from here) and remove the ansible step that includes it.
• And, that someone closes this when jenkins has been updated to include PATH for all agents.

[sxa]

Thanks for pushing this forward. So I guess adopt04 (test-osuosl-aix72-ppc64-2) is in the state you want at the moment, so we should run a build of a couple of versions (probably 8 and 20) to ensure that jenkins and the build scripts work ok there and then roll this out.

And - how did that work out?

I don't think it's been explicitly done yet ... It's just a matter of kicking off the JDK8u and JDK21u jobs so here they are (Both with ENABLE_TESTS: false since they shouldn't be too important for this, and NODE_LABEL set to test-osuosl-aix72-ppc64-2

• https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-aix-ppc64-temurin/405/
• https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-aix-ppc64-temurin/22/

[sxa]

Noting that the jdk8u build failed, but it doesn't look related to the PATH. Re-running at https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-aix-ppc64-temurin/411/console

[sxa]

Needs further investigation although I can't spend any time on it. The build in the last comment hit the same issue. Maybe it's perl related:

11:18:46  mk-ca-bundle.pl generates 144 certificates
11:18:46  Subject: CN=GlobalSign_Root_CA,OU=Root_CA,O=GlobalSign_nv-sa,C=BE
11:18:46  Generated alias: CN=GlobalSign_Root_CA,OU=Root_CA,O=GlobalSign_nv-sa,C=BE
11:18:46  Processing certificate with alias: CN=GlobalSign_Root_CA,OU=Root_CA,O=GlobalSign_nv-sa,C=BE
11:18:47  keytool error: java.lang.Exception: Input not an X.509 certificate

[aixtools]

It's been awhile. I forgot what the problem was - and how it got solved.