search

IBM / ibmichroot

A set of scripts to facilitate the use of chroot-based containers for IBM i
MIT License
21 stars 9 forks source link

python sym links #1

Closed abmusse closed 9 years ago

abmusse commented 9 years ago

Original report by Aaron Bartell (Bitbucket: aaronbartell, GitHub: aaronbartell).

Working on python chroot and one of the things pip3 binary requires is /QOpenSys/QIBM/ProdData/SC1/zlib/lib/libz.so which is sym linked via /QOpenSys/usr/lib/libz.so.

The symlink is relative:

$  ls -all  /QOpenSys/usr/lib/libz.so
lrwxrwxrwx    1 qsys     0                80 May 21 22:34 /QOpenSys/usr/lib/libz.so -> ../../QIBM/ProdData/SC1/zlib/lib/libz.so

Need to resolve all required symbolic links not only of supporting libs but also binary files. Here's a partial list:

abmusse commented 9 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

We have :ln (match pase root-2-root), :ln_rel (ls -1 relative), and :ln_fix_rel (DYI), can we close this issue now?

abmusse commented 9 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

1) As far as 'blind' (ls -l), if we do anything wlese we will have a version of chroot_OPS_SC1_versionx.x.lst for everything ... not at all desirabble. 2) PowerRuby will have to be treated special, and, we must rely on thier good judgement for taking over symbolic links for PASE related tools (node, opensssl, etc.).

abmusse commented 9 years ago

Original comment by Aaron Bartell (Bitbucket: aaronbartell, GitHub: aaronbartell).

Therefore, i get PowerRuby has it's own openssl (SC1), but it must end up in a different location other than SC1 symbolic links, yes???? ... somewhere in /PowerRuby, yes???

Here's an example concerning the node binary included with PowerRuby:

$ ls -l /QOpenSys/usr/bin/node
lrwxrwxrwx    1 aaron    0                68 May 22 21:18 /QOpenSys/usr/bin/node -> ../../../PowerRuby/prV2R1/bin/node
$ dump -H /QOpenSys/usr/bin/node

/QOpenSys/usr/bin/node:

                        ***Loader Section***
                      Loader Header Information
VERSION#         #SYMtableENT     #RELOCent        LENidSTR
0x00000001       0x0000010c       0x0000af08       0x00000093

#IMPfilID        OFFidSTR         LENstrTBL        OFFstrTBL
0x00000006       0x00084da0       0x00000772       0x00084e33

                        ***Import File Strings***
INDEX  PATH                          BASE                MEMBER
0      /PowerRuby/oss/lib:/usr/lib:/lib             <-------------the important bit
1                                    libc.a              shr.o
2                                    libpthreads.a       shr_xpg5.o
3                                    libstdc++.a         libstdc++.so.6
4                                    libpthreads.a       shr_comm.o
5                                    libgcc_s.a          shr.o

Further, IBM's latest update of 5733OPS includes symbolic links for the node binary, but they are stored in /usr/bin/node and not /QOpenSys/usr/bin/node.

$ ls -all /usr/bin/node
lrwxrwxrwx    1 qsys     0                74 Jul 21 01:15 /usr/bin/node -> /QOpenSys/QIBM/ProdData/Node/bin/node

This is why I was concerned about doing a "blind" resolution of binaries(i.e. ls -l). SSL/OpenSSL is safe for now because PowerRuby seemingly only includes libraries (i.e. /PowerRuby/oss/lib) and not callable programs (i.e. ssh-keygen). I am not sure if this is cause for concern or not.

abmusse commented 9 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Added a new directive :ln_fix_rel for fixing relative links in node js from IBM (you also requested).

#!shell

# Command:
#   > ./chroot_setup.sh chroot_OPS_NODE.lst /QOpenSys/root_path
# Note:
#   This file assume prior chroot_minimal.lst (or uncomment below)
#     > ./chroot_setup.sh chroot_minimal.lst /QOpenSys/root_path
#

#
# copy dirs list
# (
#  mkdir -p $CHROOT_DIR/path
#  cp -R  /path/* $CHROOT_DIR/path/.
# )
#
:cp_dir
/QOpenSys/QIBM/ProdData/Node

#
# fix relative links list
# (
#   cd $CHROOT_DIR/QOpenSys/usr/bin/
#   ln node ../../QIBM/ProdData/Node/bin/node
# )
#
:ln_fix_rel
/QOpenSys/usr/bin/node ../../QIBM/ProdData/Node/bin/node
/QOpenSys/usr/bin/npm  ../../QIBM/ProdData/Node/bin/npm
/QOpenSys/usr/lib/libgcc_s.a  ../../QIBM/ProdData/Node/lib/libgcc_s.a
/QOpenSys/usr/lib/libstdc++.a  ../../QIBM/ProdData/Node/lib/libstdc++.a
abmusse commented 9 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Ok, fixed python chroot links (not fully tested). 

#!shell

# Command:
#   > ./chroot_setup.sh chroot_OPS_PYTHON.lst /QOpenSys/root_path

#
# copy dirs list
# (
#  mkdir -p $CHROOT_DIR/path
#  cp -R  /path/* $CHROOT_DIR/path/.
# )
#
:cp_dir
/QOpenSys/QIBM/ProdData/OPS/Python3.4
/QOpenSys/QIBM/ProdData/OPS/Python-pkgs

#
# relative links list
# (
#   ls -l /QOpenSys/usr/bin/c_rehash
#   /QOpenSys/usr/bin/c_rehash -> ../../QIBM/ProdData/SC1/OpenSSL/openssl-0.9.8j/bin/c_rehash
#   cd $CHROOT_DIR/QOpenSys/usr/bin/
#   ln ../../QIBM/ProdData/SC1/OpenSSL/openssl-0.9.8j/bin/c_rehash c_rehash
# )
#
:ln_rel
/QOpenSys/usr/bin/2to3
/QOpenSys/usr/bin/2to3-3.4
/QOpenSys/usr/bin/easy_install-3.4
/QOpenSys/usr/bin/easy_install3
/QOpenSys/usr/bin/pip3
/QOpenSys/usr/bin/pip3.4
/QOpenSys/usr/bin/pydoc3
/QOpenSys/usr/bin/pydoc3.4
/QOpenSys/usr/bin/python3
/QOpenSys/usr/bin/python3.4
/QOpenSys/usr/bin/pyvenv
/QOpenSys/usr/bin/pyvenv-3.4
/QOpenSys/usr/lib/libpython3.4m.so
abmusse commented 9 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

My only concern is if /QOpenSys/usr/bin|lib has been altered with other symlinks (i.e. PowerRuby ships their own openssl now).

Well, i am thinking with our 'golden rule', where each xxx.lst takes care of itself, chroot_OPS_SC1.lst did the right thing. Therefore, i get PowerRuby has it's own openssl (SC1), but it must end up in a different location other than SC1 symbolic links, yes???? ... somewhere in /PowerRuby, yes??? ... therefore no 'interference' with PASE ("the law"), yes???

abmusse commented 9 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Looks like this when it is running (OUTSIDE chroot) ...

#!shell

bash-4.3$ ./chroot_setup.sh chroot_OPS_SC1.lst /QOpenSys/ranger
:
... copy completed ...
:
cd /QOpenSys/ranger/QOpenSys/usr/bin
ln -sf ../../QIBM/ProdData/SC1/OpenSSL/openssl-0.9.8j/bin/c_rehash c_rehash
:
ln -sf ../../QIBM/ProdData/SC1/OpenSSL/openssl-0.9.8j/lib/libcrypto.a libcrypto.a
cd /QOpenSys/ranger/QOpenSys/usr/lib
ln -sf ../../QIBM/ProdData/SC1/OpenSSL/openssl-0.9.8j/lib/libssl.a libssl.a
cd /QOpenSys/ranger/QOpenSys/usr/lib
ln -sf /QOpenSys/QIBM/ProdData/SC1/zlib/zlib-1.2.3/lib/libz.so libz.so
abmusse commented 9 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Bingo ... it works ...

#!shell

$ which ssh
/QOpenSys/usr/bin/ssh
$ ls -l /QOpenSys/usr/bin/ssh
lrwxrwxrwx    1 adc      0               106 Jul 27 16:10 /QOpenSys/usr/bin/ssh -> ../../QIBM/ProdData/SC1/OpenSSH/openssh-4.7p1/bin/ssh
$ ssh adc@lp0264d
Could not create directory '/QOpenSys/ranger/./home/RANGER//.ssh'.
The authenticity of host 'lp0264d (9.5.158.38)' can't be established.
ECDSA key fingerprint is c9:7d:51:ea:19:52:20:15:d9:9e:5d:57:b7:8d:12:b5.
Are you sure you want to continue connecting (yes/no)? yes
Failed to add the host to the list of known hosts (/QOpenSys/ranger/./home/RANGER/.ssh/known_hosts).
adc@lp0264d's password: 
Welcome to LP0264D
$ exit
Connection to lp0264d closed.

New :ln_rel durective ...

#!shell

# Command:
#   > ./chroot_setup.sh chroot_OPS_SC1.lst /QOpenSys/root_path

#
# copy dirs list
# (
#  mkdir -p $CHROOT_DIR/path
#  cp -R  /path/* $CHROOT_DIR/path/.
# )
#
:cp_dir
/QOpenSys/QIBM/ProdData/SC1

#
# relative links list
# (
#   ls -l /QOpenSys/usr/bin/c_rehash
#   /QOpenSys/usr/bin/c_rehash -> ../../QIBM/ProdData/SC1/OpenSSL/openssl-0.9.8j/bin/c_rehash
#   cd $CHROOT_DIR/QOpenSys/usr/bin/
#   ln ../../QIBM/ProdData/SC1/OpenSSL/openssl-0.9.8j/bin/c_rehash c_rehash
# )
#
:ln_rel
/QOpenSys/usr/bin/c_rehash
/QOpenSys/usr/bin/openssl
/QOpenSys/usr/bin/scp
/QOpenSys/usr/bin/sftp
/QOpenSys/usr/bin/slogin
/QOpenSys/usr/bin/ssh
/QOpenSys/usr/bin/ssh-add
/QOpenSys/usr/bin/ssh-agent
/QOpenSys/usr/bin/ssh-keygen
/QOpenSys/usr/bin/ssh-keyscan
/QOpenSys/usr/lib/libcrypto.a
/QOpenSys/usr/lib/libssl.a
/QOpenSys/usr/lib/libz.so
abmusse commented 9 years ago

Original comment by Aaron Bartell (Bitbucket: aaronbartell, GitHub: aaronbartell).

It's possible ls -l could work. My only concern is if /QOpenSys/usr/bin|lib has been altered with other symlinks (i.e. PowerRuby ships their own openssl now). That's why it's kind of nice to exactly spell out symbolic linkage. Though with that said I've now created a double standard because what's in chroot will be different than non-chroot. ug. Time to go find a bag of sour gummy worms (my vice). [thinking] So the issue would be if you were creating a Python chroot environment and blindly resolved ...SC1/OpenSSL... symlinks, which in turn could create a symlink to /PowerRuby/... and would break once inside chroot because we didn't copy /PowerRuby.

abmusse commented 9 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Something like this should work (maybe) ...

#!shell

#
# relative links list
# (
#   ls -l /QOpenSys/usr/bin/c_rehash
#   /QOpenSys/usr/bin/c_rehash -> ../../QIBM/ProdData/SC1/OpenSSL/openssl-0.9.8j/bin/c_rehash
#   cd $CHROOT_DIR/QOpenSys/usr/bin/
#   ln ../../QIBM/ProdData/SC1/OpenSSL/openssl-0.9.8j/bin/c_rehash c_rehash
# )
#
:ln_rel
/QOpenSys/usr/bin/c_rehash
abmusse commented 9 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Or maybe 'ls -l' is enough all by itself ...

#!shell

bash-4.3$ ls -l /QOpenSys/usr/bin/c_rehash
lrwxrwxrwx    1 qsys     0               118 Jul 11 2014  /QOpenSys/usr/bin/c_rehash -> ../../QIBM/ProdData/SC1/OpenSSL/openssl-0.9.8j/bin/c_rehash
abmusse commented 9 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Alternative: :ln_rel (link relative ../../), therein perform 'relative links' without entering chroot.

I am thinking we can use utility 'find', using only utility 'cd' to move around 

#!shell

bash-4.3$ ls -l /QOpenSys/usr/bin/c_rehash
lrwxrwxrwx    1 qsys     0               118 Jul 11 2014  /QOpenSys/usr/bin/c_rehash -> ../../QIBM/ProdData/SC1/OpenSSL/openssl-0.9.8j/bin/c_rehash

bash-4.3$ cd /QOpenSys/ranger
bash-4.3$ find QOpenSys/QIBM/ProdData/SC1 -name "c_rehash" 
QOpenSys/QIBM/ProdData/SC1/OpenSSL/openssl-0.9.8j/bin/c_rehash
abmusse commented 9 years ago

Original comment by Aaron Bartell (Bitbucket: aaronbartell, GitHub: aaronbartell).

My v7.2 machine:

$ ls -l /QOpenSys/usr/bin/c_rehash
lrwxrwxrwx    1 qsys     0                88 May 21 22:34 /QOpenSys/usr/bin/c_rehash -> ../../QIBM/ProdData/SC1/OpenSSL/bin/c_rehash

My v7.1 machine:

$ ls -l /QOpenSys/usr/bin/c_rehash
lrwxrwxrwx    1 qsys     0               118 Jan 16 2012  /QOpenSys/usr/bin/c_rehash -> ../../QIBM/ProdData/SC1/OpenSSL/openssl-0.9.8j/bin/c_rehas

Wondering if we can use uname (or other) to determine IBM i version.release and then occupy a variable based on that. For example:

:ln
/QOpenSys/usr/bin/c_rehash /QOpenSys/QIBM/ProdData/SC1/OpenSSL$openssl_path/bin/c_rehash

And then in chroot_setup.sh have something like 

if [os_version -eq '72']
  openssl_path = ''
else
  openssl_path = '/openssl-0.9.8j'
fi
abmusse commented 9 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Oh boy, my machine has a openssl version in the symbolic link openssl-0.9.8j (see below). 

#!shell

 > ls -l /QOpenSys/usr/bin/c_rehash
   lrwxrwxrwx    1 qsys     0               118 Aug 22 2011  /QOpenSys/usr/bin/c
   _rehash -> ../../QIBM/ProdData/SC1/OpenSSL/openssl-0.9.8j/bin/c_rehash
   $

Therefore we can not build a 'generic' chroot_OPS_SC1.lst.

#!shell

#
# links list (must occur after minimal copy above)
# ( chroot $CHROOT_DIR /bin/sh -c "ln -sf $1 $2" )
#
:ln
/QOpenSys/usr/bin/c_rehash /QOpenSys/QIBM/ProdData/SC1/OpenSSL/bin/c_rehash
/QOpenSys/usr/bin/openssl /QOpenSys/QIBM/ProdData/SC1/OpenSSL/bin/openssl

Mmm ...

abmusse commented 9 years ago

Original comment by Aaron Bartell (Bitbucket: aaronbartell, GitHub: aaronbartell).

Second, ok, you voted down my idea IBM i customers can handle setting PATH and LIBPATH on thier own to reach things like SC1 functions (*).

I think developers would be best served if we made chroot environments as much like non-chroot as possible right out of the gates. Because if they are using chroot to do automated testing then portions of the testing could give false positives based on differences in chroot vs. non-chroot.

With that said, I do agree that PATH isn't as common knowledge (yet) as *LIBL.

With that said (part II), if we do go down the route of setting PATH and LIBPATH then we should also include .profile configuration so things can be automatically set upon chroot'ing or ssh'ing into a chroot.

abmusse commented 9 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

First, yes, i agree completely chroot_OPS_SC1.lst should take care of itself, including, exporting any symbolic links into the 'public domain'.

Second, ok, you voted down my idea IBM i customers can handle setting PATH and LIBPATH on thier own to reach things like SC1 functions (*).

()BTW -- Cough, cough, gasp, IBM i peachy fine to set LIBL (normal), but fatal flaw in IBM i product set LIBPATH (black hole of dispair) ... so ... cough ... please allow just a little wine and cheese here.

abmusse commented 9 years ago

Original comment by Aaron Bartell (Bitbucket: aaronbartell, GitHub: aaronbartell).

Thinking through this again now that I know about the :file feature. Maybe the Python chroot needs to have the following and then in chroot_OPS_SC1.lst it takes care of any symbolic links associated with itself.

---chroot_OPS_PYTHON.lst---

:file
chroot_OPS_SC1.lst

Specifically, the following would be added to the yet-to-be-created :ln section of chroot_OPS_SC1.lst:

$ ls -la /QOpenSys/usr/bin | grep 'SC1' | awk '{print $0;}'
lrwxrwxrwx    1 qsys     0                88 May 21 22:34 c_rehash -> ../../QIBM/ProdData/SC1/OpenSSL/bin/c_rehash
lrwxrwxrwx    1 qsys     0                86 May 21 22:34 openssl -> ../../QIBM/ProdData/SC1/OpenSSL/bin/openssl
lrwxrwxrwx    1 qsys     0                78 May 21 22:34 scp -> ../../QIBM/ProdData/SC1/OpenSSH/bin/scp
lrwxrwxrwx    1 qsys     0                80 May 21 22:34 sftp -> ../../QIBM/ProdData/SC1/OpenSSH/bin/sftp
lrwxrwxrwx    1 qsys     0                78 May 21 22:34 slogin -> ../../QIBM/ProdData/SC1/OpenSSH/bin/ssh
lrwxrwxrwx    1 qsys     0                78 May 21 22:34 ssh -> ../../QIBM/ProdData/SC1/OpenSSH/bin/ssh
lrwxrwxrwx    1 qsys     0                86 May 21 22:34 ssh-add -> ../../QIBM/ProdData/SC1/OpenSSH/bin/ssh-add
lrwxrwxrwx    1 qsys     0                90 May 21 22:34 ssh-agent -> ../../QIBM/ProdData/SC1/OpenSSH/bin/ssh-agent
lrwxrwxrwx    1 qsys     0                92 May 21 22:34 ssh-keygen -> ../../QIBM/ProdData/SC1/OpenSSH/bin/ssh-keygen
lrwxrwxrwx    1 qsys     0                94 May 21 22:34 ssh-keyscan -> ../../QIBM/ProdData/SC1/OpenSSH/bin/ssh-keyscan
$ ls -la /QOpenSys/usr/lib | grep 'SC1' | awk '{print $0;}'
lrwxrwxrwx    1 qsys     0                94 May 21 22:34 libcrypto.a -> ../../QIBM/ProdData/SC1/OpenSSL/lib/libcrypto.a
lrwxrwxrwx    1 qsys     0                88 May 21 22:34 libssl.a -> ../../QIBM/ProdData/SC1/OpenSSL/lib/libssl.a
lrwxrwxrwx    1 qsys     0                80 May 21 22:34 libz.so -> ../../QIBM/ProdData/SC1/zlib/lib/libz.so

Is that basically what you are getting at? Thoughts?

abmusse commented 9 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

This has a "policy" decision. Issue: if you force 'chase symbolic links' bin/lib in SC1, likely sets precedent to 'chase symbolic links' every thing
choice 1) Force user to change his PATH, LIBPATH ... aka ... want to use SC1, then set you own PATH/LIBPATH Mr. Customer choice 2) Add new method to chroot_etup.sh to 'chase' SC1 symlinks /QOpenSys/QIBM/ProdData/SC1/..bin-or-lib to /QOpenSys/usr/bin-or-lib BTW -- not sure a saying 'chase lib/bin for IBM stuff will work'