search

IBM / ibmichroot

A set of scripts to facilitate the use of chroot-based containers for IBM i
MIT License
21 stars 9 forks source link

Copy IBM-provided Python 2.7, Node 4, and Git into chroot environment #40

Closed abmusse closed 7 years ago

abmusse commented 8 years ago

Original report by David Russo (Bitbucket: DavidRusso, GitHub: DavidRusso).

Is there any plan to provide scripts for this? I created some here:

https://bitbucket.org/DavidRusso/ibmichroot/overview

I've tested on 7.3 with the latest PTFs and they seem to do the trick. I have yet to test on 7.2 and 7.1.

The only problem I've found so far is that the /QOpenSys/QIBM/ProdData/OPS/Node4/bin/nodever.sh script doesn't work out of the box in the minimal chroot environment, due to the hashbang. The script in the Node 0.12.x install directory has the same issue.

If there is any interest in including these, I can open a pull request. I'm also happy to make adjustments if there is a different plan on how to include these.

abmusse commented 7 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

I think we are done with this issue.

abmusse commented 8 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Also updated all chroot_gen_OPS_xxxx.lst in the project (from my machine anyway).

abmusse commented 8 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Ok, fixed generator ... fix links gen_chroot_OPS_lst chroot_gen_OPS_xxxx.lst

abmusse commented 8 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Ops. The chroot_gen_OPS_xxx.lst files are not working correctly yet with symbolic links. Sorry i will look at the issue.

abmusse commented 8 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

gen_chroot_OPS_lst

Cool. We will have to decide if we like the outcome, but it appears to work. However not well tested (warning).

yum dir

Oops, sorry. We are joint development ibmichroot scripts. I did not mean to keep you in dark. However, we/lab are discussing if we will actually 'release' in PTF, so yum is all experimental at this point. Perhaps you can try it out and see what you think.

abmusse commented 8 years ago

Original comment by Aaron Bartell (Bitbucket: aaronbartell, GitHub: aaronbartell).

I was able to successfully run your utility. #humanunittesting

I noticed the /yum directory was removed from the repo. Did you mean to do that?

% gen_chroot_OPS_lst

---Directory ... /QOpenSys/QIBM/ProdData/OPS/.---
Found ... .qptfinf.QPY2.dat
Reading ... .qptfinf.QPY2.dat
Processing ... .qptfinf.QPY2.dat
Output file for .qptfinf.QPY2.dat is chroot_gen_OPS_Python2.7.lst
Writing ... ./chroot_gen_OPS_Python2.7.lst
Found ... .qptfinf.QPYTHON.dat
Reading ... .qptfinf.QPYTHON.dat
Processing ... .qptfinf.QPYTHON.dat
Output file for .qptfinf.QPYTHON.dat is chroot_gen_OPS_Python3.4.lst
Writing ... ./chroot_gen_OPS_Python3.4.lst
Found ... .qptfinf.QCI.dat
Reading ... .qptfinf.QCI.dat
Processing ... .qptfinf.QCI.dat
Output file for .qptfinf.QCI.dat is chroot_gen_OPS_cloudinit.lst
Writing ... ./chroot_gen_OPS_cloudinit.lst
Found ... .qptfinf.QPYDB.dat
Reading ... .qptfinf.QPYDB.dat
Processing ... .qptfinf.QPYDB.dat
Output file for .qptfinf.QPYDB.dat is chroot_gen_OPS_Python-pkgs.lst
Writing ... ./chroot_gen_OPS_Python-pkgs.lst
Found ... .qptfinf.QPYTLKT.dat
Reading ... .qptfinf.QPYTLKT.dat
Processing ... .qptfinf.QPYTLKT.dat
Output file for .qptfinf.QPYTLKT.dat is chroot_gen_OPS_Python-pkgs.lst
Writing ... ./chroot_gen_OPS_Python-pkgs.lst
Found ... .qptfinf.QPYFLIP.dat
Reading ... .qptfinf.QPYFLIP.dat
Processing ... .qptfinf.QPYFLIP.dat
Output file for .qptfinf.QPYFLIP.dat is chroot_gen_OPS_Python-pkgs.lst
Writing ... ./chroot_gen_OPS_Python-pkgs.lst
Found ... .qptfinf.QPYBTL.dat
Reading ... .qptfinf.QPYBTL.dat
Processing ... .qptfinf.QPYBTL.dat
Output file for .qptfinf.QPYBTL.dat is chroot_gen_OPS_Python-pkgs.lst
Writing ... ./chroot_gen_OPS_Python-pkgs.lst
Found ... .qptfinf.QPYDB2.dat
Reading ... .qptfinf.QPYDB2.dat
Processing ... .qptfinf.QPYDB2.dat
Output file for .qptfinf.QPYDB2.dat is chroot_gen_OPS_Python-pkgs.lst
Writing ... ./chroot_gen_OPS_Python-pkgs.lst
Found ... .qptfinf.QPYTLKT2.dat
Reading ... .qptfinf.QPYTLKT2.dat
Processing ... .qptfinf.QPYTLKT2.dat
Output file for .qptfinf.QPYTLKT2.dat is chroot_gen_OPS_Python-pkgs.lst
Writing ... ./chroot_gen_OPS_Python-pkgs.lst
Found ... .qptfinf.QNODE4.dat
Reading ... .qptfinf.QNODE4.dat
Processing ... .qptfinf.QNODE4.dat
Output file for .qptfinf.QNODE4.dat is chroot_gen_OPS_Node4.lst
Writing ... ./chroot_gen_OPS_Node4.lst
Found ... .qptfinf.QLIBGCC482.dat
Reading ... .qptfinf.QLIBGCC482.dat
Processing ... .qptfinf.QLIBGCC482.dat
Output file for .qptfinf.QLIBGCC482.dat is chroot_gen_OPS_tools.lst
Writing ... ./chroot_gen_OPS_tools.lst
Found ... .qptfinf.QGIT.dat
Reading ... .qptfinf.QGIT.dat
Processing ... .qptfinf.QGIT.dat
Output file for .qptfinf.QGIT.dat is chroot_gen_OPS_tools.lst
Writing ... ./chroot_gen_OPS_tools.lst
Found ... .qptfinf.QICONV.dat
Reading ... .qptfinf.QICONV.dat
Processing ... .qptfinf.QICONV.dat
Output file for .qptfinf.QICONV.dat is chroot_gen_OPS_tools.lst
Writing ... ./chroot_gen_OPS_tools.lst
Found ... .qptfinf.QBASH.dat
Reading ... .qptfinf.QBASH.dat
Processing ... .qptfinf.QBASH.dat
Output file for .qptfinf.QBASH.dat is chroot_gen_OPS_tools.lst
Writing ... ./chroot_gen_OPS_tools.lst
Found ... .qptfinf.QUNZIP.dat
Reading ... .qptfinf.QUNZIP.dat
Processing ... .qptfinf.QUNZIP.dat
Output file for .qptfinf.QUNZIP.dat is chroot_gen_OPS_tools.lst
Writing ... ./chroot_gen_OPS_tools.lst
Found ... .qptfinf.QZIP.dat
Reading ... .qptfinf.QZIP.dat
Processing ... .qptfinf.QZIP.dat
Output file for .qptfinf.QZIP.dat is chroot_gen_OPS_tools.lst
Writing ... ./chroot_gen_OPS_tools.lst
Found ... .qptfinf.QGCC.dat
Reading ... .qptfinf.QGCC.dat
Processing ... .qptfinf.QGCC.dat
Output file for .qptfinf.QGCC.dat is chroot_gen_OPS_GCC.lst
Writing ... ./chroot_gen_OPS_GCC.lst
Found ... .qptfinf.QORION.dat
Reading ... .qptfinf.QORION.dat
Processing ... .qptfinf.QORION.dat
Output file for .qptfinf.QORION.dat is chroot_gen_OPS_Orion.lst
Writing ... ./chroot_gen_OPS_Orion.lst
abmusse commented 8 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Ok i added a new utility to generate chroot_gen_OPS_xxx.lst. I also included generated files from my machine (PTFs on).

#!shell

$ ./gen_chroot_OPS_lst 

---Directory ... /QOpenSys/QIBM/ProdData/OPS/.---
Found ... .qptfinf.QPYTHON.dat
Reading ... .qptfinf.QPYTHON.dat
Processing ... .qptfinf.QPYTHON.dat
Output file for .qptfinf.QPYTHON.dat is chroot_gen_OPS_Python3.4.lst
Writing ... ./chroot_gen_OPS_Python3.4.lst

... and so on ...
abmusse commented 8 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Update --

Try 1 ... appears we will have to use a slightly different method. I added keyword 'manifest', but 'pure shell' fork/exec in chroot_setup.sh was too slow (way too slow).

Try 2 ... we will use program to generate new chroot_OPS_xxxx.lst files based on the manifest. If this works we can automate generating new files to be pushed into this git project. Also, when we fall behind latest PTFs, customer can gen the new chroot_OPS_xxxx.lst files based on the manifest, aka, less work for David Russo. (Theory anyway).

abmusse commented 8 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

I think i agree ...

I don't want to write an entire yum/rpm meets IBM i PTF design document in this issue. Anyway, noted, we need to handle this while we test out our yum part of the project (*).

Back to topic.

Again, I think we both agree we could do a better job of chroot_OPS_things.lst using a 'manifest' keyword. This fits chroot_setup.sh anyway, that is, set-up a new chroot. Although, i also find, i am adding to a existing chroot using chroot_setup.sh. Anyway, I believe many of the tasks beyond chroot_minimal.lst, are probably better served with yum/rpms (when/if we get that running).

(*) -- OTHER PEOPLE reading -- yum part of project is not updated yet, so do not bother downloading. To date, we only put a small thing from aix to see if we could get it to work at all. To really get it working, you need the other patches and tools we are working on.

abmusse commented 8 years ago

Original comment by Aaron Bartell (Bitbucket: aaronbartell, GitHub: aaronbartell).

It may be a flagrant foul in rpm land, but it's how some languages do their updates...

$ ruby -v
ruby 2.1.5p273 (2014-11-13 revision 48405) [powerpc-aix6.1]

Note the p273 in the above. It allows developers to know of minuscule changes. Node.js from IBM has separate incremental updates because there are Power specific fixes (SI59729). So even though I can do the below on IBM i, Mac, Linux, and Windows, it means something potentially entirely different on IBM i because of PTFs. If we don't store separate versions of Node.js based on the aforementioned PTF then we wouldn't know whether the performance issue was fixed without going through manual effort. I vote the effort is done once by us and then everyone benefits.

$ node -v
v4.4.3
abmusse commented 8 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

BTW -- i am not giving up. We may find a rpm way here.

abmusse commented 8 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

We'll need to think through ... PTF fix that doesn't bump the Node.js version.

This is known as a flagrant foul in RPM land. Aka, technical foul, where player must be removed from the game. Boss must dock pay, season ending, no millions for you. Wait .. i may have wondered off topic.

Anyway, i am not sure delta RPMs can save us here ... so think through is indeed something to ponder. Maybe these are stuff added with shell script (oh yuck).

abmusse commented 8 years ago

Original comment by Aaron Bartell (Bitbucket: aaronbartell, GitHub: aaronbartell).

I agree the last PTF wins (makes sense).

I also agree with the hosted rpms being the way we can get the multiple versions working. We'll need to think through that because Node.js v4.4.3 might be released and then there is a PTF fix that doesn't bump the Node.js version. So we most likely need to append the PTF id to the end of the yum package so people can operate on the PTF-id-versions independently. (hopefully I explained that well).

abmusse commented 8 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

I believe they manifest comes/goes with PTF install or remove. That is, if you have the PTF, then you have the manifest, because used in both the PTF install and remove process. The file has a dot prefix to 'hide' file from wandering admin minstrels (cough), so it should stay put with the PTF. So, summary, if you have PTF, you have the 'manifest'.

Next question (unasked) ... chroot friendly ... PTFs on PTFs for each option/product in OPS?

Nope. That is, last PTF wins, and, overlays manifest. This is nature of installing into root of the machine. Frankly, personally, we-chroot-people will better off if we can get yum/rpm working to allow you to keep 'versions' of RPMs around (*).

(*) These view are mine alone, not IBM. I am not a legal representative of IBM. We can make the technology work, but i do not understand the legal aspects of RPM distribution/hosting (TBD).

abmusse commented 8 years ago

Original comment by Aaron Bartell (Bitbucket: aaronbartell, GitHub: aaronbartell).

I like that a lot. I didn't realize those files existed. Do they already remain there, or are there some GO PTF commands an admin might take that would clean them out?

abmusse commented 8 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Thinking out loud guys ...

We maybe could do better OPS CHROOT items. That is to say, maybe we should allow a new directive 'manifest' in the .lst files. Seems to me that tool like chroot_setup.sh would be able to read the PTF manifest and copy all the right stuff into a chroot 'hole'.

#!shell

 > cd /QOpenSys/QIBM/ProdData/OPS
   $
 > head .qptfinf.QNODE4.dat
   /QOpenSys/QIBM/ProdData/OPS/Node4/bin/npm
   /QOpenSys/QIBM/ProdData/OPS/Node4/notices.txt
   /QOpenSys/QIBM/ProdData/OPS/Node4/bin/node
   /QOpenSys/QIBM/ProdData/OPS/Node4/bin/nodever.sh
   /QOpenSys/QIBM/ProdData/OPS/Node4/lib/node_modules/npm/.npmignore
   /QOpenSys/QIBM/ProdData/OPS/Node4/lib/node_modules/npm/.mailmap
   /QOpenSys/QIBM/ProdData/OPS/Node4/lib/node_modules/npm/.travis.yml
   /QOpenSys/QIBM/ProdData/OPS/Node4/lib/node_modules/npm/AUTHORS
   /QOpenSys/QIBM/ProdData/OPS/Node4/lib/node_modules/npm/CHANGELOG.md
   /QOpenSys/QIBM/ProdData/OPS/Node4/lib/node_modules/npm/CONTRIBUTING.md
   $

Of course the file is in EBCDIC for PTF tools, but, still, has all the files and symbolic links listed. After solving that issue, seem like a easier design for the chroot_OPSx.lst items (not much work).

What do you think???

abmusse commented 8 years ago

Original comment by Aaron Bartell (Bitbucket: aaronbartell, GitHub: aaronbartell).

@DavidRusso ! Glad to hear from you.

Yes, please issue a pull request. The only changed I'd ask you to make is to rename the existing chroot_OPS_PYTHON.lst file to be chroot_OPS_PYTHON3.4.lst. Concerning how planning works with this project... we've added features mostly based on real-world scenarios (i.e. if we found a need we made changes and put it in the repo).

The only problem I've found so far is that the /QOpenSys/QIBM/ProdData/OPS/Node4/bin/nodever.sh script doesn't work out of the box in the minimal chroot environment, due to the hashbang. The script in the Node 0.12.x install directory has the same issue.

If you have the time it would be great to [submit an RFE(Request For Enhancement)](Click here) for this so the fix is made upstream.