search

ituglib / ituglib-build

Build scripts for ITUGLIB pipelines
MIT License
2 stars 0 forks source link

Support distributing PAK archives #1

Closed rsbeckerca closed 4 years ago

rsbeckerca commented 4 years ago

The PAK archive format, by default, sets the user to be JENKINS and "OOOO". This should be changed to another user and "NUNU" security.

rsbeckerca commented 4 years ago

The current script matches what the tar.gz builds do: sets the owner to SUPER and security to "NUNU", which is SUPER's default. The problem I'm having is that FUP SECURE and FUP GIVE are failing when run from Jenkins in a groovy container. @xidbhonaker Do you have any wisdom or preference here?

xidbhonaker commented 4 years ago

What is different from tar.gz? Do they fail silently or generate some kind of error?

Can you make it work with chmod and chgrp?

Jojo-Schmitz commented 4 years ago

chmod and chgrp don't work on Guardian files. Hmm, chown and chgrp might actually (but mind that they are case sensitive and require a privileged user, AKA root or SUPER.SUPER on OSS), but chmod surely won't.

rsbeckerca commented 4 years ago

chmod and chgrp don't work on Guardian files. Hmm, chown and chgrp might actually (but mind that they are case sensitive and require a privileged user, AKA root or SUPER.SUPER on OSS), but chmod surely won't.

Yes, obviously. I am trying to launch FUP multiple times. This seems to be messing up groovy or Java. The FUP GIVE and FUP SECURE operations are failing inside the groovy container while running successfully outside.

rsbeckerca commented 4 years ago

What is different from tar.gz? Do they fail silently or generate some kind of error? Can you make it work with chmod and chgrp?

Using shell scripts vs. gtacl seems to be part of this. I think there is an issue in the JVM with multiple Process objects. I cannot make chmod, chown, or chgrp work on GUARDIAN files.

xidbhonaker commented 4 years ago

Can you create a text file as input, run FUP pointing to it, and then delete them?

What is being done differently than for tar.gz?

rsbeckerca commented 4 years ago

Here is what the actual PAK job during distribution does (which seems to be a success). Check the content and values:

File Mode BACKUP Program - T9074H01 (19OCT2016) (AGX)
(C)Copyright 2015 Hewlett Packard Enterprise Development LP
Drives: (\HPITUG.$Z8GW)
System: \HPITUG  Operating System: J06  Tape Version: 3
Backup options:  NO AUDITED, BLOCKSIZE 8, NO IGNORE, NO OPEN, PARTONLY OFF,
                INDEXES IMPLICIT
*WARNING-7147*  Files created and stored via OSS and SQL/MX objects are not
                supported.
*WARNING-7033*  This tape can only be restored with TNS/II RESTORE (B41, C00 or
                later).
Backup time:  8Oct2020  9:20                                          Page: 1

Tape: 1       Code             EOF      Last modif   Owner RWEP   Type   Rec Bl

$SYSTEM.SYSTEM
 GMAKE         800          1416712  6Oct2020 15:47    -1   NUNU

Summary Information

Files dumped = 1  Files not dumped = 0
Total bytes:      1420164   
Compressed bytes: 579512
xidbhonaker commented 4 years ago

So, trying to understand, the key difference is the use of groovy?

rsbeckerca commented 4 years ago

Reopening for comments.

rsbeckerca commented 4 years ago

I cannot get FUP to run from within groovy/Jenkins. It probably needs /IN [#MYTERM]/. The question is whether the results of the current scripts are acceptable. If yes, then we can just move on. I don't think debugging groovy/Jenkins is part of this issue.

rsbeckerca commented 4 years ago

Is super and nunu fine for distribution, as in the pak listall? If so, we can fine tune the scripts next week.

xidbhonaker commented 4 years ago

I think it’s working, assuming your goal is to deliver the PAK fie (gmake.pak.bin) such that it contains:

I admit to not having touched anything groovy. But if groovy uses TACL to start FUP, then you’re right. Don’t know what the ‘current’ IN FILE defaults to.

xidbhonaker commented 4 years ago

Since users can (and should?) override all of that (using VOL and MYID, and optionally TAPEDATE), not sure that it’s very critical.

I don’t think the PAK files we deliver in the legacy Tech lib are super.super owned… but I haven’t looked in like a decade.

rsbeckerca commented 4 years ago

The current distributions use SUPER.SUPER as the owner of tar.gz files.

xidbhonaker commented 4 years ago

I just checked one of the legacy files (/web/legacy/PakFiles/advent.pak.bin). The contents of that are owned by user 50,255 (which isn’t a real user) and secured NCNC. And it was backed up from $TUGLIB.ADVENT. Pretty sure it was created on the old K1000 that XID used for ITUGLIB prior to 2010.

I really don’t think it matters, though, as users can (and probably should) override ownership and location at UNPAK time.

rsbeckerca commented 4 years ago

I just checked one of the legacy files (/web/legacy/PakFiles/advent.pak.bin). The contents of that are owned by user 50,255 (which isn’t a real user) and secured NCNC. And it was backed up from $TUGLIB.ADVENT. Pretty sure it was created on the old K1000 that XID used for ITUGLIB prior to 2010.

Modified the GMAKE README to add TAPEDATE and describe the default owner and security.

rsbeckerca commented 4 years ago

I was able to run FUP SECURE for "NCNC" using gtacl -p instead of gtacl -c. So if we want to change the other to something other than SUPER.SUPER, it seems possible. Let me know.

xidbhonaker commented 4 years ago

I think in the end it doesn’t matter, much, but I think backing up from $system.system might raise curiosity as to our development environment. Seems it should come from somewhere like $data01.gmakeobj.gmake or something of the sort.

rsbeckerca commented 4 years ago

I think in the end it doesn’t matter, much, but I think backing up from $system.system might raise curiosity as to our development environment. Seems it should come from somewhere like $data01.gmakeobj.gmake or something of the sort.

I'm using the VOL $SYSTEM.SYSTEM argument on PAK, so it just looks like it's coming from $SYSTEM.SYSTEM. It's really from $DATA01.JENKTEMP.

xidbhonaker commented 4 years ago

Got it. So it doesn’t matter what we put there… Then I have opinions only, not a preferences. I’ll support anything you prefer.

rsbeckerca commented 4 years ago

Reopening to continue the discussion. Note that comments have been edited to remove extraneous email wrappers and threads.

rsbeckerca commented 4 years ago

Before we close this, please see #2 review comments regarding where to set the attributes. The volume/subvolume set in previous commit 8a6e39b5f9da1de469786cb5b30fdd9f534eb558. For reference, the line references are:

rsbeckerca commented 4 years ago

So, trying to understand, the key difference is the use of groovy?

The root cause appears to be that terminal handling is different when Process.start() runs gtacl vs. sh. The supplied standard files appear wrong when gtacl is run directly, but is fine when wrapped with a shell script, so I used that.

xidbhonaker commented 4 years ago

I’ve read those and understand them.