ar: Device busy - libiconv.a

abmusse commented 9 years ago

Original report by Aaron Bartell (Bitbucket: aaronbartell, GitHub: aaronbartell).

Running into an issue with the libiconv.a fix while in a chroot. Specifically, this is me ssh'd into a chroot environment vs. using the chroot command from the prompt.

Error:

$ ./pkg_setup.sh fix                                          
fixing /opt/freeware/lib/libiconv.a ...
ar: Device busy
ar: 0707-113 The fopen system call failed on file /opt/freeware/lib/libiconv.a.
ar: Device busy
ar: 0707-113 The fopen system call failed on file /opt/freeware/lib/libiconv.a.

It is erroring on the following two lines from pkg_setup.sh:

ar -rv /opt/freeware/lib/libiconv.a shr4.o
ar -rv /opt/freeware/lib/libiconv.a shr.o

My guess is libiconv.a is in use by my shell (zsh). In AIX/Linux land I can use fuser to learn the culprit. I searched and couldn't find a similar utility for PASE.

Ideas on how to learn what is locking `libiconv.a`?

abmusse commented 9 years ago

Original comment by Aaron Bartell (Bitbucket: aaronbartell, GitHub: aaronbartell).

Issue resolved with this commit.

abmusse commented 9 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Go ahead with git code push.

Why not bomb? PASE kernel and IFS ref counting active use give the appearance of 'removed file' libiconv.a being both dead and alive (Schrödinger's cat). Not magic, ref count means 'storage/memory' of old libiconv.a has not been deleted under running process (aka, dead cat). However, any NEW process started, will get the new version of libiconv.a (live cat). You can see this same effect with running ILE programs that are deleted/replaced, the old processes keep running the old ILE program (dead cat), and, new processes use the new ILE program (live cat). BTW -- ILE and/or PASE live/dead cat, is exactly why we tell everybody to re-start demons, or, kill xToolkit jobs, etc. (remove references still "in use").

abmusse commented 9 years ago

Original comment by Aaron Bartell (Bitbucket: aaronbartell, GitHub: aaronbartell).

Your code did fix the issue. I was surprised because I would have thought a lock would extend to any action against libiconv.a (ar or rm). Tucking this tidbit away for future knowledge.

My guess is that this issue won't be unique to zsh and we would probably do well to make the baby (slightly) ugly (implement the change). I can do it to save you time, just need a thumbs up from you.

Question if you have time: When zsh uses libiconv.a does it load a separate copy into memory and that is why/how zsh didn't bomb when rm was done?

abmusse commented 9 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Always a way ... but ... baby starts to get homely.

How to make an ugly baby ... IFS ref counts active/open files, which includes PASE opened shared libraries ( /opt/freeware/lib/libiconv.a ), therefore, without hurting running processes (your zsh), we may ...

#!shell

function package_fix_libiconv {
  name=$1
  case "$name" in
    *libiconv*)
      if (($system_OS400==1))
      then
        echo "fixing /opt/freeware/lib/libiconv.a ..."
        ar -x /QOpenSys/usr/lib/libiconv.a
        # Argh Matie, we be needin' to make a ugly baby to fool the IFS Kraken
        cp /opt/freeware/lib/libiconv.a /opt/freeware/lib/libiconv-lady-in-waiting.a
        ar -rv /opt/freeware/lib/libiconv-lady-in-waiting.a shr4.o
        ar -rv /opt/freeware/lib/libiconv-lady-in-waiting.a shr.o
        rm /opt/freeware/lib/libiconv.a
        cp /opt/freeware/lib/libiconv-lady-in-waiting.a /opt/freeware/lib/libiconv.a
      else
        echo "not IBM i, ignoring /opt/freeware/lib/libiconv.a ..."
      fi
    ;;
  esac
}

abmusse commented 9 years ago

Original comment by Aaron Bartell (Bitbucket: aaronbartell, GitHub: aaronbartell).

Looks like zsh has libiconv.a locked. I tried entering a different shell (bash) from my zsh shell to see if that would release the lock and it didn't (other shell job still running).

Is our only option to require ibmichroot to use certain shells? Other ideas?

$ ps                      
    PID    TTY  TIME CMD
 535203  pts/2  0:00 -zsh 
 535209  pts/2  0:00 ps

$ dbx -d 100 -a 535203    
Waiting to attach to process 535203 ...
Successfully attached to /opt/freeware/bin/zsh.
Type 'help' for help.
reading symbolic information ...warning: no source compiled with -g

stopped in sigsuspend at 0x20020714
0x20020714 (sigsuspend+0x1f4) 80410014         lwz   r2,0x14(r1)
(dbx) map
Entry 1:
   Object name: /opt/freeware/bin/zsh
   Text origin:     0x10000000
   Text length:     0xaad25
   Data origin:     0x30000818
   Data length:     0x19250
   File descriptor: 0x3

Entry 2:
   Object name: /opt/freeware/lib/zsh/5.0.7/zsh/compctl.so
   Text origin:     0xd1c3e000
   Text length:     0xee51
   Data origin:     0x30058488
   Data length:     0x11b4
   File descriptor: 0x4

Entry 3:
   Object name: /opt/freeware/lib/zsh/5.0.7/zsh/complete.so
   Text origin:     0xd1cff000
   Text length:     0x1ee04
   Data origin:     0x300557b0
   Data length:     0x1cfc
   File descriptor: 0x5

Entry 4:
   Object name: /opt/freeware/lib/zsh/5.0.7/zsh/zle.so
   Text origin:     0xd20bd000
   Text length:     0x3bcc0
   Data origin:     0x3003bc18
   Data length:     0x8338
   File descriptor: 0x6

Entry 5:
   Object name: /opt/freeware/lib/zsh/5.0.7/zsh/newuser.so
   Text origin:     0xd1bc9000
   Text length:     0x829
   Data origin:     0x3003a550
   Data length:     0xd8
   File descriptor: 0x7

Entry 6:
   Object name: /usr/lib/libcrypt.a
   Member name: shr.o
   Text origin:     0xd1bc8100
   Text length:     0x9be
   Data origin:     0x204cb6a8
   Data length:     0x120
   File descriptor: 0x8

Entry 7:
   Object name: /opt/freeware/lib/libiconv.a      <----------------
   Member name: libiconv.so.2
   Text origin:     0xd1dc0100
   Text length:     0xfba63
   Data origin:     0x204e1658
   Data length:     0xdcea4
   File descriptor: 0x9

Entry 8:
   Object name: /usr/lib/libcurses.a
   Member name: shr42.o
   Text origin:     0xd1d73100
   Text length:     0x4c0f8
   Data origin:     0x204cc9d0
   Data length:     0x14246
   File descriptor: 0xa

Entry 9:
   Object name: /opt/freeware/lib/libc.a
   Member name: shr.o
   Text origin:     0x20000000
   Text length:     0x3f3191
   Data origin:     0x203f4350
   Data length:     0xd6810
   File descriptor: 0xb

abmusse commented 9 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

In this case, at least one other 'PASE process' running on the machine has 'locked' /opt/freeware/libiconv.a, that is, most likely using the shared library services to run in a job.

User ... i do not know of any 'user level' tool to discover information about the kernel. However, dbx -d 100 -a pid-of-job will attach to another process, upon wich the dbx> map command can be run to see the 'load map', which will show what is being used / loaded.

Geek ... i normally use strsst macro tiale. Direct access to kernel information.

IBM / ibmichroot