IBM / ibmi-oss-issues

Important resources for anyone interested in open source on IBM i
Creative Commons Zero v1.0 Universal
13 stars 0 forks source link

Yum install / Bootstrap hanging #46

Closed CSTARLING75 closed 10 months ago

CSTARLING75 commented 11 months ago

OS 7.4 /QOpenSys/var lib didn't exist, created with mkdir in Putty. Attempted to use Access Client Solutions > Tools > Open Source Package Management to install. Installing window hung, not messages, not bootstrap.log file created (although bootstrap.sh and bootstrap.tar.Z files were created in /tmp). Relaunched but received danger msg reflecting bootstrap was successful.

Then attempted to manually install via putty with command: touch -C 819 /tmp/bootstrap.log; /QOpenSys/usr/bin/ksh /tmp/bootstrap.sh > /tmp/bootstrap.log 2>&1 Same results.

Attempted again with extra parameter (see print screen attached): touch -C 819 /tmp/bootstrap.log; /QOpenSys/usr/bin/ksh /tmp/bootstrap.sh --yes-this-is-unsupported-but-i-know-what-i-am-doing > /tmp/bootstrap.log 2>&1 While tailing the log in another putty session: touch /tmp/bootstrap.log && tail -f /tmp/bootstrap.log

Appears to just hang. Nothing of note written to the log. Network connection / firewall do not reflect any issues or blocks. /QOpenSys/var/lib has a yum directory, no source packages directory, etc.

Screen Shot 2023-10-31 at 10 42 17 AM
kadler commented 11 months ago

The touch -C is giving errors because you're not running from QSH and the PASE touch command doesn't understand that QSH-specific option, but that argument is only needed in QSH to ensure the file is created in ASCII.

Can you re-run as the following:

rm /tmp/bootstrap.log && /QOpenSys/usr/bin/ksh -x /tmp/bootstrap.sh --yes-this-is-unsupported-but-i-know-what-i-am-doing 2>&1 | tee /tmp/bootstrap.log
CSTARLING75 commented 11 months ago

Thanks Kevin. Will give that a try. Is the touch failure related to the bootstrap hanging and yum not completely installed?

kadler commented 11 months ago

I don't think that would have caused the bootstrap to hang.

The -x option will show all the commands being executed by the script, so we'll know what command it's hanging on. The script isn't that complicated, though, so it's likely just the tar command to extract the bootstrap. If that's hanging, then we'd need to see where it's hanging which is a bit more involved.

CSTARLING75 commented 11 months ago

Ran the command: rm /tmp/bootstrap.log && /QOpenSys/usr/bin/ksh -x /tmp/bootstrap.sh --yes-this-is-unsupported-but-i-know-what-i-am-doing 2>&1 | tee /tmp/bootstrap.log

Hangs at the zstd.bin -dc step, see print screen. Will attach bootstrap.log once available.

Screen Shot 2023-11-01 at 3 44 10 PM

Are there any additional options we can add to find out why it just seems to hang at that step? Or any way to verify if the decompress is running as expected but just super slow? System seems relatively quick and there's plenty of space available.

kadler commented 11 months ago

The zstd.bin is extracting bootstrap-stage2.tar.zst and piping the output to tar.bin. This is the main extraction of the bootstrap of /QOpenSys/pkgs, etc.

Do you have *ALLOBJ authority?

CSTARLING75 commented 11 months ago

Yes, the user profile we are using is secofr with allobj authority. qpusrprf.txt bootstrap.log

CSTARLING75 commented 11 months ago

Hi Kevin, Any other ideas or information needed to assist with troubleshooting?

CSTARLING75 commented 11 months ago

@kadler Any ideas or updates on this?

kadler commented 11 months ago

When running, you should be able to find the tar job in WRKACTJOB. I would monitor it for a few minutes and check its CPU usage and look at it's call stack to see if that changes. If it's using CPU and call stacks are changing then it is likely not hung, just running slowly but should finish.

If you need to investigate further, then you'll need to open a ticket with support. They can help you much more with performance issues than I can and if it is hung then we'll likely need to get the PASE team involved.

CSTARLING75 commented 10 months ago

Thanks @kadler . Regrouped with the customer today to troubleshoot further and oddly the issue no longer exists and yum is installed. We are not sure what resolved the issue, definitely nothing from what we did today as yum and other open source pkgs were found to be installed at this point (they did have other license issues that had been worked on recently so possibly related). As such, closing this issue. Thanks again for your assistance.