Open CJCShadowsan opened 4 years ago
The elilo in question for 18.04.5:
[root@service01 install]# cat /tftpboot/xcat/xnba/nodes/gpu05.elilo
default="xCAT"
delay=0
image=/tftpboot/xcat/osimage/ubuntu18.04.5-x86_64-install-compute/vmlinuz
label="xCAT"
initrd=/tftpboot/xcat/osimage/ubuntu18.04.5-x86_64-install-compute/initrd.img
append="nofb utf8 auto url=http://%N:80/install/autoinst/gpu05 xcatd=%N mirror/http/hostname=%N:80 netcfg/choose_interface=d8:c4:97:b8:32:cb console=tty0 console=ttyS0,115200 locale=en_US priority=critical hostname=gpu05 live-installer/net-image=http://%N:80/install/ubuntu18.04.5/x86_64/install/filesystem.squashfs rd.drive.blacklist=nouveau nouveau.modeset=0 ask_detect=false ramdisk_size=100000 nvme-core.multipath=n BOOTIF=%B"
I also get zero output from any tty to aid in debugging - which I appreciate is definitely not ideal!
Also note the same ISOs both install over USB fine...
can u see any message from xcat log? /var/log/xcat/cluster.log
or compute.log
for this node
how about end of console screen? you can find it in the /var/log/consoles/<nodename>
also
No, nothing inside the cluster.log - or the compute.log.
Output from the console log:
Version 2.17.1249. Copyright (C) 2019 American Megatrends, Inc.
BIOS Date: 02/12/2019 16:14:45 Ver: S2W_3A08
Press <DEL> or <F2> to enter setup.
Press <F11> for BBS POPUP menu.
Press <F12> if you want to boot from the network.
CPU: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz Speed: 2200MHz
Total Memory: 512GB (DDR4 2133)
USB Devices total: 2 KBDs, 1 MICE, 0 MASS, 3 HUBs
[2020-08-25T12:45:12Z] >>Checking Media Presence......
[2020-08-25T12:45:12Z] >>Media Present......
[2020-08-25T12:45:12Z] >>Start PXE over IPv4.
[2020-08-25T12:45:16Z] Station IP address is 172.17.4.5
[2020-08-25T12:45:16Z]
[2020-08-25T12:45:16Z] Server IP address is 172.17.1.211
[2020-08-25T12:45:16Z] NBP filename is xcat/xnba.efi
[2020-08-25T12:45:16Z] NBP filesize is 139200 Bytes
[2020-08-25T12:45:16Z] >>Checking Media Presence......
[2020-08-25T12:45:16Z] >>Media Present......
[2020-08-25T12:45:16Z] Downloading NBP file...
[2020-08-25T12:45:16Z]
[2020-08-25T12:45:16Z] Succeed to download NBP file.
[2020-08-25T12:45:16Z] xNBA initialising devices...ok
[2020-08-25T12:45:16Z]
[2020-08-25T12:45:16Z]
[2020-08-25T12:45:16Z] xCAT Network Boot Agent
[2020-08-25T12:45:16Z] iPXE 1.0.3-131028 (d603e) -- Open Source Network Boot Firmware -- http://ipxe.org
[2020-08-25T12:45:16Z] Features: HTTP HTTPS iSCSI DNS TFTP EFI
[2020-08-25T12:45:16Z] net0: d8:c4:97:b8:32:cb using <NULL> on EFI SNP (open)
[2020-08-25T12:45:16Z] [Link:up, TX:0 TXE:0 RX:0 RXE:0]
[2020-08-25T12:45:16Z] DHCP (net0 d8:c4:97:b8:32:cb)... ok
[2020-08-25T12:45:16Z] net0: 172.17.4.5/255.255.0.0 gw 172.17.1.211
[2020-08-25T12:45:16Z] Next server: 172.17.1.211
[2020-08-25T12:45:16Z] Filename: http://172.17.1.211:80/tftpboot/xcat/xnba/nodes/gpu05.uefi
[2020-08-25T12:45:16Z] http://172.17.1.211:80/tftpboot/xcat/xnba/nodes/gpu05.uefi... ok
[2020-08-25T12:45:16Z] http://172.17.1.211:80/tftpboot/xcat/elilo-x64.efi... ok
[2020-08-25T12:45:17Z] ELILO v3.14 for EFI/x86_64
[2020-08-25T12:45:17Z] Loading kernel /tftpboot/xcat/osimage/dgx-4.5-install-compute/vmlinuz... done
[2020-08-25T12:45:23Z] Loading file /tftpboot/xcat/osimage/dgx-4.5-install-compute/initrd.img...done
[2020-08-25T12:47:28Z]
Version 2.17.1249. Copyright (C) 2019 American Megatrends, Inc.
BIOS Date: 02/12/2019 16:14:45 Ver: S2W_3A08
Press <DEL> or <F2> to enter setup.
Press <F11> for BBS POPUP menu.
Press <F12> if you want to boot from the network.
Which pretty much is what I see during boot, but is the same for a stock ubuntu 18.04.5 iso as well.
it seems finish loading kernel and initrd, xCAT MN should receive another DHCP request after that. maybe checking if other server got request. xcatprobe detect_dhcpd -i <provision interface> -m d8:c4:97:b8:32:cb
Also, in the current xCAT development builds, xCAT skip the elilo https://github.com/xcat2/xcat-core/pull/6772, you can manually change uefi
file after nodeset
,
root@c910f04x35v05:/tftpboot/xcat/xnba/nodes# cat c910f04x35v07.uefi
#!gpxe
imgfetch -n kernel http://${next-server}:80/tftpboot/xcat/osimage/ubuntu18.04.5-x86_64-install-compute/vmlinuz
imgload kernel
imgargs kernel nofb utf8 auto url=http://c910f04x35v05:80/install/autoinst/c910f04x35v07 xcatd=c910f04x35v05 mirror/http/hostname=c910f04x35v05:80 netcfg/choose_interface=42:e7:0a:04:23:07 console=tty0 console=ttyS0,115200n8r locale=en_US priority=critical hostname=c910f04x35v07 live-installer/net-image=http://c910f04x35v05:80/install/ubuntu18.04.5/x86_64/install/filesystem.squashfs BOOTIF=01-${netX/mac:hexhyp} initrd=initrd
imgfetch -n initrd http://${next-server}:80/tftpboot/xcat/osimage/ubuntu18.04.5-x86_64-install-compute/initrd.img
imgexec kernel
root@c910f04x35v05:/tftpboot/xcat/xnba/nodes# cat c910f04x35v07
#!gpxe
#install ubuntu18.04.5-x86_64-compute
imgfetch -n kernel http://${next-server}:80/tftpboot/xcat/osimage/ubuntu18.04.5-x86_64-install-compute/vmlinuz
imgload kernel
imgargs kernel nofb utf8 auto url=http://c910f04x35v05:80/install/autoinst/c910f04x35v07 xcatd=c910f04x35v05 mirror/http/hostname=c910f04x35v05:80 netcfg/choose_interface=42:e7:0a:04:23:07 console=tty0 console=ttyS0,115200n8r locale=en_US priority=critical hostname=c910f04x35v07 live-installer/net-image=http://c910f04x35v05:80/install/ubuntu18.04.5/x86_64/install/filesystem.squashfs BOOTIF=01-${netX/machyp}
imgfetch -n initrd http://${next-server}:80/tftpboot/xcat/osimage/ubuntu18.04.5-x86_64-install-compute/initrd.img
imgexec kernel
it seems finish loading kernel and initrd, xCAT MN should receive another DHCP request after that. maybe checking if other server got request.
xcatprobe detect_dhcpd -i <provision interface> -m d8:c4:97:b8:32:cb
Also, in the current xCAT development builds, xCAT skip the elilo #6772, you can manually change
uefi
file afternodeset
,root@c910f04x35v05:/tftpboot/xcat/xnba/nodes# cat c910f04x35v07.uefi #!gpxe imgfetch -n kernel http://${next-server}:80/tftpboot/xcat/osimage/ubuntu18.04.5-x86_64-install-compute/vmlinuz imgload kernel imgargs kernel nofb utf8 auto url=http://c910f04x35v05:80/install/autoinst/c910f04x35v07 xcatd=c910f04x35v05 mirror/http/hostname=c910f04x35v05:80 netcfg/choose_interface=42:e7:0a:04:23:07 console=tty0 console=ttyS0,115200n8r locale=en_US priority=critical hostname=c910f04x35v07 live-installer/net-image=http://c910f04x35v05:80/install/ubuntu18.04.5/x86_64/install/filesystem.squashfs BOOTIF=01-${netX/mac:hexhyp} initrd=initrd imgfetch -n initrd http://${next-server}:80/tftpboot/xcat/osimage/ubuntu18.04.5-x86_64-install-compute/initrd.img imgexec kernel root@c910f04x35v05:/tftpboot/xcat/xnba/nodes# cat c910f04x35v07 #!gpxe #install ubuntu18.04.5-x86_64-compute imgfetch -n kernel http://${next-server}:80/tftpboot/xcat/osimage/ubuntu18.04.5-x86_64-install-compute/vmlinuz imgload kernel imgargs kernel nofb utf8 auto url=http://c910f04x35v05:80/install/autoinst/c910f04x35v07 xcatd=c910f04x35v05 mirror/http/hostname=c910f04x35v05:80 netcfg/choose_interface=42:e7:0a:04:23:07 console=tty0 console=ttyS0,115200n8r locale=en_US priority=critical hostname=c910f04x35v07 live-installer/net-image=http://c910f04x35v05:80/install/ubuntu18.04.5/x86_64/install/filesystem.squashfs BOOTIF=01-${netX/machyp} imgfetch -n initrd http://${next-server}:80/tftpboot/xcat/osimage/ubuntu18.04.5-x86_64-install-compute/initrd.img imgexec kernel
Just so I can clarify - the only difference between these two files is the initrd=initrd
bit at the end of line 4, correct?
And that's meant to be there in the .uefi file?
Ok - so with the second item below, I get:
[ 13.493678] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
[ 13.507497] Please append a correct "root=" boot option; here are the available partitions:
[ 13.522299] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 13.537120] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.3.0-28-generic #30~18.04.1-Ubuntu
[ 13.551959] Hardware name: NVIDIA DGX-1 with V100-32-MaxQ/DGX-1 with V100-32-MaxQ, BIOS S2W_3A08 02/14/2019
[ 13.567768] u usb 1-7: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[ 13.592994] dump_stack+0x6d/0x95
[ 13.607291] usb 1-7: Product: USB2.0 Hub
[ 13.617782] panic+0xfe/0x2d4
[ 13.617786] mount_block_root+0x1f4/0x2db
[ 13.629234] hub 1-7:1.0: USB hub found
[ 13.638585] ? set_debug_rodata+0x17/0x17
[ 13.638587] mount_root+0x38/0x3a
[ 13.638588] prepare_namespace+0x139/0x18e
[ 13.638589] kernel_init_freeable+0x245/0x26d
[ 13.649772] hub 1-7:1.0: 4 ports detected
[ 13.659878] ? rest_init+0xb0/0xb0
[ 13.659879] kernel_init+0xe/0x110
[ 13.672901] usb 1-6.4: new low-speed USB device number 5 using xhci_hcd
[ 13.680156] ret_from_fork+0x35/0x40
[ 13.690644] Kernel Offset: 0xc800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 13.800659] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---
So at least i'm getting some output now... But it seems to indicate i've not got a root fs to mount and then that's it.
Confirmed - initrd=initrd
needs to be initrd=initrd.img
- and then things spring into life.
Is this actually going to be default behaviour for #6772 going forwards? What's the ETA?
that's good news.
I will go over this and any changes we make that will be in the development build. our next release 2.16.1 will be at the end of October.
@CJCShadowsan , just want to confirm again. you just need to change initrd=initrd
to be initrd=initrd.img
on the uefi
, right?
Yes, that's what I did and then it worked.
On Thu, 27 Aug 2020, 18:05 cxhong, notifications@github.com wrote:
@CJCShadowsan https://github.com/CJCShadowsan , just want to confirm again. you just need to change initrd=initrd to be initrd=initrd.img on the uefi, right?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/xcat2/xcat-core/issues/6811#issuecomment-682075507, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD326MENGY4TXNEKNHRSVSDSC2G5FANCNFSM4QKVXLUA .
@cxhong - did this make this into 2.16.1?
@CJCShadowsan What xCAT version (lsxcatd -v
) were you using when you had this problem originally? I have been trying to reproduce on the latest xCAT development builds and I want to compare the code changes between the release you were using and what is in current master to try to determine if this is still a problem or not.
I was not able to reproduce the problem originally described in this issue on my test hardware, but two fixes were included with xCAT 2.16.2 that might be relevant. 1.) xNBA was rebuild to match upstream iPXE 1.20.1. 2.) Missing file elilo-x86.efi has been added to elilo-xcat-3.14-6_all.deb
Can you please provide feedback about whether xCAT 2.16.2 resolves this problem in your cluster?
Hi,
Both an installation of 18.04.4 (which was DGX-OS 4.5) and 18.04.5 (official ubuntu) both hang at the initrd stage:
And both go no further. They also reboot, and the process continues ad infinitum so it's clear something is crashing. Is there some sort of memory limitation on initrd?
OS image used is http://cdimage.ubuntu.com/releases/18.04.5/release/ubuntu-18.04.5-server-amd64.iso
This is pretty breaking - i'm trying to deploy ubuntu from a centos server and it's needed.
Note that genesis boots fine on this DGX-1 server - so this is a definite ubuntu and xCAT issue.