Need serial console access to node machines

[dledford]

This generally requires either a) we share login information to the/an iLO account or b) setup a console server that knows how to login to the iLO but only requires a valid login in the kerberos domain to access it (we use Cyclades terminal servers configured to login to the machine's iLO SoL port and log it and allow users to login to the console at Red Hat...but those units are discontinued last I knew).

Maybe we can setup builder to have console logs of the machines so that authenticated users can access their console logs?

Thoughts @lylavoie?

[shall1024]

There is a problem with console output; it seems like it is no longer being saved by Beaker at all, when looking at the console logs files on beaker:

I believe normally, any user should be able to view console output on any jobs they have submitted via the Beaker web interface. We can test this once logs are being stored by Beaker correctly.

[shall1024]

The above issue is solved; needed to restart the conserver service. @dledford , when you submit another job, please let us know if you have console log access. You should be able to access it from within the recipes within the jobs you submit; above the remaining watchdog time, you should see a link that says "Console Output".

[dledford]

I used manual provisions, there is no job related to it and no recipe, so I have no link to view the logs. However, if you add the command inst.sshd to the kernel command line options for a node, you can ssh into the machine during the install and view the logs directly. That's what I've been doing to debug things right now.

[lylavoie]

Is it possible to find them directly in the logs: https://beaker.ofa.iol.unh.edu/logs/

Cheers, Lincoln

On Thu, Jun 10, 2021 at 9:40 AM Doug Ledford @.***> wrote:

I used manual provisions, there is no job related to it and no recipe, so I have no link to view the logs. However, if you add the command inst.sshd to the kernel command line options for a node, you can ssh into the machine during the install and view the logs directly. That's what I've been doing to debug things right now.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OpenFabrics/fsdp_docs/issues/51#issuecomment-858632598, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIXKZ7PVPBRYT6RDYMYKOODTSC6FVANCNFSM46KP3AQA .

-- Lincoln Lavoie Principal Engineer, Broadband Technologies 21 Madbury Rd., Ste. 100, Durham, NH 03824 @.*** https://www.iol.unh.edu +1-603-674-2755 (m) https://www.iol.unh.edu

[lylavoie]

In terms of access. If we need users to have access to the SOL or similar for iLO, my preference would be to purchase / install the iLO licenses, which will enable support for RADIUS / LDAP on the iLO, which means user access is tied to their login and not a shared password, etc.

Second approach is to use a Cisco ISR with the Async card installed, which is basically an advanced terminal server. That would give similar allowance for AAA from radius.

[shall1024]

That link only contains recipe/job logs; not the console logs. Console logs are stored in a slightly different location, and we can't store them in this area due to how Beaker stores its recipe/job logs.

I think it should be easy to make the console logs accessible in a similar manner to these logs, however, if this is what we want to do.

The way the console logs are stored is in files by node, however, and I believe this would make it so any user can view the console logs of all 10 nodes, which is probably not desirable.

[dledford]

@lylavoie @shall1024 I would recommend we just update the default install options to include inst.sshd. For Fedora and RHEL releases of modern age, that means it enables a passwordless root ssh access during the install. So, if the install hangs, ssh in and see what's up. That's how I've been debugging things this morning and it's working quite well.

[shall1024]

inst.sshd has been added as a default install option to node-10, and I see this option has been added to all 9 other nodes. This seems like a good solution.

Let us know if there is anything else we need to do with the console logs/if there is other information you need access to, otherwise I will close the issue.

[dledford]

It seems like it will mostly work. It works with the recent Fedora releases, and it works for most of the rhel8 releases, it doesn't work on rhel7, rhel8.0 and rhel8.1 I think. Just too new of an option for those releases.

[dledford]

HPE iLO Advanced licenses for all 10 nodes have been ordered. We did not order one for builder since it's a shared resource and we don't really want people logging into its iLO.

[lylavoie]

iLO licenses arrived, we'll get those installed ASAP. HPE did ship these as paper envelopes.

[lylavoie]

All systems have the iLO5 license installed and have been configured for LDAP authentication and authorization. Cluster users have login, remote console, power/restart, and virtual media access, IOL employees have administrator login with their LDAP user.