search

coreos / fedora-coreos-tracker

Issue tracker for Fedora CoreOS
https://fedoraproject.org/coreos/
263 stars 59 forks source link

Fedore coreos 37 SEV_CAPABLE but inaccessible #1419

Open deeglaze opened 1 year ago

deeglaze commented 1 year ago

Describe the bug

On GCE, the fedora-coreos-stable images are tagged SEV_CAPABLE and appear to boot with --confidential-compute enabled, but the image is inaccessible from SSH. At first SSH window from Pantheon suggests adding an ingress firewall rule for 0.0.0.0/0 for tcp port 22, but doing that still causes the SSH connection to fail.

Reproduction steps

  1. Create an N2D machine with a fedora coreos (37.20230122.3.0) disk image and confidential-compute enabled.
  2. Navigate to VM in compute engine UI
  3. Click SSH to open an SSH window

Expected behavior

The SSH connection should succeed and give access to the VM.

Actual behavior

The SSH connection fails with "an unexpected error"

System details

Google Compute Engine

Butane or Ignition config

No response

Additional information

No response

bgilbert commented 1 year ago

Have you tried manually SSHing to the core user?

Please post a console log.

deeglaze commented 1 year ago

https://gist.github.com/deeglaze/1db6b9e5d14421935cfc388a2fc1c2bf

This is generally a problem for GCE's automatic image testing if the SSH capability is disabled by default.

bgilbert commented 1 year ago

SSH is enabled by default. You'll need to SSH to the core user, or put an Ignition config in userdata that configures SSH keys for a different user.

deeglaze commented 1 year ago

Are you asking for something other than ssh core@ip-address when you say SSH to the core user? I'm not sure I know what you mean by userdata. I focus primarily in virtualization backend stuff. I'm not on the guest images team, so please forgive my simple questions.

bgilbert commented 1 year ago

ssh core@ip-address should work fine.

I mentioned userdata only as an alternative if you don't want to use the core user. You can write an Ignition config and pass it to the VM (see the use of the --metadata-from-file option on that page). But that's somewhat more complex than just using core.

Also, I just noticed the Note in our GCP docs, which is relevant here.

deeglaze commented 1 year ago

Thanks for the note. The ssh core@$IP_ADDRESS times out for me instead of connecting.

bgilbert commented 1 year ago

I just launched fedora-coreos-37-20230122-3-0-gcp-x86-64 on n2d-standard-2 with Confidential VM Service enabled and an SSH key added via "Add manually generated SSH keys", and was able to SSH in.

I assume it continues to time out after the VM has been running for a few minutes?

deeglaze commented 1 year ago

Yes that's correct.

bgilbert commented 1 year ago

Thanks for the info. Since I haven't been able to reproduce this issue, and it appears from your logs that the OS is booting successfully, I'm not sure how to advise you further. Are you using any other non-default configuration for the VM? Is there any other info that would help us reproduce this?

dustymabe commented 1 year ago

ping @deeglaze - any more information you can provide?

deeglaze commented 1 year ago

I've since gone on with other work I've been tasked with, but I can give you an update that Fedora is currently not tested at all due to test framework incompatibilities (we use paramiko 2.10 if that's relevant?). "Fedora has a different way to SSH in than the rest of GCE."

Boot tests wait for the guest to be sshable after creating the VM making sure the test ssh keys are in the metadata service before attempting to connect. I don't have much more info than that though. I'm not on the team that is supposed to do all the image testing. I work on the confidential compute side of things.

cgwalters commented 1 year ago

"Fedora has a different way to SSH in than the rest of GCE."

Hmm. Are you sure this isn't fallout from e.g. https://www.fedoraproject.org/wiki/Changes/StrongCryptoSettings2 ?

dustymabe commented 1 year ago

"Fedora has a different way to SSH in than the rest of GCE."

Hmm. Are you sure this isn't fallout from e.g. https://www.fedoraproject.org/wiki/Changes/StrongCryptoSettings2 ?

Or maybe the fact that we don't support OS Login?

deeglaze commented 1 year ago

"Fedora has a different way to SSH in than the rest of GCE."

Hmm. Are you sure this isn't fallout from e.g. https://www.fedoraproject.org/wiki/Changes/StrongCryptoSettings2 ?

Yes, that update hit many distros and is the reason we updated paramiko from 2.7 to 2.10. Fedora issues remain, and I think @dustymabe is probably correct.