Eflow 1.4 and masked sshd

[chokosticks]

Question Summary Is sshd over tcp port 22 disabled in Eflow version 1.4?

Detailed background On Eflow 1.1 we created an user and managed to connect to iotedge over ssh. After installation of Eflow 1.4 this is no longer possible and we're getting Connection refused. The sshd-service is masked. We were able to get it working again by unmasking and enabling the service.

systemctl status sshd

• sshd.service Loaded: masked (Reason: Unit sshd.service is masked.) Active: inactive (dead) since Fri 2022-11-25 10:25:10 UTC; 3h 19min ago Main PID: 950 (code=exited, status=0/SUCCESS)

Nov 25 10:25:06 MST299D-EFLOW systemd[1]: Started OpenSSH Daemon. Nov 25 10:25:06 MST299D-EFLOW sshd[950]: Server listening on 0.0.0.0 port 22. Nov 25 10:25:06 MST299D-EFLOW sshd[950]: Server listening on :: port 22. Nov 25 10:25:10 MST299D-EFLOW systemd[1]: Stopping OpenSSH Daemon... Nov 25 10:25:10 MST299D-EFLOW sshd[950]: Received signal 15; terminating. Nov 25 10:25:10 MST299D-EFLOW systemd[1]: sshd.service: Deactivated successfully. Nov 25 10:25:10 MST299D-EFLOW systemd[1]: Stopped OpenSSH Daemon.

I found the following text regarding Eflow CR:

In the EFLOW Continuous Release (CR) version, we introduced a change in the transport channel used to establish the SSH connection. Originally, SSH service runs on TCP port 22, which can be accessed by all external devices in the same network using a TCP socket to that specific port. For security reasons, EFLOW CR runs the SSH service over Hyper-V sockets instead of normal TCP sockets. All communication over Hyper-V sockets runs between the Windows host OS and the EFLOW virtual machine, without using networking. This limits the access of the SSH service, restricting connections to only the Windows host

What would be the "go to" solution to enable the functionality to be able to connect over ssh when in the same network?

Thanks!

[fcabrera23]

Hi @chokosticks,

Thanks for your question. The documentation you shared is correct, and since EFLOW CR, we moved from SSH over TCP to SSH over vSock (Hyper-V Socket). This means that SSH is only enabled between the Windows host OS and the EFLOW VM.

If you want to enable SSH over TCP for external network access, you can unmask and enable the SSHD service and change the port (e.g 1022) so that you can have both SSH over TCP and vSock. However, SSH over TCP sockets is something that we do not recommend due to security concerns.

Thanks, Francisco

[chokosticks]

Hi,

Thanks for the response @fcabrera23.

I did try to change to port 1022, as well as added 1022 in iptables. I am now able to connect ssh over tcp when sshd.service is started.

But, when sshd is started I can't use the Connect-EflowVM command, I am getting:

Connection timed out during banner exchange.

As soon as I restart EflowVM I am able to use Connect-EflowVM. But then sshd is inactive.

Is there a possibility that the sshd service is disturbing the setup with SSH over vSock?

[fcabrera23]

Hi @chokosticks,

Thanks for testing that out. Because this is something that we do not support, I'm not really familiar with the configurations and potential issues.

Could you please share the output of systemctl status sshd_v* and systemctl status sshd*?

I imagine that both services are using /usr/bin/sshd and there's an issue with that.

Thanks, Francisco

[chokosticks]

Hi,

Yes, attached the output for each given command. The output is for when the iptables are updated, sshd is unmasked and started i.e. SSH over TCP works fine. And if I'm already in an SSH over vSock, the session will remain open. But if I exit put and try Connect-EflowVM it fails with the previous message.

sshd_v.txt sshd.txt

[chokosticks]

One solution would be, I guess, to copy and have two instances of sshd which could have different configurations than the regular sshd. I found an example from someone who did that, but I guess CBL-Mariner is a bit different in the file structure?

Why I am asking around this is because it would fit our need to be able to use SSH over TCP. To be able to execute commands and troubleshoot without going via the Windows-host.

The hosts are already behind pretty locked down networks and wouldn't be exposed to the internet.

[chokosticks]

Hi again @fcabrera23 ,

I found in /etc/systemd/system/sockets.targets.wants that sshd_vsock.socket had:

[Unit] Conflicts=sshd.service.

So I guess they can't co-exist as of now.

[fcabrera23]

Hi @chokosticks .

Thanks for the update. I'll leave it to @johnsonshih - Do you know if it's possible to have both SSH over TCP and vSock? Is it possible to change the sshd_vsock to remove the sshd.service dependency?

Thanks, Francisco

[chokosticks]

Hi @johnsonshih,

have you been able to take a look on @fcabrera23's question regarding having both SSH over TCP and vSock simultaneous?

Thanks.

[chokosticks]

Hi again @fcabrera23,

Do you know how this is going or if he's looking in to it?

Thanks in advanced.

[fcabrera23]

Hi @chokosticks,

After carefully reviewing this scenario and its potential implications, we cannot support it. You can do it at your own risk, but consider that in the future, we may enable/modify SSH service again, which could impact your deployment. If you want to enable it, you can modify the socket.targets service and remove the Conclifcts statement and try again.

Thanks, Francisco

[chokosticks]

Hi @fcabrera23 ,

thanks for the response! We'll not enable the feature. We'll try to find another solution to the problem.

Thanks for the time looking into it!

[chokosticks]

Hi again @fcabrera23,

I did try a different approach this time. I've enabled Remote Powershell session.

I am able to run Invoke-EflowVmCommand from the inteactive session, but I am not able to run Connect-EflowVm to start an interactive session towards the VM. I've understood that Connect-EflowVm will only be available via the host computer, but that also excludes the functionality regarding Remote PSSession?

Would it be possible to install another instance of open-ssh in the VM, but with a different name and running on another port other than 22?

Thanks in advance.

[fcabrera23]

Hi @chokosticks,

Thanks for the update. Remote PowerShell does not support 2-hop interactive sessions. By that I mean, first, establish a remote PowerShell and then a remote SSH.

Regarding the SSHD service over TCP, you can try that out and see if it works, but again this is something that we do NOT recommend and support.

Thanks, Francisco

[chokosticks]

Hi @fcabrera23,

thanks for the update!