Add SCP application button for easier access to the VM contents

[JustinProminic]

Per the group discussion today, it needs to be easier to SCP to the VMs created by S.H.I. We already have all the code necessary for this integration in Prominic Native, and can contribute the portions of it to S.H.I. that are relevant for the definition of an SCP client.

To keep the initial implementation easy, rather than support a variety of SCP clients as Native does, we can just implement support for FileZilla SCP since it is a good solution and cross platform.

@JoelProminic please add any other details you see fit on this issue. Thank you!

[JoelProminic]

We can get the credentials like this:

• IP: same as for the welcome page. I know that this is problematic in some cases
• Username: Hosts.yml, hosts.settings.vagrant_user
• Password: Hosts.yml, hosts.settings.vagrant_user_pass

Previously (I could be thinking of demo-tasks), I was able to SCP to the VM using the IP address or FQDN from the hosts file. However, with my recent tests with the v0.1.20 provisioners, I found that the password does not work. I confirme that a password was set for the startcloud user, but it doesn't seem to match the configuration. I'm checking if @MarkProminic has any ideas on this from the provisioner side before I start working on my own solution for this.

startcloud@nomad-1010:~$ sudo passwd -S startcloud
startcloud P 2024-01-07 0 99999 7 -1

Alternatively, we could use the vagrant-scp plugin. However, this seems to work with a command-line interface like the scp command, which is what @JustinProminic desscribed here.

vagrant scp scopedlinks.html 4882--demo.STARTcloud.com:~/

[JoelProminic]

@piotrzarzycki21, we should retest scp to the VM with the v0.1.22 provisioners (#99). If there is a problem, I'd like to debug this with the updated provisioners rather than the legacy v0.1.20.

[JoelProminic]

I confirm that SSH/SCP is still broken with the 0.1.22 provisioners. The instance that is working with SCP for shows as "demo-tasks v0.1.20", but I have other instances on 0.1.20 that are not working. I'll try a diff between those.

UPDATE: I didn't find any obvious difference. It is possible that I made some manual change for this and forgot about it.

[JoelProminic]

I did a little more inestigation, and it looks like password authentiction is disabled here.

However, there is a public/private key pair configured, which you can find here. I confirmed that I can ssh/scp to the VM like this:

ssh -i core/ssh_keys/id_rsa startcloud@test22.shi.com
scp -i core/ssh_keys/id_rsa Hosts.yml startcloud@test22.shi.com:/tmp/test_Hosts.yml

I was able to get this to work with FileZilla with some manual steps. We would need to check if the FileZilla command line supports this.

[JoelProminic]

@MarkProminic suggested that we use some sort of HTML5 file browser instead. Example

[JustinProminic]

@JoelProminic @MarkProminic I am requesting exactly what we have working with FileZilla SCP in Native already. I do not want to transition at this time to an HTML5 solution, but I'm not opposed to looking into it in the future. For now, I'm after specifically the same FileZilla SCP support we have already coded in Native. It just needs to be made to work with the S.H.I. VM provisioners without any additional steps please like what Joel had to do above for the user.

[JoelProminic]

I think the syntax used in Native for FileZilla is:

/Applications/FileZilla.app/Contents/MacOS/filezilla 'sftp://startcloud:-password-@!@demo.startcloud.com' 

There is some command line help here, but I don't see anything to use the private key directly:

~ % /Applications/FileZilla.app/Contents/MacOS/filezilla --help                
Usage: filezilla [-h] [-s] [-c <str>] [-a <str>] [-l <str>] [--verbose] [-v] [--debug-startup] [<FTP URL>]
  -h, --help            Shows this help dialog
  -s, --site-manager    Start with opened Site Manager
  -c, --site=<str>      Connect to specified Site Manager site
  -a, --local=<str>     Starts the local site in the given path
  -l, --logontype=<str> Logontype, can only be used together with FTP URL. Argument has to be either 'ask' or 'interactive'
  --verbose             Verbose log messages from wxWidgets
  -v, --version         Print version information to stdout and exit
  --debug-startup       Print diagnostic information related to startup of FileZilla

[rat-moonshine]

I confirm @JoelProminic on how's Native calls to FileZilla SCP:

/Applications/FileZilla.app/Contents/MacOS/filezilla sftp://$userName:$userPassword@$IP:$Port

[piotrzarzycki21]

I confirm @JoelProminic on how's Native calls to FileZilla SCP:

/Applications/FileZilla.app/Contents/MacOS/filezilla sftp://$userName:$userPassword@$IP:$Port

This is all nice, but how do I get login name and user password in SHI ? I'm not sure where it is all stored actually.

[piotrzarzycki21]

Actually I think Server provides probably ip and username. Maybe password is not stored there.

[JoelProminic]

You can get the username and password from hosts.settings in Hosts.yml. Note that we play to make changes for this in #114, but I would expect the credentials to remain at the same location.

I think we were writing the IP address to a file somewhere, but I remember that there were some cases when it got out of sync with the VM.

[piotrzarzycki21]

I have spent today couple of hours on this and still no luck with connecting to ftp server. I did change this line https://github.com/Moonshine-IDE/Super.Human.Installer/blob/6e027e32c966da3a631318c82c706d7ed7a211b3/Assets/provisioners/demo-tasks/0.1.22/scripts/core/Hosts.rb#L35 but it end up with some weird prompt for password on SHI console instead of moving forward.

I have started experimenting with above instruction from you and end up with following url:

sftp://${this._organization.value}:STARTcloud22@!@${ftpAddress} where sftp://startcloud:STARTcloud22@!@moon.startcloud.com

It keeps reporting me on Filezilla console:

Status:         Connecting to moon.startcloud.com...
Response:   fzSftp started, protocol_version=9
Command:    open "startcloud@moon.startcloud.com" 22
Error:          ssh_init: nodename nor servname provided, or not known
Error:          Could not connect to server
Status:         Waiting to retry...
Status:         Connecting to moon.startcloud.com...
Response:   fzSftp started, protocol_version=9
Command:    open "startcloud@moon.startcloud.com" 22
Error:          ssh_init: nodename nor servname provided, or not known
Error:          Could not connect to server

My vagrant server looks like this:

[JoelProminic]

To make progress on the SCP testing, you can manually enable SSH like this:

sudo su -
passwd startcloud
# Enter the configured password and confirm

After this, you should be able to SCP to the VM.

@MarkProminic, do you remember if there is something else required to let server.ssh.password work? I'm doing some research for this now, and I'll update the issue if I have any ideas.

[piotrzarzycki21]

To make progress on the SCP testing, you can manually enable SSH like this:

sudo su -
passwd startcloud
# Enter the configured password and confirm

After this, you should be able to SCP to the VM.

@MarkProminic, do you remember if there is something else required to let server.ssh.password work? I'm doing some research for this now, and I'll update the issue if I have any ideas.

This doesn't work for me. I think we need to have small session how should I actually build my command in order to open filezilla with login/password.

[JoelProminic]

Ideally we should read the credentials from the generated Hosts.yml. See the lines in the template here. If we don't have a good API for parsing Hosts.yml, you can use a simple regex like:

\svagrant_user:\s+(\w+)$
\svagrant_password:\s+'?([^']*)'?$

[JoelProminic]

We have an issue to change how we define the password, but I think we should still be reading this from Hosts.yml as the official source of truth for the instance configuration.

[piotrzarzycki21]

Hi @Igazine could you please share what was the source of currently used Assets (Icons) in SHI ? Did you download them from somewhere ? Thanks!

[Igazine]

I believe those are Material Icons: https://fonts.google.com/icons With a few custom colored ones

[MarkProminic]

If the IP becomes out of sync with the VM via the .vagrant/provisioned-briged-ip.txt file, ( in the Hosts.rb, provisioned-briged-ip.txt is only updated when the Ansible Playbook is run)

You could run a command to get the IP:

vagrant ssh -- -t 'ip addr show'

[MarkProminic]

Hello Piotr,

SSH key and SSH password authentication should work when connecting to the VM.

The templates are built with the following username and password (which I believe is also in the Hosts.yml)

startcloud STARTcloud24@! (or packer template v0.0.3 its STARTcloud22@!)

Manual access via password or key should work.

However the playbooks are executed via Key as that is industry best practice:

https://docs.ansible.com/ansible/latest/inventory_guide/connection_details.html

By default, Ansible assumes you are using SSH keys to connect to remote machines. SSH keys are encouraged, but you can use password authentication if needed with the --ask-pass option. If you need to provide a password for privilege escalation (sudo, pbrun, and so on), use --ask-become-pass.

Note

Ansible does not expose a channel to allow communication between the user and the ssh process to accept a password manually to decrypt an ssh key when using the ssh connection plugin (which is the default). The use of ssh-agent is highly recommended.

[MarkProminic]

In my next release of the Provisioners I will include a role that I use else where, startcloud_lockdown, that does the following:

Changes the users and roots password from that of the Templates default. Changes the SSH key (though this can be tricky if wanting to run more provisioners after) Disables some services that come with the OS like postfix,cockpit,exim etc.

[JoelProminic]

@MarkProminic, are you saying that the password is based on the configured box rather than vagrant_user_pass? In that case we should update vagrant_user_pass to match the box in Hosts.template.yml

[MarkProminic]

Yes, that is the password to connect to the vagrant server, not the password it will change the default user to use.

So unless we had a role like startcloud_lockdown to change the password during the provisioning process to a dynamic/auto generated pass, the password will always be that of the template.

I did not realize you were not using the password that was in the Template.

[JoelProminic]

@MarkProminic, I spun up a new SHI instance with 0.1.22 and the v0.0.4 box, and I found that STARTcloud24@! does not work as the password either. Is there another password we should try?

UPDATE: Nevermind. The 24 password worked when I tested it from within the VM. I suspect that I had an IP conflict or some similar issue in my environment. I shut down all other Vagrant instances and restarted the new instance, and I found that the password worked properly when tested from my workstation. I will watch for more errors with this.

[piotrzarzycki21]

@JoelProminic I have pushed changes for this issue. Give it a try with build. I still have to put proper FileZilla icon, that's why I'm not releasing build yet. Settings are working, but need to be expanded on button which check whether path is correct. In general if path is incorrect - button won't work and you should get proper message on user's server console.

Link to 0.9.10: https://github.com/Moonshine-IDE/Super.Human.Installer/releases/tag/untagged-9ef56a0cdee753532da7

[piotrzarzycki21]

I have released 0.9.12 version with significant update to path validation. It turns out it wasn't trivial task on Mac due to this masking unde *.app folder real executable. I ended up using lot's of logic from Native in that task.

[JoelProminic]

Test	macOS	Windows
Filezilla button with proper address and credentials	Works	Confirmed by @MarkProminic
Default path for Filezilla matches default installation	Confirmed	Confirmed by @MarkProminic
Path can be updated for Filezilla	Confirmed	Confirmed by @MarkProminic

One oddity is that the application name is editable for Filezilla. This doesn't seem to hurt anything, but it caused some confusion when it got cleared for Mark.

We don't want to get stuck debugging the Windows provisioning issues for this release, but I'd like to quickly confirm the remaining cases above on Windows.

[rat-moonshine]

This might be a know issue. I'm just doing a quick dump:

[piotrzarzycki21]

This might be a know issue. I'm just doing a quick dump:

If server is configured in hosts, than it may be a password mismatch - between what is in Hosts.yml and what is actually needed.

[piotrzarzycki21]

This has been implemented and tested on Mac and Windows. Raise separate issue in case of a problems.

[rat-moonshine]

If server is configured in hosts, than it may be a password mismatch - between what is in Hosts.yml and what is actually needed.

I don't know what this means. Can you point me to location(s) which I needs to compare?

[MarkProminic]

I think that if /etc/hosts or C:\Windows\System32\drivers\etc\hosts has the old or a different IP address they will still get a password mismatch?

The reason for using the Hostname I think was because people were manually updating the IP because done.txt, or the file bridged-adapters.yml, wasn't updating on reboots, or subsequent provisioning or in case their DHCP server gave their VM a new IP.

in v0.1.23, you will be able to read results.yml, which will be updated every time the machine is reloaded or provisioned.

[MarkProminic]

Sorry, I didn't finish my sentence, in v0.1.23, you will be able to read results.yml, which will be updated every time the machine is reloaded or provisioned. This will include the updated IP, for FileZilla to connect to.