Adding VCH 0.7.0 as a host in Admiral (lightweight Container Management platform)

[vmwarelab]

VIC Version: 0.7.0

// A self-contained demonstration of the problem follows...

Trying to add a VCH VM deployed with TLS as a host in Admiral A new TLS options were introduced it seems in VIC 0.7.0 and and i m trying to understand which is the right supported option that i can use so i can successfully add it as a host in Admiral

TLS Options

Common Name must be provided when generating certificates for client authentication:
--tls-cname=<FQDN or static IP>    # for the appliance VM
--tls-cname=<*.yourdomain.com>  # if DNS has entries in that form for DHCP addresses (less secure)
--no-tlsverify    # disables client authentication (anyone can connect to the VCH)
--no-tls          # disables TLS entirely

One of the things i used to see with VIC 0.6.0 without explicitly using any TLS options during the deployment cause they simply didnt exists, that it used to create one pair of public and private (key) certificates on the client machine for you after the deployment in the same folder where VIC files were extracted.. then I would use these certificate as credentials leveraging the credential certificate option in Admiral when adding VCH as a host.

Problems

1. When using --no-tls . no certificate is generated, there for i am simply forced to use a username 'root' and a password when adding theVCH VM as a host in Admiral but again i have no idea what the password is. so i m stuck .

2. When using --no-tlsverify . it Generate a self-signed certificate/key pair - private key in ./VCH01/key.pem (VCH01 is a folder named after the name of the deployed VCH VM) . When i switch to the folder the referenced file isnt there.!!! there is no key.pem file . the only file found in that folder is a file called VCH01.env. so here i m stuck again.

3. When using the --tls-cname= the difficulties i find here with the VIC 0.7.0 is that the certificates now are created in a folder named after the name of the deployed VCH VM which is great but there were more than one certificate pair generated here which was confusing to decide which pair i am suppose to use for the host credentials in Admiral leveraging the credential certificate option

   Generating CA certificate/key pair - private key in ./VCH01/ca-key.pem
   Generating server certificate/key pair - private key in ./VCH01/server-key.pem
   Generating client certificate/key pair - private key in ./VCH01/key.pem
   Generated browser friendly PFX client certificate - certificate in ./VCH01/cert.pfx

Any guidance on this is appreciated.

[vdeger]

Your progress is better than me, I can not communicate with Admiral with vchhostip:8282 yet :) VCH 0.6 credentials: root/password VCH 0.7 credentials: root/Vmware123! ( I did not try yet.)

[vmwarelab]

@vdeger unfortunately Vmware123! doesn't work

[vdeger]

Could you try to activate debug mode? .\vic-machine-linux debug --name VCH --target 192.x.x.x --user vchuser@vsphere.local --password Vmware123! --enable-ssh --rootpw 'Vmware123!'

[vmwarelab]

@vdeger i don't think the command is correct as there suppose to be a value for --enable-ssh. its not accepting the --enable-ssh parameter also is this done only during the deployment time ?

[mlh78750]

@vmwarelab and @vdeger

Have you all tried the instructions here?: https://github.com/vmware/vic-product/blob/master/docs/setup/beta/install-configure-admiral.md

The vic-product repo includes workflows for installing and configuring VIC Engine, Harbor, and Admiral together.

For enabling ssh, you can do that after install with the vic-machine debug command. The password you pass in is up to you. The root password will be set to that password. Be careful if you are using unix and you try to set the password to !## as some shells will replace that with other characters.

@vmwarelab Details on the certs start here in the docs: https://github.com/vmware/vic/blob/master/doc/user_doc/vic_installation/vch_installer_options.md#security

[vmwarelab]

the commands worked but like this

./vic-machine-linux debug -t 'administrator@vmwarelab.local:*****@192.168.0.20' --enable-ssh --rootpw 'VMware1!' --thumbprint AA:A2:7F:E1:2E:98:00:F6:24:6B:8B:73:E8:DC:EF:65:F2:D3:40:5A --compute-resource 'Management CL' --name VCH01

in my case i had to mention the thumbprint as it wouldn't accept --force also i had to mention which compute-resource the VCH machine is running on since i had more than one cluster.

but again i couldn't use the root and the password in Admiral as a credential option as it generated an error saying " A valid key pair was not found". it seems it only works with the certificate option for credentials to be able to add the host.

[mlh78750]

For adding a VCH to Admiral you need to use the keys and not user/password auth. Take a look at the docs here: https://github.com/vmware/vic/blob/master/doc/user_doc/vic_installation/vch_installer_options.md#security

Specifically you should be able to use cert.pem key.pem to setup the auto to the docker endpoint. The user/password option will not work with VIC Engine and is only used with Docker hosts.

[mlh78750]

cc @mreferre

[vmwarelab]

@mlh78750 Thank you . i have looked at the instruction you mentioned around adding a host in admiral this paragraph specifically

"Since we have instantiate the VCH with the --no-tls flag, there is no need to configure the credentials. Should you have instantiated it with proper certificates you would have needed to configure the certificates based credentials in Admiral before connecting to the VCH."

1. if you i use --no-tls the statement say there is no need to configure credentials which is somewhat true but you can't verify connection or add a host if the credential field is not populated . so if you do create one using any credentials you want ( don't have to be the right credentials ) i can add the host using port 2375 which isnt secure. thats option 1
2. that leaves us with the --no-tlsverify and --tls-cname deployment options which seems to both requires certificate key pair to be able to add the host in admiral.

with --no-tlsverify

there are no certificate generated even though it mention generating a self signed certificate /keypair - private in ./VCH01/key.pem which doesn't really get created in the folder specified .

if i set the root password using the debug command and i did and use the username option for credentials . it still fails asking for a valid key pair.

with --tls-cname

there is a group of certificate that i listed in my initial post . but my questions which one i m suppose to use for the certificate credential options ?

[vmwarelab]

@mlh78750 if i do use the cert and key .pem files generated with using the --tls-cname parameter and use it as credentials i get the following error when adding the host

Socket channel closed:xn-cnx-tag-default:vch01.vmwarelab.local:2376

its just not clear in any documentation what type of VCH deployment you need to have to be able to add it as a host in Admiral . the only one worked so far as i mentioned if i use --no-tls and choose some random password to use as credentials on port 2375 .. buts i m trying to do add a host using port 2376 which is more secure i m assuming and i m failing to do so.

so many moving parts here making it very complex . it was working for me when i was using the certs generated in 0.6.0 but then had other issues.

[mlh78750]

cc: @sflxn

I actually have not tried this recently, it is possible there is an issue with Admiral. Are you using the latest release of Admiral as well?

[vmwarelab]

@mlh78750 not sure i m using admiral that is being shipped with vRA 7.1 Tech preview feature . i didn't install admiral my self as a stand alone.

[hickeng]

@vmwarelab

Certificates in general

A certificate is made up of two parts and I try to refer to those two parts as a single entity, but do fall into plurals on occasion:

• public certificate portion (given out to anyone who asks)
• private key (kept secret)

The naming conventions are as follows for paired certificate and key files: cert.pem & key.pem <prefix>.pem & <prefix>-key.pem <prefix>-cert.pem & <prefix>-key.pem

Docker and certificate usage

There are four certificates in play with a tlsverify configuration:

1. client certificate (held by the client)
2. server certificate (held by the server)
3. certificate authority (CA) used to sign the server certificate
4. certificate authority used to sign the client certificate (held by the server)

As a convenience, vic-machine generates one CA that is used for both (3) and (4), and uses that to sign both (1) and (2). This means that the clients must be given the public part of the CA in order to be able to trust the server, as well as the client certificate, for each VCH they use (unless sharing access across VCHs).

In a production deployment I would expect: (3) to be signed by a company certificate, rooted by a public or corporate trust authority (e.g. verisign) (4) to be unique per client or group of clients - this allows each client to have a unique certificate, but for the group to be authorized as a whole. This could be one per VCH, multiple per VCH, or one per group of VCHs.

DOCKER_CERT_PATH

Docker requires the following names to be used if certificates are to be automatically consumed from DOCKER_CERT_PATH on the client machine: cert.pem key.pem - the client certifcate (1) server-cert.pem server-key.pem - the server certificate (2) ca.pem - the public portion of the certificate authority that signed the servers certificate (3)

vic-machine TLS options:

--no-tls: this is a horrible deployment choice for anything but a lab but was common prior to the addition of the tlsverify mechanism in 0.7.0 - hopefully Admiral will be updated soon to allow for the tlsverify configuration

--no-tlsverify: the certificate generated with this option is a server certificate and should exist on the VCH endpoint VM and should not be needed by Admiral - the fact it's not saved in 0.7.0 is entirely my fault (see PR at the bottom of this comment). This certificate allows the client to confirm the servers identity (as with banking websites, etc) so long as the client can validate the certificate chain - this may require the CA be provided to the client if using self-signed certificates. The server certificate files should now be named as expected by the docker client. This does not require the client to verify it's identity for the server.

--tls-cname: supplying this option also creates a certificate authority and client certificates signed by that authority so allow the server (endpointVM) to confirm the clients identity. The client certificate files should be the ones that Admiral requires.

• ca.pem (used on the server to confirm a client is authorized)
• cert.pem - (client certificate - should be in DOCKER_CERT_PATH on the client machine)
• key.pem - (the private key only the client needs - should be in DOCKER_CERT_PATH)

If generating authority (CA), client, and server certificates then the same CA is used for both client and server. In this case the ca.pem file needs to be: a. placed in DOCKER_CERT_PATH or specified on the docker client command line b. registered with a browser in order to access the vic-admin log server on port 2378

Commentary

Just after 0.7.0 was tagged we added an additional authentication mechanism for the vic-admin server in tlsverify mode - authentication via vSphere credentials in addition to the client certificate option.

This PR fixes the missing certificates in the --no-tlsverify case, and simplifies re-use of previously generated certificates if present: https://github.com/vmware/vic/pull/3141

Regarding username/password - those are useless to Admiral as credentials. The endpoint VM provides no CLI mechanism to interact with or administer the API, even once SSH is enabled - the configuration for the endpoint is done in a 12factor model and persisted in vSphere infrastructure, not in the guest.

[hickeng]

@vmwarelab I made some significant edits to the previous comment - so don't rely on an emailed copy to read it.

[hickeng]

@stuclem It occurs to me that this might be useful for you as well

[vmwarelab]

@hickeng Great write up. if i understood this correctly and please bare with me here. are you saying that the only option we can work with today is to use the --no-tls to successfully add a VCH 0.7.0 as a host in admiral ?

When i was using 0.6.0 release there was --no-tls options but yet there was 1 certificate (cert, key) generated/created in the same location where the vic-machine command was executed that i used for certificate type credentials to add the VCH host in Admiral which worked fine then.

if what i said is correct, whats the plan going forward .. will all 3 tls options be supported to add VCH as hosts in admiral

thank you

Maher

[stuclem]

@hickeng yes the write up above is extremely useful. Thank you!

[vmwarelab]

@hickeng Hi Georage .. any comments on my last post .. thank you

[hickeng]

@vmwarelab @chengwang86 has confirmed that admiral currently allows for adding a VCH (v0.8) configured with either --no-tls or with the TLSVERIFY setup.