search

CatalystCode / project-fortis-pipeline

Project Fortis is a data ingestion, analysis and visualization pipeline.
Apache License 2.0
14 stars 9 forks source link

Installer creates extra VM and leaves it running #248

Closed xtophs closed 6 years ago

xtophs commented 6 years ago

Installer creates a VM to run installation scripts from inside the subscription and then leaves the VM running indefinitely creating charges.

Consider a remote scripting approach like https://github.com/xtophs/acs-k8s-cassandra-multi-dc/blob/master/scripts/deploy.sh#L154 or running scripts on the Kubernetes master like https://github.com/xtophs/k8s-setup-federation-cluster/blob/master/setup-cluster.sh#L63

c-w commented 6 years ago

This may be by design so that we have a jump-box that can be used to interact with the Fortis resources post-deployment. @erikschlegel can confirm.

xtophs commented 6 years ago

Why not use the k8s master as the jump box?

c-w commented 6 years ago

For more context:

The question now is whether it's possible to remove the need for this VM and use the k8s master instead, as Christoph suggested. Alternatively, we can also add documentation to the deployment that the user may tear down the FortisDeployVM after a successful deployment, but then we need to ensure that the k8s master is still accessible and we need to document how to connect to it.

Open to any suggestions in this area.

xtophs commented 6 years ago

Look at https://github.com/xtophs/acs-k8s-cassandra-multi-dc/blob/master/scripts/deploy-singledc.sh on how to deploy Cassandra via SSH without the need for an extra VM

NathanielRose commented 6 years ago

Following up, did a bit of research and it seems you're not supposed to go on the k8s master node. Seems a bit hacky and not best practice. Was there a reason why we couldn't use acs-engine? The binary generates the ARM template for the cluster and we can add the fortis repo as an extension for reference with a bash script on the cluster that executes the same script as fortis-deploy.sh. Currently investigating and will share a sample if successful.

xtophs commented 6 years ago

I’m using acs-engine https://github.com/xtophs/acs-k8s-cassandra-multi-dc/blob/master/scripts/deploy-singledc.sh. We may need custom node labels at some point, which I’m not sure ACS / AKS will support. I know acs-engine does (because I added them for DC/OS a while back). Switching to acs-engine generated templates would help. We could also commit the ARM template generated from acs-engine in the GitHub repo, which gives us a little bit more control over the cluster configuration.

A while back, I’ve experimented with nested ARM templates for Fortis, which allows to reference the acs-engine generated the template from the template that’s provisioning the Fortis resources today. I can dig for it if you’re interested.

I’m wondering what the concern about going on the master are. The approach above remotely executes config scripts via SSH. The security exposure is about the same, since admin name and public key are already used to connect to the master are already part of the ARM template. Switching to a script extension wouldn’t avoid that.

Thoughts?

NathanielRose commented 6 years ago

Node Labels are supported for kubernetes, it was added 6 months ago by Bhargav. So at first I was not seeing where the need for node labels would be utilized. In fortis-deploy.sh it seems we only define the node type counts (k8sSpark + k8sCassandra) which in this case would correspond to the agentpoolProfile count. Inside the profile I can define an acs extension to run. Would I be running @xtophs cassandra chart for one pool and the fortis spark config as another extension in another pool? May be a gap in the documents but open to general thoughts on this.

NathanielRose commented 6 years ago

Finalized approach discussed with @xtophs

  1. Remove vm extenstion to run fortis-deploy.sh (https://github.com/CatalystCode/project-fortis-pipeline/blob/master/azuredeploy.json#L472)
  2. Remove deployment VM from ARM template: https://github.com/CatalystCode/project-fortis-pipeline/blob/master/azuredeploy.json#L363
  3. Define acs-engine template with an extension in the masterprofile (https://github.com/Azure/acs-engine/blob/0224c051212524b2a71e00db8cf40783fe677359/docs/extensions.md#masterprofile) to run fortis-deploy.sh on a single master: https://github.com/Azure/acs-engine/blob/0224c051212524b2a71e00db8cf40783fe677359/docs/extensions.md
  4. Create ARM template using acs-engine (https://github.com/Azure/acs-engine/blob/e3b22c480c0298b2cf1dce41c4a7e87ac49f46f3/docs/kubernetes/deploy.md)
  5. Switch creation of k8s cluster to use acs-engine created for k8s cluster instead of cluster resource https://github.com/CatalystCode/project-fortis-pipeline/blob/master/azuredeploy.json#L439 a. Look at nested ARM templates like I started here: https://github.com/xtophs/project-fortis-pipeline/tree/multi-dc-deploy
  6. Remove az cli stuff from https://github.com/CatalystCode/project-fortis-pipeline/blob/master/fortis-deploy.sh a. Az cli is only needed to ssh to master to get kubeconfig to run kubectl b. Since you’re now running on the master, you already have kubeconfig
c-w commented 6 years ago

Thanks for the detailed plan @NathanielRose. How much effort do you estimate this will require?

c-w commented 6 years ago

Thanks Nate and Christoph for the investigation and plan forward. Given that this isn't a blocking issue for Fortis users right now, I've put it onto the backlog for the V.next: https://github.com/CatalystCode/project-fortis/issues/32