Loftsman requires us to rename the helm chart directory to include version numbers

[agaur-lbl]

Hello,

We recently hit the issue where we tried to use loftsman ship command but it kept failing because it wasn't finding the directory it was expecting. This is the behaviour we saw:

We pulled down a helm chart from upstream:

helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update
helm pull vm//victoria-metrics-operator --version 0.1.10

This creates a victoria-metrics-operator-0.1.10.tgz file in the cwd.

After that we created a loftsman manifest shown below:

---
apiVersion: manifests/v1beta1
metadata:
  name: otg-victoria-metrics-operator
spec:
  charts:
  - name: victoria-metrics-operator
    namespace: otg-monitoring
    version: 0.1.10
    values:
      image:
        repository: registry.local/nersc/otg/operator

And then try to run loftsman ship command as shown:

loftsman ship --charts-path ./victoria-metrics-operator-0.1.10.tgz --manifest-path ./manifest.yaml

However this command fails because loftsman expects the directory unpacked from the .tar.gz file to contain version numbers.

Logs:

2021-03-30T14:01:28-07:00 INF Initializing the connection to the Kubernetes cluster using KUBECONFIG (system default), and context (current-context) command=ship
2021-03-30T14:01:28-07:00 INF Initializing helm client object command=ship
         |\
         | \
         |  \
         |___\      Shipping your Helm workloads with Loftsman
       \--||___/
  ~~~~~~\_____/~~~~~~~

2021-03-30T14:01:28-07:00 INF Ensuring that the loftsman namespace exists command=ship
2021-03-30T14:01:28-07:00 INF Loftsman will use the packaged charts at ./victoria-metrics-operator as the Helm install source command=ship
2021-03-30T14:01:28-07:00 INF Running a release for the provided manifest at ./manifest.yaml command=ship

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Releasing victoria-metrics-operator v0.1.10
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

2021-03-30T14:01:29-07:00 ERR Unable to find chart victoria-metrics-operator v0.1.10 in the configured charts location chart=victoria-metrics-operator command=ship namespace=otg-monitoring version=0.1.10
2021-03-30T14:01:29-07:00 INF Ship status: failed. Recording status, manifest, and log data to configmap loftsman-otg-victoria-metrics-operator in namespace loftsman command=ship

Encountered errors during the manifest release:

2021-03-30T14:01:29-07:00 ERR Unable to find chart victoria-metrics-operator v0.1.10 in the configured charts location chart=victoria-metrics-operator command=ship namespace=otg-monitoring version=0.1.10

We were able to get it working by unpacking the tar.gz file ourselves which yields a victoria-metrics-operator directory and then renaming the directory to victoria-metrics-operator-0.1.10. And then when we run the ship by pointing it to our renamed directory- the ship works.

This looks like a bug to me but please correct me if I am wrong. We would prefer this behaviour to work based on whatever directory the chart unpacks and not having us to rename it.

Let me know what you think.

[rockholla]

Hi there @agaur-lbl, the charts-path argument is a path to directory that contains one or many packaged charts as defined in your manifest. So, if you were using that way of running loftsman, you'd have some path like /tmp/charts, put your chart or charts into it, e.g. /tmp/charts/victoria-metrics-operator-0.1.10.tgz, And run something like loftsman ship --charts-path /tmp/charts --manifest-path=./manifest.yaml

Using loftsman --help, loftsman ship --help can be a really good source of help for documentation, understanding arguments and such, so know that is available to you.

Just let us know if you have other questions or need further help here.

$ loftsman ship --help
   _        __ _                                |\
  | | ___  / _| |_ ___ _ __ ___   __ _ _ __     | \
  | |/ _ \| |_| __/ __|  _   _ \ / _  |  _ \    |  \
  | | |_| |  _| |_\__ \ | | | | | |_| | | | |   |___\
  |_|\___/|_|  \__|___/_| |_| |_|\__,_|_| |_|  \--||___/
                                  ~~~~~~~~~~~~~~\_____/~~~~~~~~~~
Shipping will prep your cluster and then ship out your Helm charts to install or upgrade your workloads in the cluster

Usage:
  loftsman ship [flags]

Flags:
      --charts-path string            Local path to a directory containing helm-packaged charts, e.g. files like my-chart-0.1.0.tgz (required if not using charts-repo)
      --charts-repo string            The root URL for an external helm chart repo to use for installing/upgrading charts. (required if not using charts-path)
      --charts-repo-password string   The password for charts-repo, if applicable
      --charts-repo-username string   The username for charts-repo, if applicable
  -h, --help                          help for ship
      --manifest-path string          Local path to the Loftsman YAML manifest file, instruction on what charts to install and how to install them. See
                                      loftsman manifest --help for more info (required)

Global Flags:
      --helm-binary string          The Helm binary to use, helpful in being able to have Helm 3 installed alternatively (default "helm")
      --json-log-path string        Path to the file where JSON/machine-readable logs written, note that this is in addition to the stdout logs (default "/Users/patrickforce/workspace/loftsman/loftsman.log")
      --kube-context string         The name of the Kubernetes config context to use (default is the current-context in kubeconfig used)
      --kubeconfig string           Path to the Kubernetes config file to use (default is the system default)
      --loftsman-namespace string   The namespace where loftsman records are stored: manifests, logs, etc. (default "loftsman")```

[agaur-lbl]

Ah great! That clarifies it.

Doing this works:

 loftsman ship --charts-path . --manifest-path manifest.yaml

Thank you!

[agaur-lbl]

Hey there,

One more question- Can loftsman operate on an unpacked chart or it must be packaged? We often have unpacked charts that we need to look into before creating loftsman manifests. And in some cases we are writing our own helm charts for slurm etc- for those cases- it seems we first have to package them into an archive before loftsman can accept it into a --charts-path argument?

[rockholla]

That's correct, it only deals in packaged charts. At it's heart, Loftsman is a CI/CD tool, so dealing with standard Helm chart artifacts is what it's aimed at doing.

If you're actively developing a chart and you had some need to iterate on testing changes in a test cluster in a manual way or something, there would be a handful of options available to you for one-off applying some such chart-in-progress. Two that come to mind:

1. Just testing out an install/upgrade of the chart directly using helm
2. Packaging the test chart on the fly, something like a wrapper script or something

for unpackaged in ./test-charts/*; do
  helm package $unpackaged -d ./charts/
done
loftsman ship --charts-path ./charts --manifest-path ./manifest.yaml

something like that

[agaur-lbl]

Ah thank you again. I didn't know about helm package that sounds useful for us.

[rockholla]

you're welcome!