We need to be able to bump neonKUBE version numbers at will; we've been stuck at neonkube-0.1.0-alpha since the beginning of time. There are some problems:
[x] We've accumulated multiple version constants over time in both of the neonKUBE and neonCLOUD repos. We need to consolidate these into a single constant.
[x] openebs-cstor-csi-driver setup image build is failing
We implemented multi-part image uploads as GitHub releases and this was really nice because we did MD5 checks top ensure that the latest version was always downloaded. Unfortunately, we ran into GitHub rate limits when because cluster setup will not have the user's GitHub credentials available. So we've reverted to single-part S3 images. This has been causing lots of confusion because it's very easy to forget to clear the %USERPROFILE%\.neonkube\vm-images folder when deploying clusters.
The solution is to generalize the multi-part downloads so these can be hosted on S3 as well:
[x] Neon.Deployment: Add an AwsCli.UploadMultipartFile() method that uploads a multi-part file to S3 and returns a Download instance. This will work much the same as GitHubReleaseApt.UploadMultipartAsset().
[x] Add a unit tests
[x] Generalize the GitHubReleaseApi.DownloadAsync() method by relocating it to DeploymentHelper() since this not GitHub Releases specific any more.
[x] Let's deprecate the S3 downloads folder and relocate the the download JSON file as well as the part files to the vm-images folder. This will look like:
[x] neonCLOUD image publish.ps1 script need to publish the neonKUBE base and service images too
[x] neonKUBE publish.ps1 script is not building neon-acme and neon-cluster-api containers
[x] neon-image build containers needs verify that all of the containers exist in the GHCR registry with the correct tad and are visible to everyone.
JEFF: This ended up being a hardcoded container image tag
[x] The neon-cluster-api service is not deploying correctly. This container image probably hasn't been rebuild for some time due to the first misc issue above. We need to remove this service and related database from cluster install and replace it with a CRD and a refactored neon-cluster-operator.
[x] Note that we're going to change the file type for the download from .gz (which is confusing because it isn't actually gzipped) to gz.manifest. We'll need tweak how image building and cluster prepare works to so that the image name after downloading ends in .gz. We may already be doing this via the Download.Filename property.
[x] Remove the GitHub neon-image publication options
jefflill
commented
2 years ago
We need to be able to bump neonKUBE version numbers at will; we've been stuck at neonkube-0.1.0-alpha since the beginning of time. There are some problems:
[x] We've accumulated multiple version constants over time in both of the neonKUBE and neonCLOUD repos. We need to consolidate these into a single constant.
[x] openebs-cstor-csi-driver setup image build is failing
We implemented multi-part image uploads as GitHub releases and this was really nice because we did MD5 checks top ensure that the latest version was always downloaded. Unfortunately, we ran into GitHub rate limits when because cluster setup will not have the user's GitHub credentials available. So we've reverted to single-part S3 images. This has been causing lots of confusion because it's very easy to forget to clear the
%USERPROFILE%\.neonkube\vm-imagesfolder when deploying clusters.The solution is to generalize the multi-part downloads so these can be hosted on S3 as well:
[x] Neon.Deployment: Add an
AwsCli.UploadMultipartFile()method that uploads a multi-part file to S3 and returns aDownloadinstance. This will work much the same asGitHubReleaseApt.UploadMultipartAsset().[x] Generalize the
GitHubReleaseApi.DownloadAsync()method by relocating it toDeploymentHelper()since this not GitHub Releases specific any more.[x] Let's deprecate the S3
downloadsfolder and relocate the the download JSON file as well as the part files to thevm-imagesfolder. This will look like:[x] Misc issues
JEFF: This ended up being a hardcoded container image tag
[x] The neon-cluster-api service is not deploying correctly. This container image probably hasn't been rebuild for some time due to the first misc issue above. We need to remove this service and related database from cluster install and replace it with a CRD and a refactored neon-cluster-operator.
[x] Note that we're going to change the file type for the download from .gz (which is confusing because it isn't actually gzipped) to gz.manifest. We'll need tweak how image building and cluster prepare works to so that the image name after downloading ends in .gz. We may already be doing this via the
Download.Filenameproperty.[x] Remove the GitHub neon-image publication options
[ ] neon-image multi-part support