terascope / teraslice

Scalable data processing pipelines in JavaScript
https://terascope.github.io/teraslice/
Apache License 2.0
50 stars 13 forks source link

Introduce faster build times for kubernetes development #3664

Open sotojn opened 2 weeks ago

sotojn commented 2 weeks ago

As of right now the development workflow for kubernetes is as follows:

  1. Run yarn k8s to build teraslice image and create all resources needed in kindcluster
  2. Test build
  3. Make a change to teraslice code
  4. Run yarn k8s:rebuild which triggers the following: 4.1. Deletes teraslice deployment resource 4.2. Rebuilds teraslice image and re-uploads to kubernetes control plane 4.3. Re-creates teraslice deployment resource
  5. Repeat starting from step 2

My proposal is optimizing steps 1 and 4 because building the docker image takes up a significant amount of time. Build times can take anywhere from 1-2 minutes each time not including the time it takes to redeploy the resource and upload the image to kubernetes.

If we instead give the master deployment the base image and provide a volume of the host machines repo, we can completely skip the build step and make instant changes with build:watch running. Perhaps we can also change yarn k8s:rebuild to simply restart the master pod which would re-trigger the node service command.