search

Alluxio / k8s-operator

An operator for managing Alluxio system on Kubernetes cluster
https://www.alluxio.io/
Apache License 2.0
11 stars 9 forks source link

Use load on dora #21

Closed ssz1997 closed 1 year ago

ssz1997 commented 1 year ago

Update the command to use the new load; The new load is async, so use a script to periodically check the status.

ssz1997 commented 1 year ago

@Kai-Zhang PTAL. Haven't done any test because the load command is not in Dora branch yet.

ssz1997 commented 1 year ago

@Xenorith I don't have a way to do testing now because we not only don't have the load command ready, but also don't have any similar async command that can check progress. Put this aside, since we need an image that has alluxio with go installed, I just install go to the alluxio image and get a new image. This new image will be used for all future jobs (for example, metadata sync). What do you think of the approach? I remember you are not in favor of a tons of images so want to get some input.

Xenorith commented 1 year ago

any issues with installing go as part of the default, so it would be included in all images? i dont think the image size will increase by much proportionally; the installation tarball is about 100MB. i definitely don't want to create a new set of images

ssz1997 commented 1 year ago

I don't think there's any issue, which is why I'm not sure. There's only one image being created though, not a set of images. Different scripts will not use different images.

ssz1997 commented 1 year ago

go is installed in this PR: https://github.com/Alluxio/alluxio/pull/17444

ssz1997 commented 1 year ago

@Xenorith @Kai-Zhang PTAL. Thanks!

ssz1997 commented 1 year ago

Blocked by a bug while testing

ssz1997 commented 1 year ago

@Xenorith PTAL at the latest go script. Thanks! @Kai-Zhang This is how we add the go script into the operator container, and when we need to load the data, we create a job w/ Alluxio image and run the script inside the container. PTAL. Thanks!