[discussion] Reduce the Overhead of Kubeflow

[gaocegege]

Now it takes about 10 seconds to set up the containers and run the distributed code (Docker push is not included). We need to try to reduce the overhead of Kubeflow for better UX.

/priority p3

[gaocegege]

Approaches:

• Docker registry in the internal network.
• Better image build tool.
• Lower query interval. (Now is 6s)

[gaocegege]

/kind design

[ddysher]

do you know time spent on each stage?

Now it takes about 10 seconds to set up the containers and run the distributed code (Docker push is not included). We need to try to reduce the overhead of Kubeflow for better UX.

[gaocegege]

• ~6s to create image and the TFJob
• ~6s to run and get the log from the TFJob

And it depends on the implementation, maybe. We set the query interval to 6s

[gaocegege]

I think it is acceptable now.