uber / fiber

Distributed Computing for AI Made Simple
https://uber.github.io/fiber/
Apache License 2.0
1.04k stars 108 forks source link

can't pull latest container image for pod process #16

Open cometta opened 4 years ago

cometta commented 4 years ago

when running in kubernetes, the pod/process-XXX does not pull the latest new image. May i know which file in this repo that use to create pod/process-XXX ?

calio commented 4 years ago

pod/process-XXX uses the same image as the master process. For the master process, you specify the image when you launch it.

Could you describe the issue a little bit more in detail? Things like how did you run your code, what docker image is expected what docker image is actually used would be very helpful.

cometta commented 4 years ago

i use this command to run the job

kubectl create job fiber-test --image=/fiber-testing -- python3 /root/run.py

let say first time i run this job with /fiber-testing , then i build and push the new image . it still use back the old container image

calio commented 4 years ago

After you build and push image, do you still use kubectl create job fiber-test --image=<hub_name>/fiber-testing -- python3 /root/run.py ?

Can verify that you launch fiber master process with the correct image? Seems more like to be running kubectl with wrong image tag or something.

cometta commented 4 years ago

@calio of course i ran the command again =) i know there is somewhere need to set this imagePullPolicy=Always, can you point me to the source code where it script use to create pod/process-xxx

calio commented 4 years ago

OK, here is the place where new pod is created https://github.com/uber/fiber/blob/master/fiber/kubernetes_backend.py#L117