Error creating example functions

rctunisi commented 6 years ago

hello!

ive got an issue after successfully deploying dispatch and trying to create my first example functions following the quickstart tutorial

dispatch create seed-images

dispatch get base-images NAME | URL | STATUS | CREATED DATE

dispatch get images NAME | URL | BASEIMAGE | STATUS | CREATED DATE

all fine until i used seed.yaml script to create the functions

dispatch get functions NAME | FUNCTIONIMAGE | STATUS | CREATED DATE

after got this errors, i went back to kubectl to check the pods and got this

kubectl get po -n openfaas NAME READY STATUS RESTARTS AGE alertmanager-bccc88d4-n9hhp 1/1 Running 1 1h faas-netesd-5f758fccb7-mscsv 1/1 Running 1 1h gateway-598d6bd8-hcc6h 1/1 Running 4 1h nats-86955fb749-kkbbg 1/1 Running 1 1h of-0a44d502-3198-4df5-86f5-9955eeafd26f-c854f8959-jd2lf 0/1 ImagePullBackOff 0 12m of-44848f44-344e-4d3f-b1e5-a1b638c5e574-6f95bc6dd7-zdvrl 0/1 ImagePullBackOff 0 12m of-630394b9-17fd-4bb3-a057-ec0228af6e25-546fbbff65-bsc2h 0/1 ImagePullBackOff 0 12m of-d282b5ed-515b-4103-b1ad-ebc56f39b974-6f5cb889f5-pn2zt 0/1 ImagePullBackOff 0 12m prometheus-b996ddfcb-p7vqv 1/1 Running 1 1h queue-worker-6674f9f946-z6w5l 1/1 Running 3 1h

this is the output of kubectl logs from one of those ImagePullBackOff pods

Error from server (BadRequest): container "of-0a44d502-3198-4df5-86f5-9955eeafd26f" in pod "of-0a44d502-3198-4df5-86f5-9955eeafd26f-c854f8959-jd2lf" is waiting to start: trying and failing to pull image

any hints on whats going on??

minikube version: v0.28.2 (running kubernetes 1.10 and --vm-driver=none) dispatch version Client: v0.1.23 / Server: v0.1.23

berndtj commented 6 years ago

Well, my guess is that the k8s node cannot pull from the insecure docker registry that we are using. I haven't tried your exact setup centOS + minikube, but I know this is often an issue and will give the same results. The fix is to configure the docker server to allow insecure registries across the cluserIP space: 10.0.0.0/8 will work.

See the documentation here:

https://vmware.github.io/dispatch/documentation/guides/eks#insecure-registries

And let us know if that works. The other work around would be to configure dispatch to use dockerhub as the registry. For instance:

dispatch:
  # A hostname for the dispatch API, set this in Route53 when prompted
  host: host.example.com
  port: 443
  tls:
    ca: letsEncrypt
  faas: openfaas
  eventTransport: kafka
  imageRegistry:
    name: dockerhub-org
    username: dockerhub-user
    password: **********

rctunisi commented 6 years ago

@berndtj thanks for the reply! the docker daemon config did the trick

add "insecure-registries": ["10.0.0.0/8"] to /etc/docker/daemon.json
systemctl restart docker
get bored of all the errors and crashing because i forgot to minikube stop before doing that!
reboot the box
openfaas function pods were all running \o/ (after a few less pod crashing)

openfaas      of-0a44d502-3198-4df5-86f5-9955eeafd26f-c854f8959-jd2lf    1/1       **Running**   0          15h
openfaas      of-44848f44-344e-4d3f-b1e5-a1b638c5e574-6f95bc6dd7-zdvrl   1/1       **Running**   0          15h
openfaas      of-630394b9-17fd-4bb3-a057-ec0228af6e25-546fbbff65-bsc2h   1/1       **Running**   0          15h
openfaas      of-d282b5ed-515b-4103-b1ad-ebc56f39b974-6f5cb889f5-pn2zt   1/1       **Running**   0          15h

however, functions were still marked with ERROR on dispatch get functions

  hello-py  | 10.104.180.36:5000/func-630394b9-17fd-4bb3-a057-ec0228af6e25:latest | **ERROR**  | Sat Sep  1 16:03:18 -03 2018
  http-py   | 10.104.180.36:5000/func-0a44d502-3198-4df5-86f5-9955eeafd26f:latest | **ERROR**  | Sat Sep  1 16:03:18 -03 2018
  hello-js  | 10.104.180.36:5000/func-44848f44-344e-4d3f-b1e5-a1b638c5e574:latest | **ERROR**  | Sat Sep  1 16:03:18 -03 2018
  hello-ps1 | 10.104.180.36:5000/func-d282b5ed-515b-4103-b1ad-ebc56f39b974:latest | **ERROR**  | Sat Sep  1 16:03:18 -03 2018

my solution to that was running a dispatch update on the seed.yaml file, which made the functions re-create and then become ready

cd /dispatch/examples
dispatch update --file seed.yaml

dispatch get functions

    NAME    |                            FUNCTIONIMAGE                            | STATUS |         CREATED DATE
----------------------------------------------------------------------------------------------------------------------
  http-py   | 10.104.180.36:5000/func-e46e025a-c604-4d03-bc76-a8dab815f2dc:latest | READY  | Sat Sep  1 16:03:18 -03 2018
  hello-py  | 10.104.180.36:5000/func-3b2631be-b588-46cb-af8c-8afe5ce16259:latest | READY  | Sat Sep  1 16:03:18 -03 2018
  hello-js  | 10.104.180.36:5000/func-58a07c63-e00d-4546-a75e-ca2fbd76ea5d:latest | READY  | Sat Sep  1 16:03:18 -03 2018
  hello-ps1 | 10.104.180.36:5000/func-c8e8b71b-af3c-4c0f-83b5-68d4d899ef8a:latest | READY  | Sat Sep  1 16:03:18 -03 2018

berndtj commented 6 years ago

I'm glad! We're aware that the documentation isn't in a very consistent state right now. We're workong on this now (particularly around events).

vmware-archive / dispatch