Panic when "null" labels are given in deployment

rienafairefr commented 4 years ago

Expected Behaviour

Function is created :-)

Current Behaviour

Function is not created :-( faasd / faasd-provider shows a panic

Sep 11 10:08:50 faasd faasd[12711]: 2020/09/11 10:08:50 [Deploy] request: {"service":"address-modified","image":"registry.gitlab.com/**/**/**/address-modified:openfaas","network":"","envProcess":"sha512sum","envVars":{"MAPBOX_API_KEY":"[redacted]"},"constraints":null,"secrets":null,"labels":null,"annotations":{"CreatedDate":"Mon Sep  3 07:15:55 BST 2018"},"limits":null,"requests":null,"readOnlyRootFilesystem":false}
Sep 11 10:08:51 faasd faasd[12719]: [faasd] proxy: http://10.62.0.6:8080/system/function/test-analysis
Sep 11 10:08:51 faasd faasd[12719]: 2020/09/11 10:08:51 Validated request 200.
Sep 11 10:08:51 faasd faasd[12719]: 2020/09/11 10:08:51 Forwarded [GET] to /system/function/test-analysis - [404] - 0.001787s seconds
Sep 11 10:08:51 faasd faasd[12711]: 2020/09/11 10:08:51 Deploy registry.gitlab.com/**/**/*/address-modified:openfaas size: 34971859
Sep 11 10:08:51 faasd faasd[12711]: 2020/09/11 10:08:51 http: panic serving 10.62.0.6:38720: runtime error: invalid memory address or nil pointer dereference
Sep 11 10:08:51 faasd faasd[12711]: goroutine 153253 [running]:
Sep 11 10:08:51 faasd faasd[12711]: net/http.(*conn).serve.func1(0xc000473860)
Sep 11 10:08:51 faasd faasd[12711]:         /home/travis/.gimme/versions/go1.13.linux.amd64/src/net/http/server.go:1767 +0x139
Sep 11 10:08:51 faasd faasd[12711]: panic(0xd4a000, 0x15bd3d0)
Sep 11 10:08:51 faasd faasd[12711]:         /home/travis/.gimme/versions/go1.13.linux.amd64/src/runtime/panic.go:679 +0x1b2
Sep 11 10:08:51 faasd faasd[12711]: github.com/openfaas/faasd/pkg/provider/handlers.deploy(0xfaaa00, 0xc0005637a0, 0xc00045ac60, 0x10, 0xc000c77f80, 0x55, 0x0, 0x0, 0xc00045ac70, 0x9, ...)
Sep 11 10:08:51 faasd faasd[12711]:         /home/travis/gopath/src/github.com/openfaas/faasd/pkg/provider/handlers/deploy.go:111 +0x867
Sep 11 10:08:51 faasd faasd[12711]: github.com/openfaas/faasd/pkg/provider/handlers.MakeDeployHandler.func1(0xfa6a40, 0xc0002c8fc0, 0xc000c78300)
Sep 11 10:08:51 faasd faasd[12711]:         /home/travis/gopath/src/github.com/openfaas/faasd/pkg/provider/handlers/deploy.go:58 +0x47f
Sep 11 10:08:51 faasd faasd[12711]: net/http.HandlerFunc.ServeHTTP(...)
Sep 11 10:08:51 faasd faasd[12711]:         /home/travis/.gimme/versions/go1.13.linux.amd64/src/net/http/server.go:2007
Sep 11 10:08:51 faasd faasd[12711]: github.com/openfaas/faasd/vendor/github.com/openfaas/faas-provider/auth.DecorateWithBasicAuth.func1(0xfa6a40, 0xc0002c8fc0, 0xc000c78300)
Sep 11 10:08:51 faasd faasd[12711]:         /home/travis/gopath/src/github.com/openfaas/faasd/vendor/github.com/openfaas/faas-provider/auth/basic_auth.go:28 +0x272
Sep 11 10:08:51 faasd faasd[12711]: net/http.HandlerFunc.ServeHTTP(0xc0002f2c80, 0xfa6a40, 0xc0002c8fc0, 0xc000c78300)
Sep 11 10:08:51 faasd faasd[12711]:         /home/travis/.gimme/versions/go1.13.linux.amd64/src/net/http/server.go:2007 +0x44
Sep 11 10:08:51 faasd faasd[12711]: github.com/openfaas/faasd/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc0002ca000, 0xfa6a40, 0xc0002c8fc0, 0xc000c78100)
Sep 11 10:08:51 faasd faasd[12711]:         /home/travis/gopath/src/github.com/openfaas/faasd/vendor/github.com/gorilla/mux/mux.go:212 +0xe2
Sep 11 10:08:51 faasd faasd[12711]: net/http.serverHandler.ServeHTTP(0xc0002c81c0, 0xfa6a40, 0xc0002c8fc0, 0xc000c78100)
Sep 11 10:08:51 faasd faasd[12711]:         /home/travis/.gimme/versions/go1.13.linux.amd64/src/net/http/server.go:2802 +0xa4
Sep 11 10:08:51 faasd faasd[12711]: net/http.(*conn).serve(0xc000473860, 0xfaa940, 0xc000c9da00)
Sep 11 10:08:51 faasd faasd[12711]:         /home/travis/.gimme/versions/go1.13.linux.amd64/src/net/http/server.go:1890 +0x875
Sep 11 10:08:51 faasd faasd[12711]: created by net/http.(*Server).Serve
Sep 11 10:08:51 faasd faasd[12711]:         /home/travis/.gimme/versions/go1.13.linux.amd64/src/net/http/server.go:2927 +0x38e
Sep 11 10:08:51 faasd faasd[12719]: 2020/09/11 10:08:51 error with upstream request to: /system/functions, Post http://faasd-provider:8081/system/functions: EO

Your Environment

OS and architecture: deploying the functions through terraform-provider-openfaas, but the error is the same with direct API on the gateway faasd is deployed on a digitalocean droplet with the cloud-config template https://raw.githubusercontent.com/openfaas/faasd/master/docs/bootstrap/digitalocean-terraform/cloud-config.tpl
Configurations the docker auth config was copied
```
root@faasd:/var/lib/faasd# cat /var/lib/faasd/.docker/config.json 
{
"auths": {
    "registry.gitlab.com": {
        "auth": "*****"
    }
}
}
```
the auth is identical to ~/.docker/config.json on my laptop, which can pull the registry.gitlab.com/**/**/**/address-modified:openfaas image without issues.
Versions:

go version
go version go1.10.4 linux/amd64

containerd -version
containerd github.com/containerd/containerd v1.3.5 9b6f3ec0307a825c38617b93ad55162b5bb94234

uname -a
Linux faasd 4.15.0-115-generic #116-Ubuntu SMP Wed Aug 26 14:04:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Interesting

not sure of where (before, after, during) in the deploy this happens, maybe it's unrelated, but I see this in the logs of faasd:

Sep 11 10:07:34 faasd faasd[12711]: 2020/09/11 10:07:34 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 400 Bad Request\nCache-Control: no-cache\nConnection: close\nContent-Type: text/html\n\n<!DOCTYPE html>\n<html>\n<head>\n  <meta content=\"width=device-width, initial-scale=1, maximum-scale=1\" name=\"viewport\">\n  <title>400 Bad Request</title>\n  <style>\n    body {\n      color: #666;\n      text-align: center;\n      font-family: \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n      margin: auto;\n      font-size: 14px;\n    }\n\n    h1 {\n      font-size: 56px;\n      line-height: 100px;\n      font-weight: normal;\n      color: #456;\n    }\n\n    h2 {\n      font-size: 24px;\n      color: #666;\n      line-height: 1.5em;\n    }\n\n    h3 {\n      color: #456;\n      font-size: 20px;\n      font-weight: normal;\n      line-height: 28px;\n    }\n\n    hr {\n      max-width: 800px;\n      margin: 18px auto;\n      border: 0;\n      border-top: 1px solid #EEE;\n      border-bottom: 1px solid white;\n    }\n\n    img {\n      max-width: 40vw;\n    }\n\n    .container {\n      margin: auto 20px;\n    }\n  </style>\n</head>\n\n<body>\n  <h1>\n    <img src=\"data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMjEwIiBoZWlnaHQ9IjIxMCIgdmlld0JveD0iMCAwIDIxMCAyMTAiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CiAgPHBhdGggZD0iTTEwNS4wNjE0IDIwMy42NTVsMzguNjQtMTE4LjkyMWgtNzcuMjhsMzguNjQgMTE4LjkyMXoiIGZpbGw9IiNlMjQzMjkiLz4KICA8cGF0aCBkPSJNMTA1LjA2MTQgMjAzLjY1NDhsLTM4LjY0LTExOC45MjFoLTU0LjE1M2w5Mi43OTMgMTE4LjkyMXoiIGZpbGw9IiNmYzZkMjYiLz4KICA8cGF0aCBkPSJNMTIuMjY4NSA4NC43MzQxbC0xMS43NDIgMzYuMTM5Yy0xLjA3MSAzLjI5Ni4xMDIgNi45MDcgMi45MDYgOC45NDRsMTAxLjYyOSA3My44MzgtOTIuNzkzLTExOC45MjF6IiBmaWxsPSIjZmNhMzI2Ii8+CiAgPHBhdGggZD0iTTEyLjI2ODUgODQuNzM0Mmg1NC4xNTNsLTIzLjI3My03MS42MjVjLTEuMTk3LTMuNjg2LTYuNDExLTMuNjg1LTcuNjA4IDBsLTIzLjI3MiA3MS42MjV6IiBmaWxsPSIjZTI0MzI5Ii8+CiAgPHBhdGggZD0iTTEwNS4wNjE0IDIwMy42NTQ4bDM4LjY0LTExOC45MjFoNTQuMTUzbC05Mi43OTMgMTE4LjkyMXoiIGZpbGw9IiNmYzZkMjYiLz4KICA8cGF0aCBkPSJNMTk3Ljg1NDQgODQuNzM0MWwxMS43NDIgMzYuMTM5YzEuMDcxIDMuMjk2LS4xMDIgNi45MDctMi45MDYgOC45NDRsLTEwMS42MjkgNzMuODM4IDkyLjc5My0xMTguOTIxeiIgZmlsbD0iI2ZjYTMyNiIvPgogIDxwYXRoIGQ9Ik0xOTcuODU0NCA4NC43MzQyaC01NC4xNTNsMjMuMjczLTcxLjYyNWMxLjE5Ny0zLjY4NiA2LjQxMS0zLjY4NSA3LjYwOCAwbDIzLjI3MiA3MS42MjV6IiBmaWxsPSIjZTI0MzI5Ii8+Cjwvc3ZnPgo=\" alt=\"GitLab Logo\" /><br />\n    400\n  </h1>\n  <div class=\"container\">\n    <h3>Your browser sent an invalid request.</h3>\n    <hr />\n    <p>Please contact your GitLab administrator if you think this is a mistake.</p>\n  </div>\n</body>\n</html>\n<html>\n"; err=<nil>

Which is a 400 bad request HTML from gitlab, so faasd or faasd-provider is making a request to the registry with wrong informations

alexellis commented 4 years ago

/set title: Error pulling from GitLab.com private registry

alexellis commented 4 years ago

Hi @rienafairefr I'm not a GitLab.com user, but I wonder if they have any known issues with containerd? Until recently even GitHub's registry didn't work with containerd and faasd is built with containerd.

go version go1.10.4 linux/amd64

I am noticing that you have Go installed, and a really old version. Why is that? You don't need it to install faasd, but if you built it you must use the minimum version we recommend.

terraform-provider-openfaas

I've not used this provider, perhaps you could use faas-cli deploy/up to confirm that it's working as expected?

I see a large size here, is your disk running out of space? size 34971859

"envVars":{"MAPBOX_API_KEY":"

You may prefer using a secret to env-vars for anything confidential. Do you need to scrub this key from the GitHub issue?

Private registry support is tested and working with the Docker Hub and GitHub's container registry. Maybe @AkihiroSuda has some ideas? @MrSimonEmms also has a GitLab account and may be able to help us test / verify with one of his images.

Alex

alexellis commented 4 years ago

One thing that stands out to me is how many / slashes you have in your image name. It could be an edge case where some part of the code is expecting a standard image URL i.e. server.com/repo/image:version (x3) instead of registry.gitlab.com/**/**/**/address-modified:openfaas (x5)

rienafairefr commented 4 years ago

One thing that stands out to me is how many / slashes you have in your image name. It could be an edge case where some part of the code is expecting a standard image URL i.e. server.com/repo/image:version (x3) instead of registry.gitlab.com/**/**/**/address-modified:openfaas (x5)

That might be the edge case I'm running into, and why I explicitely didn't redact that detail. In our case, it's a namespace thing for all our images:

registry.gitlab.com/company/project/functions/functiona

registry.gitlab.com/company/project/functions/functionb

registry.gitlab.com/company/project/serviceA etc ..., because in gitlab you can create groups inside groups, etc, and the docker registry attached to a project gets the same namespace (project is at https://gitlab.com/company/project/functions, then the registry is gitlab.com/company/project/functions)

mrsimonemms commented 4 years ago

@alexellis the number slashes look normal - GitLab allows you to have up to three levels on the container registry (which is very useful).

I'll have a dig around later today and let you know what I find

alexellis commented 4 years ago

@rienafairefr can you try a top-level image to see if it still breaks that way?

rienafairefr commented 4 years ago

Hi @rienafairefr I'm not a GitLab.com user, but I wonder if they have any known issues with containerd? Until recently even GitHub's registry didn't work with containerd and faasd is built with containerd.

go version go1.10.4 linux/amd64

I am noticing that you have Go installed, and a really old version. Why is that? You don't need it to install faasd, but if you built it you must use the minimum version we recommend.

the go version is the one on the droplet, i didn't install it myself. might be installed as part of the faasd cloud-config, not sure. Might be better to update all that, that's for sure

terraform-provider-openfaas

I've not used this provider, perhaps you could use faas-cli deploy/up to confirm that it's working as expected?

Weirdly, faas deploy seems to work (details below)

I see a large size here, is your disk running out of space? size 34971859

"envVars":{"MAPBOX_API_KEY":"

You may prefer using a secret to env-vars for anything confidential. Do you need to scrub this key from the GitHub issue?

You're right, secret will be better, I'm still trying out ;-) I redacted the api_key, good catch !

Private registry support is tested and working with the Docker Hub and GitHub's container registry. Maybe @AkihiroSuda has some ideas? @MrSimonEmms also has a GitLab account and may be able to help us test / verify with one of his images.

Alex

Might not be about the image registry after all:

when the deploy fails (using the api or terraform) Sep 11 10:49:36 faasd faasd[12711]: 2020/09/11 10:49:36 [Update] request: {"service":"address-modified","image":"registry.gitlab.com/**/**/**/address-modified:openfaas","network":"","envProcess":"sha512sum","envVars":{"MAPBOX_API_KEY":"[redacted]"},"constraints":null,"secrets":null,"labels":null,"annotations":{"CreatedDate":"Mon Sep 3 07:15:55 BST 2018"},"limits":null,"requests":null,"readOnlyRootFilesystem":false}

when the deploy returns 200 ok (using faas deploy): Sep 11 10:51:28 faasd faasd[12711]: 2020/09/11 10:51:28 [Deploy] request: {"service":"address-modified","image":"registry.gitlab.com/**/**/**/address-modified:openfaas","network":"","envProcess":"node index.js","envVars":{},"constraints":null,"secrets":[],"labels":{},"annotations":{},"limits":null,"requests":null,"readOnlyRootFilesystem":false,"namespace":"openfaas-fn"} the images are the same, but the rest of the config has differences, not sure what's happenning

alexellis commented 4 years ago

So if faas-cli deploy works, it may make sense to raise an issue on the functions terraform provider, which is a third-party component.

I can help with the API call though. Can you share what you're writing into your curl statement? faas-cli deploy just creates a HTTP request, so I'm curious what's going show up here.

Let's format that JSON?

Your first example (terraform)

{
  "service": "address-modified",
  "image": "registry.gitlab.com/**/**/**/address-modified:openfaas",
  "network": "",
  "envProcess": "sha512sum",
  "envVars": {
    "MAPBOX_API_KEY": "[redacted]"
  },
  "constraints": null,
  "secrets": null,
  "labels": null,
  "annotations": {
    "CreatedDate": "Mon Sep 3 07:15:55 BST 2018"
  },
  "limits": null,
  "requests": null,
  "readOnlyRootFilesystem": false
}

Second example: (faas-cli):

{
  "service": "address-modified",
  "image": "registry.gitlab.com/**/**/**/address-modified:openfaas",
  "network": "",
  "envProcess": "node index.js",
  "envVars": {},
  "constraints": null,
  "secrets": [],
  "labels": {},
  "annotations": {},
  "limits": null,
  "requests": null,
  "readOnlyRootFilesystem": false,
  "namespace": "openfaas-fn"
}

alexellis commented 4 years ago

I suspect that the code handling something like labels may be expecting a value, where as the terraform provider is giving null instead.

alexellis commented 4 years ago

/set title: Panic when "null" labels are given in deployment

alexellis commented 4 years ago

I've found the cause of the issue - the last output and the JSON formatting helped with that.

PR #102 is up and when CI passes I'll release it.

For now, include a label in your terraform to get around the issue?

Alex

rienafairefr commented 4 years ago

This closes the issue, adding a label works for deployment ! I chased the most obvious 'difference' in the private registry but that wasn't it. Thanks @alexellis

alexellis commented 4 years ago

No problem. Here is your release https://github.com/openfaas/faasd/releases/tag/0.9.4

You may enjoy weekly updates on OpenFaaS including faasd and the other community projects via GitHub Sponsors - https://www.openfaas.com/support - click "Community Insiders Program"

openfaas / faasd