search

goharbor / harbor

An open source trusted cloud native registry project that stores, signs, and scans content.
https://goharbor.io
Apache License 2.0
23.38k stars 4.69k forks source link

Pushing image to harbor with error: "Received unexpected HTTP status: 500 Internal Server Error" #3127

Closed windyon9 closed 6 years ago

windyon9 commented 6 years ago

Harbor version: v1.1.2 Docker version: 17.06.0-ce Docker-compose version: 1.8.0

My harbor run well for some time(handling well with login, pull, or push), and all of sudden, clients encounter failure of error message of "Received unexpected HTTP status: 500 Internal Server Error" when pushing NEW image, without any changes on harbor side, like config modification, or restarting etc. clients could still login, pull image or push a image that EXISTING in harbor.

docker client returns: Received unexpected HTTP status: 500 Internal Server Error

Related UI logs: Aug 29 10:43:59 172.18.0.1 ui[2646]: 2017-08-29T02:43:59Z [DEBUG] [token.go:35]: URL for token request: /service/token?account=admin&scope=repository%3Atest%2Fdebian%3Apush%2Cpull&service=harbor-registry Aug 29 10:43:59 172.18.0.1 ui[2646]: 2017-08-29T02:43:59Z [DEBUG] [creator.go:206]: scopes: [repository:test/debian:push,pull] Aug 29 10:43:59 172.18.0.1 ui[2646]: 2017-08-29T02:43:59Z [WARNING] Failed to get secret cookie, error: http: named cookie not present Aug 29 10:43:59 172.18.0.1 ui[2646]: 2017-08-29T02:43:59Z [DEBUG] [authenticator.go:59]: Current AUTH_MODE is db_auth

Related registry log: Aug 29 10:43:59 172.18.0.1 registry[2646]: time="2017-08-29T02:43:59.60763337Z" level=debug msg="authorizing request" go.version=go1.6 http.request.host=tcregistry2.starcloud.io http.request.id=639f28b5-9ad5-4bcf-a3c5-2cde36e7f0e6 http.request.method=GET http.request.remoteaddr=101.95.183.46 http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.5.3 git-commit/20f81dd kernel/4.0.4 os/linux arch/amd64" instance.id=c33d8b9a-ad30-4545-9e75-a5d045a8b81c service=registry version="v2.4.1+unknown" Aug 29 10:43:59 172.18.0.1 registry[2646]: time="2017-08-29T02:43:59.608096769Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.6 http.request.host=tcregistry2.starcloud.io http.request.id=639f28b5-9ad5-4bcf-a3c5-2cde36e7f0e6 http.request.method=GET http.request.remoteaddr=101.95.183.46 http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.5.3 git-commit/20f81dd kernel/4.0.4 os/linux arch/amd64" instance.id=c33d8b9a-ad30-4545-9e75-a5d045a8b81c service=registry version="v2.4.1+unknown" Aug 29 10:43:59 172.18.0.1 registry[2646]: 172.18.0.7 - - [29/Aug/2017:02:43:59 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/1.10.3 go/go1.5.3 git-commit/20f81dd kernel/4.0.4 os/linux arch/amd64"

please help take a look, thanks!

attaching logs files harbor_logs.tar.gz

ywk253100 commented 6 years ago 
Aug 29 10:43:59 172.18.0.1 registry[2646]: time="2017-08-29T02:43:59.797478234Z" level=error msg="response completed with error" auth.user.name=admin err.code=unknown err.detail="cos: qcloud API Error: RequestId: NTlhNGQ0ZWZfZWJhMDY4NjRfOWNlZF8xMjI2N2Y= Status Code: 404 Code: NoSuchKey Message: The specified key does not exist." err.message="unknown error" go.version=go1.6 http.request.host=tcregistry2.starcloud.io http.request.id=cc36e5a9-ff8b-45fd-bc2d-aacc531a6b7e http.request.method=HEAD http.request.remoteaddr=101.95.183.46 http.request.uri="/v2/test/debian/blobs/sha256:8a98b0bd9ee8b33ba99aac4942f25fac7f78a9748b4f73f3ec8c826f52fe7398" http.request.useragent="docker/1.10.3 go/go1.5.3 git-commit/20f81dd kernel/4.0.4 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=22.316999ms http.response.status=500 http.response.written=434 instance.id=c33d8b9a-ad30-4545-9e75-a5d045a8b81c service=registry vars.digest="sha256:8a98b0bd9ee8b33ba99aac4942f25fac7f78a9748b4f73f3ec8c826f52fe7398" vars.name="test/debian" version="v2.4.1+unknown" 
Aug 29 10:43:59 172.18.0.1 registry[2646]: 172.18.0.7 - - [29/Aug/2017:02:43:59 +0000] "HEAD /v2/test/debian/blobs/sha256:8a98b0bd9ee8b33ba99aac4942f25fac7f78a9748b4f73f3ec8c826f52fe7398 HTTP/1.1" 500 434 "" "docker/1.10.3 go/go1.5.3 git-commit/20f81dd kernel/4.0.4 os/linux arch/amd64"

According to the logs, I think the error below is the cause of the issue:

cos: qcloud API Error: RequestId: NTlhNGQ0ZWZfZWJhMDY4NjRfOWNlZF8xMjI2N2Y= Status Code: 404 Code: NoSuchKey Message: The specified key does not exist.
windyon9 commented 6 years ago

thanks ywk253100, we just verified it may due to storage access issue, we are going to investigate from the storage side, thanks!