search

owncloud / ocis

:atom_symbol: ownCloud Infinite Scale Stack
https://doc.owncloud.com/ocis/next/
Apache License 2.0
1.41k stars 184 forks source link

settings & storage-system: runtime error: invalid memory address or nil pointer dereference #4961

Closed NexZhu closed 2 years ago

NexZhu commented 2 years ago

Describe the bug

I deployed the latest version owncloud/ocis:2.0.0-beta.8 with the official Helm chart and configured it to authenticate users with an OIDC provider Authelia instance, here's what happens:

  1. Open owncloud URL, redirected to Authelia.
  2. Login and clickAccept in the OAuth2 consent page.
  3. Redirected to https://owncloud.example.com/oidc-callback.html?code=a0j4L6_ZuH2QbH5mPFJJbCVLH-OO9jIJg5mscuxPSiw.oqHE9PYJP1SnVboJth46PyRES17YKo9FT2WyONQhSTg&scope=openid+profile+email&state=9d6a21e638474d40bdaab11ebfb27088
  4. Browser sent XHR request to https://owncloud.example.com/ocs/v1.php/cloud/user with header: authorization: Bearer p9UAAo2wIMfb8RLlN_f7dW9u5mpxUl76IfUH3jAMEos.4b8WcEj1mLZOen34w719wsjoJa5tpq_ArXvub4M6NrY, resulted in 500 status code.
  5. I'm redirected back to OAuth provider's consent page.

Error logs

settings pod:

2022-10-28T08:31:37Z DBG setup cors middleware allow_credentials=true allowed_headers="Authorization, Origin, Content-Type, Accept, X-Requested-With" allowed_methods="GET, POST, PUT, PATCH, DELETE, OPTIONS" allowed_origins=* service=settings
2022-10-28T08:33:38Z DBG bytes=219330 duration=3.629855 method=GET path=/settings.js proto=HTTP/1.1 request= service=settings status=200
2022-10-28T08:33:40Z ERR error initializing metadata client error="rpc error: code = Internal desc = runtime error: invalid memory address or nil pointer dereference" service=ocis
2022-10-28T08:33:40Z ERR error initializing metadata client error="rpc error: code = Internal desc = runtime error: invalid memory address or nil pointer dereference" service=ocis
2022-10-28T08:34:00Z DBG bytes=219330 duration=0.146869 method=GET path=/settings.js proto=HTTP/1.1 request= service=settings status=200
2022-10-28T08:34:11Z DBG bytes=219330 duration=0.188705 method=GET path=/settings.js proto=HTTP/1.1 request= service=settings status=200
2022-10-28T08:34:11Z ERR error initializing metadata client error="rpc error: code = Internal desc = runtime error: invalid memory address or nil pointer dereference" service=ocis
2022-10-28T08:34:11Z ERR error initializing metadata client error="rpc error: code = Internal desc = runtime error: invalid memory address or nil pointer dereference" service=ocis

storage-system pod:

2022-10-28T08:50:33Z INF user idp:"internal" opaque_id:"4735a501-2b40-4ddc-a238-58727bc72ad2" type:USER_TYPE_PRIMARY  authenticated pkg=rgrpc service=storage-system traceid=00000000000000000000000000000000
2022-10-28T08:50:33Z DBG unary code=OK end="28/Oct/2022:08:50:33 +0000" from=tcp://127.0.0.1:53452 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=681650 traceid=00000000000000000000000000000000 uri=/cs3.auth.provider.v1beta1.ProviderAPI/Authenticate user-agent=grpc-go/1.49.0
2022-10-28T08:50:33Z DBG unary code=OK end="28/Oct/2022:08:50:33 +0000" from=tcp://172.16.11.148:57038 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=1301450 traceid=00000000000000000000000000000000 uri=/cs3.gateway.v1beta1.GatewayAPI/Authenticate user-agent=grpc-go/1.49.0
2022-10-28T08:50:33Z DBG unary code=OK end="28/Oct/2022:08:50:33 +0000" from=tcp://127.0.0.1:52036 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=72744 traceid=00000000000000000000000000000000 uri=/cs3.permissions.v1beta1.PermissionsAPI/CheckPermission user-agent=grpc-go/1.49.0
goroutine 1609 [running]:
runtime/debug.Stack()
    runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
    runtime/debug/stack.go:16 +0x19
github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x3cb11e0, 0xc000e6bc80}, {0x38b5f40, 0x4fecfc0})
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/recovery/recovery.go:48 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x3cb11e0?, 0xc000e6bc80?}, {0x38b5f40?, 0x4fecfc0?}, 0xc001316558?)
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:29 +0x7b
panic({0x38b5f40, 0x4fecfc0})
    runtime/panic.go:838 +0x207
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Node).InternalPath(...)
    github.com/cs3org/reva/v2@v2.10.0/pkg/storage/utils/decomposedfs/node/node.go:465
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).CreateStorageSpace(0xc0000c2050, {0x3cb11e0, 0xc0010e4150}, 0xc012d160c0)
    github.com/cs3org/reva/v2@v2.10.0/pkg/storage/utils/decomposedfs/spaces.go:95 +0x4b0
github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).CreateStorageSpace(0xc000112000, {0x3cb11e0, 0xc0010e4150}, 0xc012d160c0)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/services/storageprovider/storageprovider.go:469 +0x49
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler.func1({0x3cb11e0, 0xc0010e4150}, {0x3b971c0?, 0xc012d160c0})
    github.com/cs3org/go-cs3apis@v0.0.0-20220818202316-e92afdddac6d/cs3/storage/provider/v1beta1/provider_api.pb.go:6441 +0x78
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x3cb11e0, 0xc0010e4060}, {0x3b971c0, 0xc012d160c0}, 0xc0014071c0, 0xc000c13fe0)
    go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.34.0/interceptor.go:325 +0x664
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0010e4060?}, {0x3b971c0?, 0xc012d160c0?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func1({0x3cb11e0, 0xc000e6bc80}, {0x3b971c0, 0xc012d160c0}, 0xc0014071c0, 0xc0014071e0)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/auth/auth.go:145 +0xa06
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?}, 0xc000edf2b8?, 0x4905b7?)
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x3cb11e0, 0xc000e6bc80}, {0x3b971c0, 0xc012d160c0}, 0xc0014071c0, 0xc001407220)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x3cb11e0, 0xc000e6bc80}, {0x3b971c0, 0xc012d160c0}, 0x1?, 0xc001407240)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/useragent/useragent.go:38 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x3cb11e0, 0xc000e6b620}, {0x3b971c0, 0xc012d160c0}, 0x5191050?, 0xc001407260)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/token/token.go:44 +0x158
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6b620?}, {0x3b971c0?, 0xc012d160c0?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x3cb11e0, 0xc000e6b110}, {0x3b971c0, 0xc012d160c0}, 0x7f1171c7c3a8?, 0xc001407280)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6b110?}, {0x3b971c0?, 0xc012d160c0?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x3cb11e0, 0xc000e6b110}, {0x3b971c0, 0xc012d160c0}, 0xc001325ac8?, 0x38b4c20?)
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbf
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler({0x3c434e0?, 0xc000112000}, {0x3cb11e0, 0xc000e6b110}, 0xc00029eb60, 0xc0010f2120)
    github.com/cs3org/go-cs3apis@v0.0.0-20220818202316-e92afdddac6d/cs3/storage/provider/v1beta1/provider_api.pb.go:6443 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000000b40, {0x3cbfd38, 0xc000cf1380}, 0xc000044480, 0xc0010f2570, 0x500e150, 0x0)
    google.golang.org/grpc@v1.49.0/server.go:1301 +0xb0b
google.golang.org/grpc.(*Server).handleStream(0xc000000b40, {0x3cbfd38, 0xc000cf1380}, 0xc000044480, 0x0)
    google.golang.org/grpc@v1.49.0/server.go:1642 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
    google.golang.org/grpc@v1.49.0/server.go:938 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
    google.golang.org/grpc@v1.49.0/server.go:936 +0x28a
2022-10-28T08:50:33Z ERR runtime error: invalid memory address or nil pointer dereference; stack: goroutine 1609 [running]:
runtime/debug.Stack()
    runtime/debug/stack.go:24 +0x65
github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x3cb11e0, 0xc000e6bc80}, {0x38b5f40, 0x4fecfc0})
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/recovery/recovery.go:50 +0x65
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x3cb11e0?, 0xc000e6bc80?}, {0x38b5f40?, 0x4fecfc0?}, 0xc001316558?)
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:29 +0x7b
panic({0x38b5f40, 0x4fecfc0})
    runtime/panic.go:838 +0x207
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Node).InternalPath(...)
    github.com/cs3org/reva/v2@v2.10.0/pkg/storage/utils/decomposedfs/node/node.go:465
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).CreateStorageSpace(0xc0000c2050, {0x3cb11e0, 0xc0010e4150}, 0xc012d160c0)
    github.com/cs3org/reva/v2@v2.10.0/pkg/storage/utils/decomposedfs/spaces.go:95 +0x4b0
github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).CreateStorageSpace(0xc000112000, {0x3cb11e0, 0xc0010e4150}, 0xc012d160c0)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/services/storageprovider/storageprovider.go:469 +0x49
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler.func1({0x3cb11e0, 0xc0010e4150}, {0x3b971c0?, 0xc012d160c0})
    github.com/cs3org/go-cs3apis@v0.0.0-20220818202316-e92afdddac6d/cs3/storage/provider/v1beta1/provider_api.pb.go:6441 +0x78
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x3cb11e0, 0xc0010e4060}, {0x3b971c0, 0xc012d160c0}, 0xc0014071c0, 0xc000c13fe0)
    go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.34.0/interceptor.go:325 +0x664
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0010e4060?}, {0x3b971c0?, 0xc012d160c0?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func1({0x3cb11e0, 0xc000e6bc80}, {0x3b971c0, 0xc012d160c0}, 0xc0014071c0, 0xc0014071e0)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/auth/auth.go:145 +0xa06
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?}, 0xc000edf2b8?, 0x4905b7?)
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x3cb11e0, 0xc000e6bc80}, {0x3b971c0, 0xc012d160c0}, 0xc0014071c0, 0xc001407220)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x3cb11e0, 0xc000e6bc80}, {0x3b971c0, 0xc012d160c0}, 0x1?, 0xc001407240)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/useragent/useragent.go:38 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x3cb11e0, 0xc000e6b620}, {0x3b971c0, 0xc012d160c0}, 0x5191050?, 0xc001407260)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/token/token.go:44 +0x158
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6b620?}, {0x3b971c0?, 0xc012d160c0?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x3cb11e0, 0xc000e6b110}, {0x3b971c0, 0xc012d160c0}, 0x7f1171c7c3a8?, 0xc001407280)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6b110?}, {0x3b971c0?, 0xc012d160c0?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x3cb11e0, 0xc000e6b110}, {0x3b971c0, 0xc012d160c0}, 0xc001325ac8?, 0x38b4c20?)
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbf
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler({0x3c434e0?, 0xc000112000}, {0x3cb11e0, 0xc000e6b110}, 0xc00029eb60, 0xc0010f2120)
    github.com/cs3org/go-cs3apis@v0.0.0-20220818202316-e92afdddac6d/cs3/storage/provider/v1beta1/provider_api.pb.go:6443 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000000b40, {0x3cbfd38, 0xc000cf1380}, 0xc000044480, 0xc0010f2570, 0x500e150, 0x0)
    google.golang.org/grpc@v1.49.0/server.go:1301 +0xb0b
google.golang.org/grpc.(*Server).handleStream(0xc000000b40, {0x3cbfd38, 0xc000cf1380}, 0xc000044480, 0x0)
    google.golang.org/grpc@v1.49.0/server.go:1642 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
    google.golang.org/grpc@v1.49.0/server.go:938 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
    google.golang.org/grpc@v1.49.0/server.go:936 +0x28a
 pkg=rgrpc service=storage-system traceid=00000000000000000000000000000000
2022-10-28T08:50:33Z ERR unary code=Internal end="28/Oct/2022:08:50:33 +0000" from=tcp://172.16.11.148:57040 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=1297155 traceid=00000000000000000000000000000000 uri=/cs3.storage.provider.v1beta1.ProviderAPI/CreateStorageSpace user-agent=grpc-go/1.49.0
2022-10-28T08:50:33Z DBG skipping auth method=/cs3.gateway.v1beta1.GatewayAPI/Authenticate pkg=rgrpc service=storage-system traceid=00000000000000000000000000000000
2022-10-28T08:50:33Z DBG skipping auth method=/cs3.auth.registry.v1beta1.RegistryAPI/GetAuthProviders pkg=rgrpc service=storage-system traceid=00000000000000000000000000000000
2022-10-28T08:50:33Z DBG unary code=OK end="28/Oct/2022:08:50:33 +0000" from=tcp://127.0.0.1:53450 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=53140 traceid=00000000000000000000000000000000 uri=/cs3.auth.registry.v1beta1.RegistryAPI/GetAuthProviders user-agent=grpc-go/1.49.0
2022-10-28T08:50:33Z DBG skipping auth method=/cs3.auth.provider.v1beta1.ProviderAPI/Authenticate pkg=rgrpc service=storage-system traceid=00000000000000000000000000000000
2022-10-28T08:50:33Z DBG skipping auth method=/cs3.gateway.v1beta1.GatewayAPI/GetUserByClaim pkg=rgrpc service=storage-system traceid=00000000000000000000000000000000
2022-10-28T08:50:33Z DBG skipping auth method=/cs3.identity.user.v1beta1.UserAPI/GetUserByClaim pkg=rgrpc service=storage-system traceid=00000000000000000000000000000000
2022-10-28T08:50:33Z DBG unary code=OK end="28/Oct/2022:08:50:33 +0000" from=tcp://127.0.0.1:53456 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=70877 traceid=00000000000000000000000000000000 uri=/cs3.identity.user.v1beta1.UserAPI/GetUserByClaim user-agent=grpc-go/1.49.0
2022-10-28T08:50:33Z DBG unary code=OK end="28/Oct/2022:08:50:33 +0000" from=tcp://127.0.0.1:53454 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=478103 traceid=00000000000000000000000000000000 uri=/cs3.gateway.v1beta1.GatewayAPI/GetUserByClaim user-agent=grpc-go/1.49.0
2022-10-28T08:50:33Z INF user idp:"internal" opaque_id:"4735a501-2b40-4ddc-a238-58727bc72ad2" type:USER_TYPE_PRIMARY  authenticated pkg=rgrpc service=storage-system traceid=00000000000000000000000000000000
2022-10-28T08:50:33Z DBG unary code=OK end="28/Oct/2022:08:50:33 +0000" from=tcp://127.0.0.1:53452 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=907968 traceid=00000000000000000000000000000000 uri=/cs3.auth.provider.v1beta1.ProviderAPI/Authenticate user-agent=grpc-go/1.49.0
2022-10-28T08:50:33Z DBG unary code=OK end="28/Oct/2022:08:50:33 +0000" from=tcp://172.16.11.148:57038 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=1628545 traceid=00000000000000000000000000000000 uri=/cs3.gateway.v1beta1.GatewayAPI/Authenticate user-agent=grpc-go/1.49.0
2022-10-28T08:50:33Z DBG unary code=OK end="28/Oct/2022:08:50:33 +0000" from=tcp://127.0.0.1:52036 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=68173 traceid=00000000000000000000000000000000 uri=/cs3.permissions.v1beta1.PermissionsAPI/CheckPermission user-agent=grpc-go/1.49.0
goroutine 54 [running]:
runtime/debug.Stack()
    runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
    runtime/debug/stack.go:16 +0x19
github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x3cb11e0, 0xc0013bd6b0}, {0x38b5f40, 0x4fecfc0})
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/recovery/recovery.go:48 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x3cb11e0?, 0xc0013bd6b0?}, {0x38b5f40?, 0x4fecfc0?}, 0xc001316558?)
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:29 +0x7b
panic({0x38b5f40, 0x4fecfc0})
    runtime/panic.go:838 +0x207
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Node).InternalPath(...)
    github.com/cs3org/reva/v2@v2.10.0/pkg/storage/utils/decomposedfs/node/node.go:465
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).CreateStorageSpace(0xc0000c2050, {0x3cb11e0, 0xc0013bd9e0}, 0xc000fe6e40)
    github.com/cs3org/reva/v2@v2.10.0/pkg/storage/utils/decomposedfs/spaces.go:95 +0x4b0
github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).CreateStorageSpace(0xc000112000, {0x3cb11e0, 0xc0013bd9e0}, 0xc000fe6e40)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/services/storageprovider/storageprovider.go:469 +0x49
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler.func1({0x3cb11e0, 0xc0013bd9e0}, {0x3b971c0?, 0xc000fe6e40})
    github.com/cs3org/go-cs3apis@v0.0.0-20220818202316-e92afdddac6d/cs3/storage/provider/v1beta1/provider_api.pb.go:6441 +0x78
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x3cb11e0, 0xc0013bd8f0}, {0x3b971c0, 0xc000fe6e40}, 0xc0003a0be0, 0xc000cc2d38)
    go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.34.0/interceptor.go:325 +0x664
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd8f0?}, {0x3b971c0?, 0xc000fe6e40?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func1({0x3cb11e0, 0xc0013bd6b0}, {0x3b971c0, 0xc000fe6e40}, 0xc0003a0be0, 0xc0003a0c00)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/auth/auth.go:145 +0xa06
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?}, 0xc000f772b8?, 0x4905b7?)
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x3cb11e0, 0xc0013bd6b0}, {0x3b971c0, 0xc000fe6e40}, 0xc0003a0be0, 0xc0003a0c40)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x3cb11e0, 0xc0013bd6b0}, {0x3b971c0, 0xc000fe6e40}, 0x1?, 0xc0003a0c60)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/useragent/useragent.go:38 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x3cb11e0, 0xc0013bd620}, {0x3b971c0, 0xc000fe6e40}, 0x5191050?, 0xc0003a0c80)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/token/token.go:44 +0x158
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd620?}, {0x3b971c0?, 0xc000fe6e40?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x3cb11e0, 0xc0013bd530}, {0x3b971c0, 0xc000fe6e40}, 0x7f11718a25e0?, 0xc0003a0ca0)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd530?}, {0x3b971c0?, 0xc000fe6e40?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x3cb11e0, 0xc0013bd530}, {0x3b971c0, 0xc000fe6e40}, 0xc012d1aac8?, 0x38b4c20?)
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbf
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler({0x3c434e0?, 0xc000112000}, {0x3cb11e0, 0xc0013bd530}, 0xc0003a44d0, 0xc0010f2120)
    github.com/cs3org/go-cs3apis@v0.0.0-20220818202316-e92afdddac6d/cs3/storage/provider/v1beta1/provider_api.pb.go:6443 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000000b40, {0x3cbfd38, 0xc000cf1380}, 0xc00029b200, 0xc0010f2570, 0x500e150, 0x0)
    google.golang.org/grpc@v1.49.0/server.go:1301 +0xb0b
google.golang.org/grpc.(*Server).handleStream(0xc000000b40, {0x3cbfd38, 0xc000cf1380}, 0xc00029b200, 0x0)
    google.golang.org/grpc@v1.49.0/server.go:1642 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
    google.golang.org/grpc@v1.49.0/server.go:938 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
    google.golang.org/grpc@v1.49.0/server.go:936 +0x28a
2022-10-28T08:50:33Z ERR runtime error: invalid memory address or nil pointer dereference; stack: goroutine 54 [running]:
runtime/debug.Stack()
    runtime/debug/stack.go:24 +0x65
github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x3cb11e0, 0xc0013bd6b0}, {0x38b5f40, 0x4fecfc0})
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/recovery/recovery.go:50 +0x65
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x3cb11e0?, 0xc0013bd6b0?}, {0x38b5f40?, 0x4fecfc0?}, 0xc001316558?)
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:29 +0x7b
panic({0x38b5f40, 0x4fecfc0})
    runtime/panic.go:838 +0x207
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Node).InternalPath(...)
    github.com/cs3org/reva/v2@v2.10.0/pkg/storage/utils/decomposedfs/node/node.go:465
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).CreateStorageSpace(0xc0000c2050, {0x3cb11e0, 0xc0013bd9e0}, 0xc000fe6e40)
    github.com/cs3org/reva/v2@v2.10.0/pkg/storage/utils/decomposedfs/spaces.go:95 +0x4b0
github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).CreateStorageSpace(0xc000112000, {0x3cb11e0, 0xc0013bd9e0}, 0xc000fe6e40)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/services/storageprovider/storageprovider.go:469 +0x49
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler.func1({0x3cb11e0, 0xc0013bd9e0}, {0x3b971c0?, 0xc000fe6e40})
    github.com/cs3org/go-cs3apis@v0.0.0-20220818202316-e92afdddac6d/cs3/storage/provider/v1beta1/provider_api.pb.go:6441 +0x78
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x3cb11e0, 0xc0013bd8f0}, {0x3b971c0, 0xc000fe6e40}, 0xc0003a0be0, 0xc000cc2d38)
    go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.34.0/interceptor.go:325 +0x664
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd8f0?}, {0x3b971c0?, 0xc000fe6e40?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func1({0x3cb11e0, 0xc0013bd6b0}, {0x3b971c0, 0xc000fe6e40}, 0xc0003a0be0, 0xc0003a0c00)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/auth/auth.go:145 +0xa06
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?}, 0xc000f772b8?, 0x4905b7?)
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x3cb11e0, 0xc0013bd6b0}, {0x3b971c0, 0xc000fe6e40}, 0xc0003a0be0, 0xc0003a0c40)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x3cb11e0, 0xc0013bd6b0}, {0x3b971c0, 0xc000fe6e40}, 0x1?, 0xc0003a0c60)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/useragent/useragent.go:38 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x3cb11e0, 0xc0013bd620}, {0x3b971c0, 0xc000fe6e40}, 0x5191050?, 0xc0003a0c80)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/token/token.go:44 +0x158
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd620?}, {0x3b971c0?, 0xc000fe6e40?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x3cb11e0, 0xc0013bd530}, {0x3b971c0, 0xc000fe6e40}, 0x7f11718a25e0?, 0xc0003a0ca0)
    github.com/cs3org/reva/v2@v2.10.0/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd530?}, {0x3b971c0?, 0xc000fe6e40?})
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x3cb11e0, 0xc0013bd530}, {0x3b971c0, 0xc000fe6e40}, 0xc012d1aac8?, 0x38b4c20?)
    github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbf
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler({0x3c434e0?, 0xc000112000}, {0x3cb11e0, 0xc0013bd530}, 0xc0003a44d0, 0xc0010f2120)
    github.com/cs3org/go-cs3apis@v0.0.0-20220818202316-e92afdddac6d/cs3/storage/provider/v1beta1/provider_api.pb.go:6443 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000000b40, {0x3cbfd38, 0xc000cf1380}, 0xc00029b200, 0xc0010f2570, 0x500e150, 0x0)
    google.golang.org/grpc@v1.49.0/server.go:1301 +0xb0b
google.golang.org/grpc.(*Server).handleStream(0xc000000b40, {0x3cbfd38, 0xc000cf1380}, 0xc00029b200, 0x0)
    google.golang.org/grpc@v1.49.0/server.go:1642 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
    google.golang.org/grpc@v1.49.0/server.go:938 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
    google.golang.org/grpc@v1.49.0/server.go:936 +0x28a
 pkg=rgrpc service=storage-system traceid=00000000000000000000000000000000
2022-10-28T08:50:33Z ERR unary code=Internal end="28/Oct/2022:08:50:33 +0000" from=tcp://172.16.11.148:57040 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=1112793 traceid=00000000000000000000000000000000 uri=/cs3.storage.provider.v1beta1.ProviderAPI/CreateStorageSpace user-agent=grpc-go/1.49.0
butonic commented 2 years ago

@NexZhu I added an error check to prevent the panic in https://github.com/cs3org/reva/pull/3430. It seems we get an error from the os when trying to read the node. It might be a different ERRNO from the internal stat or getxatr call. Maybe the filesystem does not support xattrs? we will know more when trying with that PR.

NexZhu commented 2 years ago

@NexZhu I added an error check to prevent the panic in cs3org/reva#3430. It seems we get an error from the os when trying to read the node. It might be a different ERRNO from the internal stat or getxatr call. Maybe the filesystem does not support xattrs? we will know more when trying with that PR.

Yeah, I think you guessed right. I'm using Alibaba Cloud NAS and I think it doesn't support xattrs.

Is there a way to workaround this limitation in the software layer?

micbar commented 2 years ago

Is there a way to workaround this limitation in the software layer?

Unfortunately with the current ocis storage interface we need xattrs to store file metadata and access permissions.

NexZhu commented 2 years ago

I tested with volumes witch support xattrs, now storage-system and store pods are pending forever and I'm getting these errors, could you please take a look?

storage-system pod failed to start:

PodScheduled    False   2022-11-03 23:51:27 SchedulerError  running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition

The only log is:

{"code":"SERVER_ERROR_CODE","message":"Cannot invoke method getContent() on null object","requestId":"b6f922f7-5bb0-46ae-b771-ad0ed239b49a","successResponse":false}

store pod, the same:

{"code":"SERVER_ERROR_CODE","message":"Cannot invoke method getContent() on null object","requestId":"f50b4b50-def7-4587-bee3-2c08193b3468","successResponse":false}
rhafer commented 2 years ago

I tested with volumes witch support xattrs, now storage-system and store pods are pending forever and I'm getting these errors, could you please take a look?

storage-system pod failed to start:

PodScheduled  False   2022-11-03 23:51:27 SchedulerError  running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition

To me this look like an error of your underlying infrastructure. Kubernetes fails to schedule the pod because of some issue with the volumes. I don't think there is much we can do about that. You might be able to find some more detail about what failed by taking a look at the volumes and pvc (using kubectl), or by looking at the lower level kubernetes logs (if you have access to that).