Document rclone setup - Githubissues

michielbdejong commented 1 year ago

I don't see any documentation about setting up an rclone server in https://developer.sciencemesh.io/docs/technical-documentation/ - are sites expected to run one?

I found https://github.com/cs3org/reva/blob/396dd741bcb4c86ab6be8165451b6ccdfdfcba75/examples/datatx/datatx.toml#L13 but I haven't found any docs yet about which version of rclone should be run.

CC @redblom

michielbdejong commented 1 year ago

Einstein uses transfer-create and then Marie should see the incoming share. Testing this with nrro-testing:

>> ocm-share-list-received
error: code=CODE_INTERNAL msg="error listing received shares" support_trace="468b619f04e99ecfe9753b1f8f8c52ab"

Next thing to try: with the revad configs from https://github.com/cs3org/reva/tree/master/examples/datatx, or at least with the JSON share manager.

michielbdejong commented 1 year ago

Now working in a gitpod.io/empty env, trying out this script:

#!/bin/bash
set -e

docker run --restart=always -d --network=testnet --name=revad1.docker -e HOST=revad1 pondersource/dev-stock-revad
docker run --restart=always -d --network=testnet --name=rclone1.docker rclone/rclone rcd -vv --rc-user=rcloneuser --rc-pass=eilohtho9oTahsuongeeTh7reedahPo1Ohwi3aek --rc-addr=0.0.0.0:5572 --server-side-across-configs=true --log-file=/dev/stdout
docker run --restart=always -d --network=testnet --name=revad2.docker -e HOST=revad2 pondersource/dev-stock-revad
docker run --restart=always -d --network=testnet --name=rclone2.docker rclone/rclone rcd -vv --rc-user=rcloneuser --rc-pass=eilohtho9oTahsuongeeTh7reedahPo1Ohwi3aek --rc-addr=0.0.0.0:5572 --server-side-across-configs=true --log-file=/dev/stdout

still debugging some errors with that, will continue tomorrow.

michielbdejong commented 1 year ago

2023/02/22 09:17:41 open /root/.reva-token: no such file or directory

michielbdejong commented 1 year ago

2023-02-22 09:28:16.282 ERR ../../reva/internal/grpc/services/ocmproviderauthorizer/ocmproviderauthorizer.go:111 > error getting provider info error="error: not found: revad1.docker" pid=25 pkg=rgrpc traceid=118323692a6bc6e469593dd395b5f0a4

michielbdejong commented 1 year ago

>> ocm-invite-forward -idp revad1.docker -token 4c510ada-c86b-4815-8820-42cdf82c3d51
error: code=CODE_INTERNAL msg="error forwarding invite:json: error sending accept post request: 401 Unauthorized: {\n  \"code\": \"UNAUTHENTICATED\",\n  \"message\": \"provider not authorized\"\n}" support_trace="baee63edc4caa2d59a1838e245920903"

michielbdejong commented 1 year ago

Switched to the pondersource/sciencemesh branch of reva (a fork of cs3org/master)

2023-02-22 10:50:12.342 ERR ../../reva/internal/http/services/ocmd/reqres.go:62 > provider not authorized error="error verifying mesh provider" pid=25 pkg=rhttp traceid=f8e3d484dd32339b8aed3472df82cd27

michielbdejong commented 1 year ago

And before that:


2023-02-22 10:50:12.341 ERR ../../reva/internal/grpc/services/ocmproviderauthorizer/ocmproviderauthorizer.go:125 > error verifying mesh provider error="internal error: json: ocm host not specified for mesh provider" pid=25 pkg=rgrpc traceid=eab4efa5a91af2973caf6f82bec00307

michielbdejong commented 1 year ago

With latest master of reva, revad1.docker says:

2023-02-22 11:52:07.119 ERR ../../reva/internal/grpc/services/ocmproviderauthorizer/ocmproviderauthorizer.go:124 > error verifying mesh provider error="json: error looking up client IP: lookup https://revad2.docker: no such host" pid=1 pkg=rgrpc traceid=e669f06a379e9522e3a30dcd62b160b1

michielbdejong commented 1 year ago

This is a response from net.LookupIP(ocmHost) so let's remove https:// from [i].services[j].host in providers.testnet.json

michielbdejong commented 1 year ago

With that, new error is:

2023-02-22 12:31:48.877 WRN ../../workspace/empty/reva/internal/grpc/services/ocminvitemanager/ocminvitemanager.go:252 > token not found pid=1 pkg=rgrpc traceid=f1d3761d25229567a3111e9a3912ab86

michielbdejong commented 1 year ago

Ah wait, I was confusing token with user_id.opaque_id. Fixed that, now it works!

>> ocm-invite-forward -idp revad1.docker -token 4891e4a3-0fed-4bae-8085-4be7b7f38b84
status:<code:CODE_OK trace:"0cc2e8209e0b349300b819d35d210b06" > user_id:<idp:"revad1.docker" opaque_id:"4c510ada-c86b-4815-8820-42cdf82c3d51" type:USER_TYPE_PRIMARY > email:"einstein@revad1-email.com" display_name:"Albert Einstein"

And I see ocm-find-accepted-users lists the other user on both sides, awesome. :) Now I can continue trying to do a data transfer :)

michielbdejong commented 1 year ago

Hm, einstein can create a transfer to marie with transfer-create but it ends up as SHARE_TYPE_USER and nothing happens yet. Screenshot 2023-02-22 at 13 43 39

Now, I think the next step is for Marie to accept the share but that is not working yet:


>> ocm-share-update-received -state accepted 6945bf85-0ab3-4bcb-8b52-c4d1ce7b2177
rpc error: code = Internal desc = runtime error: invalid memory address or nil pointer dereference

michielbdejong commented 1 year ago

This is the full log of revad2.docker:


023-02-22 13:31:13.171 DBG ../../workspace/empty/reva/internal/grpc/interceptors/auth/auth.go:101 > skipping auth method=/cs3.gateway.v1beta1.GatewayAPI/ListReceivedOCMShares pid=1 pkg=rgrpc traceid=3f68b853b68a2027ca7adbace5858e06
2023-02-22 13:31:13.172 DBG ../../workspace/empty/reva/internal/grpc/interceptors/log/log.go:66 > unary code=OK end="22/Feb/2023:13:31:13 +0000" from=tcp://127.0.0.1:58194 pid=1 pkg=rgrpc start="22/Feb/2023:13:31:13 +0000" time_ns=299490 traceid=b3ddf6b54737d7b7eaed06bfa2129773 uri=/cs3.sharing.ocm.v1beta1.OcmAPI/ListReceivedOCMShares user-agent=grpc-go/1.52.0
2023-02-22 13:31:13.173 DBG ../../workspace/empty/reva/internal/grpc/interceptors/log/log.go:66 > unary code=OK end="22/Feb/2023:13:31:13 +0000" from=tcp://[::1]:38908 pid=1 pkg=rgrpc start="22/Feb/2023:13:31:13 +0000" time_ns=1159659 traceid=3f68b853b68a2027ca7adbace5858e06 uri=/cs3.gateway.v1beta1.GatewayAPI/ListReceivedOCMShares user-agent=grpc-go/1.52.0
2023-02-22 13:31:13.974 DBG ../../workspace/empty/reva/internal/grpc/interceptors/auth/auth.go:101 > skipping auth method=/cs3.gateway.v1beta1.GatewayAPI/GetReceivedOCMShare pid=1 pkg=rgrpc traceid=dcbd7874d3828c930a510b8027074cbf
2023-02-22 13:31:13.975 DBG ../../workspace/empty/reva/internal/grpc/interceptors/log/log.go:66 > unary code=OK end="22/Feb/2023:13:31:13 +0000" from=tcp://127.0.0.1:58194 pid=1 pkg=rgrpc start="22/Feb/2023:13:31:13 +0000" time_ns=340750 traceid=cfcf95b1213ab17cf2c8e822d1d439d0 uri=/cs3.sharing.ocm.v1beta1.OcmAPI/GetReceivedOCMShare user-agent=grpc-go/1.52.0
2023-02-22 13:31:13.975 DBG ../../workspace/empty/reva/internal/grpc/interceptors/log/log.go:66 > unary code=OK end="22/Feb/2023:13:31:13 +0000" from=tcp://[::1]:38924 pid=1 pkg=rgrpc start="22/Feb/2023:13:31:13 +0000" time_ns=1187299 traceid=dcbd7874d3828c930a510b8027074cbf uri=/cs3.gateway.v1beta1.GatewayAPI/GetReceivedOCMShare user-agent=grpc-go/1.52.0
2023-02-22 13:31:13.976 DBG ../../workspace/empty/reva/internal/grpc/interceptors/auth/auth.go:101 > skipping auth method=/cs3.gateway.v1beta1.GatewayAPI/UpdateReceivedOCMShare pid=1 pkg=rgrpc traceid=abfb134e5fef9302ec335ab11db7afbc
2023-02-22 13:31:13.978 DBG ../../workspace/empty/reva/internal/grpc/interceptors/log/log.go:66 > unary code=OK end="22/Feb/2023:13:31:13 +0000" from=tcp://127.0.0.1:58194 pid=1 pkg=rgrpc start="22/Feb/2023:13:31:13 +0000" time_ns=1606440 traceid=57b4767f02e99a3bffa8e6580967acad uri=/cs3.sharing.ocm.v1beta1.OcmAPI/UpdateReceivedOCMShare user-agent=grpc-go/1.52.0
2023-02-22 13:31:13.978 DBG ../../workspace/empty/reva/internal/grpc/interceptors/log/log.go:66 > unary code=OK end="22/Feb/2023:13:31:13 +0000" from=tcp://127.0.0.1:58194 pid=1 pkg=rgrpc start="22/Feb/2023:13:31:13 +0000" time_ns=239180 traceid=4ddfd1f389442a68dc57837bfdb93050 uri=/cs3.sharing.ocm.v1beta1.OcmAPI/GetReceivedOCMShare user-agent=grpc-go/1.52.0
goroutine 5073 [running]:
runtime/debug.Stack()
        /home/gitpod/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
        /home/gitpod/go/src/runtime/debug/stack.go:16 +0x19
github.com/cs3org/reva/internal/grpc/interceptors/recovery.recoveryFunc({0x2265738, 0xc0044d69c0}, {0x17e3b20, 0x2e2d360})
        /workspace/empty/reva/internal/grpc/interceptors/recovery/recovery.go:47 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x2265738?, 0xc0044d69c0?}, {0x17e3b20?, 0x2e2d360?}, 0x0?)
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:29 +0x7b
panic({0x17e3b20, 0x2e2d360})
        /home/gitpod/go/src/runtime/panic.go:884 +0x212
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End.func1()
        /workspace/go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.11.2/trace/span.go:383 +0x2a
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0xc000850000, {0x0, 0x0, 0x2edce60?})
        /workspace/go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.11.2/trace/span.go:421 +0x942
panic({0x17e3b20, 0x2e2d360})
        /home/gitpod/go/src/runtime/panic.go:884 +0x212
github.com/cs3org/reva/internal/grpc/services/gateway.(*svc).createOCMReference(0xc00092a180, {0x2265738, 0xc0044d6d50}, 0xc000872bb0)
        /workspace/empty/reva/internal/grpc/services/gateway/ocmshareprovider.go:429 +0x212
github.com/cs3org/reva/internal/grpc/services/gateway.(*svc).UpdateReceivedOCMShare(0xc00092a180, {0x2265738, 0xc0044d6d50}, 0xc0008cb700)
        /workspace/empty/reva/internal/grpc/services/gateway/ocmshareprovider.go:358 +0x790
github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1._GatewayAPI_UpdateReceivedOCMShare_Handler.func1({0x2265738, 0xc0044d6d50}, {0x1976b80?, 0xc0008cb700})
        /workspace/go/pkg/mod/github.com/cs3org/go-cs3apis@v0.0.0-20230221082129-bcf2b5cf8870/cs3/gateway/v1beta1/gateway_api.pb.go:3677 +0x7b
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x2265738, 0xc0044d6c90}, {0x1976b80, 0xc0008cb700}, 0xc0002c2ea0, 0xc00084c750)
        /workspace/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.37.0/interceptor.go:349 +0x528
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d6c90?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/auth.NewUnary.func1({0x2265738, 0xc0044d69c0}, {0x1976b80, 0xc0008cb700}, 0xc0002c2ea0, 0xc0002c2ec0)
        /workspace/empty/reva/internal/grpc/interceptors/auth/auth.go:116 +0x305
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?}, 0xc000a671f8?, 0x4d3c57?)
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/log.NewUnary.func1({0x2265738, 0xc0044d69c0}, {0x1976b80, 0xc0008cb700}, 0xc0002c2ea0, 0xc0002c2f40)
        /workspace/empty/reva/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/useragent.NewUnary.func1({0x2265738, 0xc0044d69c0}, {0x1976b80, 0xc0008cb700}, 0x1?, 0xc0002c2f80)
        /workspace/empty/reva/internal/grpc/interceptors/useragent/useragent.go:38 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/token.NewUnary.func1({0x2265738, 0xc0044d6930}, {0x1976b80, 0xc0008cb700}, 0xc004687620?, 0xc0002c2fa0)
        /workspace/empty/reva/internal/grpc/interceptors/token/token.go:44 +0x158
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d6930?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/appctx.NewUnary.func1({0x2265738, 0xc0044d6840}, {0x1976b80, 0xc0008cb700}, 0x7f040e6fd518?, 0xc0002c2fc0)
        /workspace/empty/reva/internal/grpc/interceptors/appctx/appctx.go:42 +0x5e6
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d6840?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x2265738, 0xc0044d6840}, {0x1976b80, 0xc0008cb700}, 0xc004809a20?, 0x17e2860?)
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbe
github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1._GatewayAPI_UpdateReceivedOCMShare_Handler({0x1a1ece0?, 0xc00092a180}, {0x2265738, 0xc0044d6840}, 0xc000793c00, 0xc0008c6510)
        /workspace/go/pkg/mod/github.com/cs3org/go-cs3apis@v0.0.0-20230221082129-bcf2b5cf8870/cs3/gateway/v1beta1/gateway_api.pb.go:3679 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000aa83c0, {0x226fda0, 0xc0042b2000}, 0xc0047118c0, 0xc0008c6a80, 0x2e489e0, 0x0)
        /workspace/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:1336 +0xd23
google.golang.org/grpc.(*Server).handleStream(0xc000aa83c0, {0x226fda0, 0xc0042b2000}, 0xc0047118c0, 0x0)
        /workspace/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:1704 +0xa2f
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /workspace/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /workspace/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:963 +0x28a
2023-02-22 13:31:13.979 ERR ../../workspace/empty/reva/internal/grpc/interceptors/recovery/recovery.go:49 > runtime error: invalid memory address or nil pointer dereference; stack: goroutine 5073 [running]:
runtime/debug.Stack()
        /home/gitpod/go/src/runtime/debug/stack.go:24 +0x65
github.com/cs3org/reva/internal/grpc/interceptors/recovery.recoveryFunc({0x2265738, 0xc0044d69c0}, {0x17e3b20, 0x2e2d360})
        /workspace/empty/reva/internal/grpc/interceptors/recovery/recovery.go:49 +0x65
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x2265738?, 0xc0044d69c0?}, {0x17e3b20?, 0x2e2d360?}, 0x0?)
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:29 +0x7b
panic({0x17e3b20, 0x2e2d360})
        /home/gitpod/go/src/runtime/panic.go:884 +0x212
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End.func1()
        /workspace/go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.11.2/trace/span.go:383 +0x2a
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0xc000850000, {0x0, 0x0, 0x2edce60?})
        /workspace/go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.11.2/trace/span.go:421 +0x942
panic({0x17e3b20, 0x2e2d360})
        /home/gitpod/go/src/runtime/panic.go:884 +0x212
github.com/cs3org/reva/internal/grpc/services/gateway.(*svc).createOCMReference(0xc00092a180, {0x2265738, 0xc0044d6d50}, 0xc000872bb0)
        /workspace/empty/reva/internal/grpc/services/gateway/ocmshareprovider.go:429 +0x212
github.com/cs3org/reva/internal/grpc/services/gateway.(*svc).UpdateReceivedOCMShare(0xc00092a180, {0x2265738, 0xc0044d6d50}, 0xc0008cb700)
        /workspace/empty/reva/internal/grpc/services/gateway/ocmshareprovider.go:358 +0x790
github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1._GatewayAPI_UpdateReceivedOCMShare_Handler.func1({0x2265738, 0xc0044d6d50}, {0x1976b80?, 0xc0008cb700})
        /workspace/go/pkg/mod/github.com/cs3org/go-cs3apis@v0.0.0-20230221082129-bcf2b5cf8870/cs3/gateway/v1beta1/gateway_api.pb.go:3677 +0x7b
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x2265738, 0xc0044d6c90}, {0x1976b80, 0xc0008cb700}, 0xc0002c2ea0, 0xc00084c750)
        /workspace/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.37.0/interceptor.go:349 +0x528
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d6c90?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/auth.NewUnary.func1({0x2265738, 0xc0044d69c0}, {0x1976b80, 0xc0008cb700}, 0xc0002c2ea0, 0xc0002c2ec0)
        /workspace/empty/reva/internal/grpc/interceptors/auth/auth.go:116 +0x305
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?}, 0xc000a671f8?, 0x4d3c57?)
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/log.NewUnary.func1({0x2265738, 0xc0044d69c0}, {0x1976b80, 0xc0008cb700}, 0xc0002c2ea0, 0xc0002c2f40)
        /workspace/empty/reva/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/useragent.NewUnary.func1({0x2265738, 0xc0044d69c0}, {0x1976b80, 0xc0008cb700}, 0x1?, 0xc0002c2f80)
        /workspace/empty/reva/internal/grpc/interceptors/useragent/useragent.go:38 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/token.NewUnary.func1({0x2265738, 0xc0044d6930}, {0x1976b80, 0xc0008cb700}, 0xc004687620?, 0xc0002c2fa0)
        /workspace/empty/reva/internal/grpc/interceptors/token/token.go:44 +0x158
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d6930?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/appctx.NewUnary.func1({0x2265738, 0xc0044d6840}, {0x1976b80, 0xc0008cb700}, 0x7f040e6fd518?, 0xc0002c2fc0)
        /workspace/empty/reva/internal/grpc/interceptors/appctx/appctx.go:42 +0x5e6
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d6840?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x2265738, 0xc0044d6840}, {0x1976b80, 0xc0008cb700}, 0xc004809a20?, 0x17e2860?)
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbe
github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1._GatewayAPI_UpdateReceivedOCMShare_Handler({0x1a1ece0?, 0xc00092a180}, {0x2265738, 0xc0044d6840}, 0xc000793c00, 0xc0008c6510)
        /workspace/go/pkg/mod/github.com/cs3org/go-cs3apis@v0.0.0-20230221082129-bcf2b5cf8870/cs3/gateway/v1beta1/gateway_api.pb.go:3679 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000aa83c0, {0x226fda0, 0xc0042b2000}, 0xc0047118c0, 0xc0008c6a80, 0x2e489e0, 0x0)
        /workspace/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:1336 +0xd23
google.golang.org/grpc.(*Server).handleStream(0xc000aa83c0, {0x226fda0, 0xc0042b2000}, 0xc0047118c0, 0x0)
        /workspace/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:1704 +0xa2f
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /workspace/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /workspace/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:963 +0x28a
 pid=1 pkg=rgrpc traceid=abfb134e5fef9302ec335ab11db7afbc
2023-02-22 13:31:13.979 ERR ../../workspace/empty/reva/internal/grpc/interceptors/log/log.go:66 > unary code=Internal end="22/Feb/2023:13:31:13 +0000" from=tcp://[::1]:38924 pid=1 pkg=rgrpc start="22/Feb/2023:13:31:13 +0000" time_ns=3756660 traceid=abfb134e5fef9302ec335ab11db7afbc uri=/cs3.gateway.v1beta1.GatewayAPI/UpdateReceivedOCMShare user-agent=grpc-go/1.52.0

michielbdejong commented 1 year ago

Line 429 of internal/grpc/services/gateway/ocmshareprovider.go is:

targetURI = fmt.Sprintf("webdav://%s@%s?name=%s", d.SharedSecret, share.Creator.Idp, share.Name)

michielbdejong commented 1 year ago

The problem is d.SharedSecret. I'll replace it with "asdf" and continue testing.

redblom commented 1 year ago

Just to be clear, for a transfer type share this part is not touched.

michielbdejong commented 1 year ago

it was touched by @gmgigi96 last week: https://github.com/cs3org/reva/blame/94ced01208864b66b21d2c54977c6df548d37974/internal/grpc/services/gateway/ocmshareprovider.go#L429

michielbdejong commented 1 year ago

-> split out to https://github.com/cs3org/reva/issues/3677

michielbdejong commented 1 year ago

Next issue: Hm, ocm-share-update-received returns OK but the state stays pending:


>> ocm-share-list-received
+--------------------------------------+---------------+--------------------------------------+--------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-------------------------------+-------------------------------+---------------------+-----------------+
| #                                    | OWNER.IDP     | OWNER.OPAQUEID                       | RESOURCEID                                                               | TYPE              | GRANTEE.IDP   | GRANTEE.OPAQUEID                     | CREATED                       | UPDATED                       | STATE               | SHARETYPE       |
+--------------------------------------+---------------+--------------------------------------+--------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-------------------------------+-------------------------------+---------------------+-----------------+
| 5d20f59c-e161-4e55-bd9d-847e82360d20 | revad1.docker | 4c510ada-c86b-4815-8820-42cdf82c3d51 | opaque_id:"123e4567-e89b-12d3-a456-426655440000:fileid-einstein%2Fasdf"  | GRANTEE_TYPE_USER | revad2.docker | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | 2023-02-22 13:37:27 +0000 UTC | 2023-02-22 13:37:27 +0000 UTC | SHARE_STATE_PENDING | SHARE_TYPE_USER |
+--------------------------------------+---------------+--------------------------------------+--------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-------------------------------+-------------------------------+---------------------+-----------------+
>> ocm-share-update-received -state accepted 5d20f59c-e161-4e55-bd9d-847e82360d20
OK
>> ocm-share-list-received
+--------------------------------------+---------------+--------------------------------------+--------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-------------------------------+-------------------------------+---------------------+-----------------+
| #                                    | OWNER.IDP     | OWNER.OPAQUEID                       | RESOURCEID                                                               | TYPE              | GRANTEE.IDP   | GRANTEE.OPAQUEID                     | CREATED                       | UPDATED                       | STATE               | SHARETYPE       |
+--------------------------------------+---------------+--------------------------------------+--------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-------------------------------+-------------------------------+---------------------+-----------------+
| 5d20f59c-e161-4e55-bd9d-847e82360d20 | revad1.docker | 4c510ada-c86b-4815-8820-42cdf82c3d51 | opaque_id:"123e4567-e89b-12d3-a456-426655440000:fileid-einstein%2Fasdf"  | GRANTEE_TYPE_USER | revad2.docker | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | 2023-02-22 13:37:27 +0000 UTC | 2023-02-22 13:37:27 +0000 UTC | SHARE_STATE_PENDING | SHARE_TYPE_USER |
+--------------------------------------+---------------+--------------------------------------+--------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-------------------------------+-------------------------------+---------------------+-----------------+
>>

redblom commented 1 year ago

it was touched by @gmgigi96 last week: https://github.com/cs3org/reva/blame/94ced01208864b66b21d2c54977c6df548d37974/internal/grpc/services/gateway/ocmshareprovider.go#L429

I mean, for a transfer type share there is no OCM reference being created. When accepting a share this function (createOCMReference) is not called. So there must be something else going on.

redblom commented 1 year ago

Next issue: Hm, ocm-share-update-received returns OK but the state stays pending:

That is correct. ~The status IS pending immediately after accepting. The transfer is offloaded to rclone. Next command would be transfer-list to find the share/transfer combo. And use the transfer id for the transfer-get-status command to discover the up to date status.~ Ah apparently you know that. So basically the transfer is offloaded to rclone and it may take some time for the status to change. If it stays at status pending than rclone may not be responding as expected.

michielbdejong commented 1 year ago

Ah ok. But I don't see it show up in transfer-list. What I do is, on revad1.docker (Einstein):

>> login basic
username: einstein
password: OK
>> ocm-invite-generate
status:<code:CODE_OK trace:"ff1619d4c157866ccb1a73ff3229ad60" > invite_token:<token:"7626b9d8-489a-44d8-bdf0-859031fb7f31" user_id:<idp:"revad1.docker" opaque_id:"4c510ada-c86b-4815-8820-42cdf82c3d51" type:USER_TYPE_PRIMARY > expiration:<seconds:1677161020 > >

Then on revad2.docker (Marie):

>> login basic
username: marie
password: OK
>> ocm-invite-forward -idp revad1.docker -token 7626b9d8-489a-44d8-bdf0-859031fb7f31
status:<code:CODE_OK trace:"b610c39d6da2b8c9cdb7cdd81083e0b5" > user_id:<idp:"revad1.docker" opaque_id:"4c510ada-c86b-4815-8820-42cdf82c3d51" type:USER_TYPE_PRIMARY > email:"einstein@revad1-email.com" display_name:"Albert Einstein"

Then on revad1.docker (Einstein):

>> ocm-find-accepted-users
+--------------------------------------+---------------+------------------------+-------------+
| OPAQUEID                             | IDP           | MAIL                   | DISPLAYNAME |
+--------------------------------------+---------------+------------------------+-------------+
| f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | revad2.docker | marie@revad2-email.com | Marie Curie |
+--------------------------------------+---------------+------------------------+-------------+
>> mkdir /home/asdf
>> transfer-create -idp revad2.docker -grantee f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c /home/asdf
+--------------------------------------+---------------+--------------------------------------+---------------------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-----------------+-------------------------------+-------------------------------+
| #                                    | OWNER.IDP     | OWNER.OPAQUEID                       | RESOURCEID                                                                            | TYPE              | GRANTEE.IDP   | GRANTEE.OPAQUEID                     | SHARETYPE       | CREATED                       | UPDATED                       |
+--------------------------------------+---------------+--------------------------------------+---------------------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-----------------+-------------------------------+-------------------------------+
| 8f981117-7937-4b95-949b-1288caee547a | revad1.docker | 4c510ada-c86b-4815-8820-42cdf82c3d51 | storage_id:"123e4567-e89b-12d3-a456-426655440000" opaque_id:"fileid-einstein%2Fasdf"  | GRANTEE_TYPE_USER | revad2.docker | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | SHARE_TYPE_USER | 2023-02-22 14:06:28 +0000 UTC | 2023-02-22 14:06:28 +0000 UTC |
+--------------------------------------+---------------+--------------------------------------+---------------------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-----------------+-------------------------------+-------------------------------+

Then on revad2.docker (Marie):


>> ocm-share-list-received
+--------------------------------------+---------------+--------------------------------------+--------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-------------------------------+-------------------------------+---------------------+-----------------+
| #                                    | OWNER.IDP     | OWNER.OPAQUEID                       | RESOURCEID                                                               | TYPE              | GRANTEE.IDP   | GRANTEE.OPAQUEID                     | CREATED                       | UPDATED                       | STATE               | SHARETYPE       |
+--------------------------------------+---------------+--------------------------------------+--------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-------------------------------+-------------------------------+---------------------+-----------------+
| 7e40795a-8987-4cf2-ac1b-979437a1fea4 | revad1.docker | 4c510ada-c86b-4815-8820-42cdf82c3d51 | opaque_id:"123e4567-e89b-12d3-a456-426655440000:fileid-einstein%2Fasdf"  | GRANTEE_TYPE_USER | revad2.docker | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | 2023-02-22 14:06:28 +0000 UTC | 2023-02-22 14:06:28 +0000 UTC | SHARE_STATE_PENDING | SHARE_TYPE_USER |
+--------------------------------------+---------------+--------------------------------------+--------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-------------------------------+-------------------------------+---------------------+-----------------+
>> ocm-share-update-received -state accepted 7e40795a-8987-4cf2-ac1b-979437a1fea4
OK
>> transfer-list
+------------------+-------------+
| SHAREID.OPAQUEID | ID.OPAQUEID |
+------------------+-------------+
+------------------+-------------+
>>

redblom commented 1 year ago

Ah, it may be that the new OCM spec has not been implemented fully yet for transfer type shares. It looks like the share is not discovered as a transfer type share: https://github.com/cs3org/reva/blob/2b0cc8a125f2ffe338e9847c6a20969d6b0d92ce/internal/grpc/services/gateway/ocmshareprovider.go#L208 Can you detect rclone being called?

michielbdejong commented 1 year ago

No, nothing is happening in the logs of the rclone server.

michielbdejong commented 1 year ago

It's also worth noting that I don't even get this far if I don't set a fake value for d.SharedSecret in line 429 of internal/grpc/services/gateway/ocmshareprovider.go, due to https://github.com/cs3org/reva/issues/3677

redblom commented 1 year ago

It's also worth noting that I don't even get this far if I don't set a fake value for d.SharedSecret in line 429 of internal/grpc/services/gateway/ocmshareprovider.go, due to cs3org/reva#3677

Yeah, but that method shouldn't even be called if we're dealing with a transfer.

redblom commented 1 year ago

Ok, I will test this myself locally, see what needs to be done.

redblom commented 1 year ago

I see that the share protocol of the transfer request is not set: https://github.com/cs3org/reva/blob/2b0cc8a125f2ffe338e9847c6a20969d6b0d92ce/internal/grpc/services/ocmshareprovider/ocmshareprovider.go#L192-205

func (s *service) getProtocols(ctx context.Context, info *providerpb.ResourceInfo, methods []*ocm.AccessMethod) ocmd.Protocols {
    var p ocmd.Protocols
    for _, m := range methods {
        switch t := m.Term.(type) {
        case *ocm.AccessMethod_WebdavOptions:
            p = append(p, s.getWebdavProtocol(ctx, info, t))
        case *ocm.AccessMethod_WebappOptions:
            // TODO
        case *ocm.AccessMethod_TransferOptions:
            // TODO
        }
    }
    return p
}

So that must be the issue.

redblom commented 1 year ago

So I can fix the transfers. But even regular ocm shares are not working. In createOCMReference the protocol is retrieved here:

Line 414    d, _ := getTransferProtocol(share)

This needs an implementation like getWebDAVProtocol to retrieve the secret from. That's where your nil pointer came from at:

Line 429    targetURI = fmt.Sprintf("webdav://%s@%s?name=%s", d.SharedSecret, share.Creator.Idp, share.Name)

@michielbdejong regular sharing also fails right?

michielbdejong commented 1 year ago

CC @MahdiBaghbani @parhamin2010

redblom commented 1 year ago

Can be closed. Dealt with in PR https://github.com/cs3org/reva/pull/3847. Transfers tutorial will follow shortly.

pondersource / sciencemesh-php

Document rclone setup #159