search

go-kratos / kratos

Your ultimate Go microservices framework for the cloud-native era.
https://go-kratos.dev
MIT License
23.37k stars 4.01k forks source link

[Question] context deadline exceeded #1111

Closed zplzpl closed 3 years ago

zplzpl commented 3 years ago

version: github.com/go-kratos/kratos/v2 v2.0.0-rc6

按教程配置kratos，并使用etcd注册发现服务。 registry.Discovery的GetService是可以获取得到服务的Endpoints信息。

conn, err := grpc.DialInsecure(
        context.Background(),
        grpc.WithEndpoint("discovery:///servicename"),
        grpc.WithDiscovery(r),
    )
    if err != nil {
        panic(err)
    }

    cli := ccv1.NewConnectorClient(conn)
    reply, err := cli.CreateConnect(context.Background(), &ccv1.CreateConnectRequest{
        Username: "root",
        Password: "root",
    })
    if err != nil {
        log.Fatal(err)
    }

但是这样会报错： rpc error: code = DeadlineExceeded desc = context deadline exceeded

这可能是什么原因造成？

zplzpl commented 3 years ago

跑了一下exmaple etcd，也会同样问题

shenqidebaozi commented 3 years ago

是否可以提供一个最小复现仓库

tonybase commented 3 years ago

可以查看下etcd是否注册成功了？

zplzpl commented 3 years ago

确认注册成功。

因为下面这段代码是能返回Endpoints

    list, err := r.GetService(context.Background(), "servicename")
    if err != nil {
        log.Fatal(err.Error())
    }
    log.Printf("%+v\n", list)
zplzpl commented 3 years ago

@shenqidebaozi

https://github.com/go-kratos/kratos/tree/v2.0.0-rc6/examples/registry/etcd

使用这个example即可，我运行这个example也是会有同样的情形。

zplzpl commented 3 years ago

etcdmain: etcd Version: 3.3.13

shenqidebaozi commented 3 years ago

好像是ETCD版本问题，之前有人问过，你升级一下版本试试

shenqidebaozi commented 3 years ago

您是否已经尝试过解决方案呢？

shenqidebaozi commented 3 years ago

先关闭了，如果后续有问题可以再打开

zplzpl commented 3 years ago

您是否已经尝试过解决方案呢？

升级ETCD后，无问题。 etcd server version v3.5.0

exuan commented 3 years ago

2.0.1 upgrade 2.0.2 之后 报 context canceled

 ~/space/go/frontend   kratos upgrade 
go get -u github.com/go-kratos/kratos/cmd/kratos/v2
go get: added github.com/go-kratos/kratos/cmd/kratos/v2 v2.0.0-20210726034313-9808ceb7a805
go get: upgraded github.com/go-kratos/kratos/v2 v2.0.1 => v2.0.2
go get: upgraded github.com/mattn/go-isatty v0.0.12 => v0.0.13
go get: added github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d
go get: upgraded golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1 => v0.0.0-20210615171337-6886f2dfbf5b
go get -u github.com/go-kratos/kratos/cmd/protoc-gen-go-http/v2
go get: added github.com/go-kratos/kratos/cmd/protoc-gen-go-http/v2 v2.0.0-20210726034313-9808ceb7a805
go get -u github.com/go-kratos/kratos/cmd/protoc-gen-go-errors/v2
go get: added github.com/go-kratos/kratos/cmd/protoc-gen-go-errors/v2 v2.0.0-20210726034313-9808ceb7a805
go get -u google.golang.org/protobuf/cmd/protoc-gen-go
go get -u google.golang.org/grpc/cmd/protoc-gen-go-grpc
go get: added google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.1.0
go get -u github.com/envoyproxy/protoc-gen-validate
go get: upgraded github.com/envoyproxy/protoc-gen-validate v0.1.0 => v0.6.1
go get: added github.com/iancoleman/strcase v0.2.0
go get: added github.com/lyft/protoc-gen-star v0.5.3

 ~/space/go/frontend  make run                                                                      
cd cmd/frontend/ && go run .
2021/07/26 15:30:46 192.168.68.2:2379
2021/07/26 15:30:46 [http://192.168.68.2:11011 grpc://192.168.68.2:11012]   backend.service
ERROR msg=Failed to watch discovery endpoint: context canceled
INFO msg=[gRPC] server listening on: [::]:11001
INFO msg=[HTTP] server listening on: [::]:11000
ERROR msg=Failed to watch discovery endpoint: context canceled
ERROR msg=Failed to watch discovery endpoint: context canceled
{"level":"warn","ts":"2021-07-26T15:30:49.996+0800","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004a3880/#initially=[192.168.68.2:2379]","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"}
ERROR msg=Failed to watch discovery endpoint: context canceled
gozelus commented 3 years ago

2.0.1 upgrade 2.0.2 之后 报 context canceled

 ~/space/go/frontend   kratos upgrade 
go get -u github.com/go-kratos/kratos/cmd/kratos/v2
go get: added github.com/go-kratos/kratos/cmd/kratos/v2 v2.0.0-20210726034313-9808ceb7a805
go get: upgraded github.com/go-kratos/kratos/v2 v2.0.1 => v2.0.2
go get: upgraded github.com/mattn/go-isatty v0.0.12 => v0.0.13
go get: added github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d
go get: upgraded golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1 => v0.0.0-20210615171337-6886f2dfbf5b
go get -u github.com/go-kratos/kratos/cmd/protoc-gen-go-http/v2
go get: added github.com/go-kratos/kratos/cmd/protoc-gen-go-http/v2 v2.0.0-20210726034313-9808ceb7a805
go get -u github.com/go-kratos/kratos/cmd/protoc-gen-go-errors/v2
go get: added github.com/go-kratos/kratos/cmd/protoc-gen-go-errors/v2 v2.0.0-20210726034313-9808ceb7a805
go get -u google.golang.org/protobuf/cmd/protoc-gen-go
go get -u google.golang.org/grpc/cmd/protoc-gen-go-grpc
go get: added google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.1.0
go get -u github.com/envoyproxy/protoc-gen-validate
go get: upgraded github.com/envoyproxy/protoc-gen-validate v0.1.0 => v0.6.1
go get: added github.com/iancoleman/strcase v0.2.0
go get: added github.com/lyft/protoc-gen-star v0.5.3
 ~/space/go/frontend  make run                                                                      
cd cmd/frontend/ && go run .
2021/07/26 15:30:46 192.168.68.2:2379
2021/07/26 15:30:46 [http://192.168.68.2:11011 grpc://192.168.68.2:11012]   backend.service
ERROR msg=Failed to watch discovery endpoint: context canceled
INFO msg=[gRPC] server listening on: [::]:11001
INFO msg=[HTTP] server listening on: [::]:11000
ERROR msg=Failed to watch discovery endpoint: context canceled
ERROR msg=Failed to watch discovery endpoint: context canceled
{"level":"warn","ts":"2021-07-26T15:30:49.996+0800","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004a3880/#initially=[192.168.68.2:2379]","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"}
ERROR msg=Failed to watch discovery endpoint: context canceled

同样的问题

yhfgithub commented 3 years ago

2.0.1 upgrade 2.0.2 之后 报 context canceled

 ~/space/go/frontend   kratos upgrade 
go get -u github.com/go-kratos/kratos/cmd/kratos/v2
go get: added github.com/go-kratos/kratos/cmd/kratos/v2 v2.0.0-20210726034313-9808ceb7a805
go get: upgraded github.com/go-kratos/kratos/v2 v2.0.1 => v2.0.2
go get: upgraded github.com/mattn/go-isatty v0.0.12 => v0.0.13
go get: added github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d
go get: upgraded golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1 => v0.0.0-20210615171337-6886f2dfbf5b
go get -u github.com/go-kratos/kratos/cmd/protoc-gen-go-http/v2
go get: added github.com/go-kratos/kratos/cmd/protoc-gen-go-http/v2 v2.0.0-20210726034313-9808ceb7a805
go get -u github.com/go-kratos/kratos/cmd/protoc-gen-go-errors/v2
go get: added github.com/go-kratos/kratos/cmd/protoc-gen-go-errors/v2 v2.0.0-20210726034313-9808ceb7a805
go get -u google.golang.org/protobuf/cmd/protoc-gen-go
go get -u google.golang.org/grpc/cmd/protoc-gen-go-grpc
go get: added google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.1.0
go get -u github.com/envoyproxy/protoc-gen-validate
go get: upgraded github.com/envoyproxy/protoc-gen-validate v0.1.0 => v0.6.1
go get: added github.com/iancoleman/strcase v0.2.0
go get: added github.com/lyft/protoc-gen-star v0.5.3
 ~/space/go/frontend  make run                                                                      
cd cmd/frontend/ && go run .
2021/07/26 15:30:46 192.168.68.2:2379
2021/07/26 15:30:46 [http://192.168.68.2:11011 grpc://192.168.68.2:11012]   backend.service
ERROR msg=Failed to watch discovery endpoint: context canceled
INFO msg=[gRPC] server listening on: [::]:11001
INFO msg=[HTTP] server listening on: [::]:11000
ERROR msg=Failed to watch discovery endpoint: context canceled
ERROR msg=Failed to watch discovery endpoint: context canceled
{"level":"warn","ts":"2021-07-26T15:30:49.996+0800","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004a3880/#initially=[192.168.68.2:2379]","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"}
ERROR msg=Failed to watch discovery endpoint: context canceled

同样的问题

我的也是 ，不是etcd的问题，降低kratos版本后就没有了。

xmsz commented 3 years ago

我也遇到 （不是每次都会触发，大概试了几次就会触发了）

rpc error: code = DeadlineExceeded desc = context deadline exceeded

的提示，我能知道是哪行代码，但是我不知道如何处理

panic: rpc error: code = DeadlineExceeded desc = context deadline exceeded
goroutine 2576 [running]:
app/account/internal/service.(*AccountService).GetList.func2.1(0xc0005ae680, 0xc0003d6180, 0xe, 0x10, 0xc0003c0be0, 0x1909240, 0xc00013c4b0, 0xc0005ae690, 0xa)
    app/account/internal/service/account.go:127 +0x1f3
created by we-mp-admin-services/app/account/internal/service.(*AccountService).GetList.func2
    app/account/internal/service/account.go:113 +0x1ed
exit status 2
ERROR: exit status 1

我的代码

// service/account.go

countRes, err := s.uc.GetOfficialCount(ctx, officialItem.Appid)

// data/account.go
func (r *accountRepo) CountOfficialFans(ctx context.Context, appId string) (*officialV1.GetFansCountRes, error) {
    return r.data.officialService.GetFansCount(ctx, &officialV1.GetFansCountReq{
        AppId: appId,
    })
}
// data/data.go
func NewOfficialServiceClient() officialV1.OfficialClient {
    conn, err := grpc.DialInsecure(
        context.Background(),
        grpc.WithEndpoint("127.0.0.1:9001"),
    )

    if err != nil {
        panic(err)
    }

    c := officialV1.NewOfficialClient(conn)
    return c
}

然后另外一个服务也是kratos，且出错时没有收到请求和保存

然后可能是我代码的问题，能否帮我看看

shenqidebaozi commented 3 years ago

我也遇到 （不是每次都会触发，大概试了几次就会触发了）

rpc error: code = DeadlineExceeded desc = context deadline exceeded

的提示，我能知道是哪行代码，但是我不知道如何处理

panic: rpc error: code = DeadlineExceeded desc = context deadline exceeded
goroutine 2576 [running]:
app/account/internal/service.(*AccountService).GetList.func2.1(0xc0005ae680, 0xc0003d6180, 0xe, 0x10, 0xc0003c0be0, 0x1909240, 0xc00013c4b0, 0xc0005ae690, 0xa)
  app/account/internal/service/account.go:127 +0x1f3
created by we-mp-admin-services/app/account/internal/service.(*AccountService).GetList.func2
  app/account/internal/service/account.go:113 +0x1ed
exit status 2
ERROR: exit status 1

我的代码

// service/account.go

countRes, err := s.uc.GetOfficialCount(ctx, officialItem.Appid)
// data/account.go
func (r *accountRepo) CountOfficialFans(ctx context.Context, appId string) (*officialV1.GetFansCountRes, error) {
  return r.data.officialService.GetFansCount(ctx, &officialV1.GetFansCountReq{
      AppId: appId,
  })
}
// data/data.go
func NewOfficialServiceClient() officialV1.OfficialClient {
  conn, err := grpc.DialInsecure(
      context.Background(),
      grpc.WithEndpoint("127.0.0.1:9001"),
  )

  if err != nil {
      panic(err)
  }

  c := officialV1.NewOfficialClient(conn)
  return c
}

然后另外一个服务也是kratos，且出错时没有收到请求和保存

然后可能是我代码的问题，能否帮我看看

你这个也没用到 etcd，应该是 ctx 超时了吧？可以看下超时配置是否小于业务耗时

xmsz commented 3 years ago

你这个也没用到 etcd，应该是 ctx 超时了吧？可以看下超时配置是否小于业务耗时

好的 我再看看

xmsz commented 3 years ago

可以看下超时配置是否小于业务耗时

我把所有configs/config.yaml里的timeout都设置为 10s 然后重启服务，还是会遇到

shenqidebaozi commented 3 years ago

可以看下超时配置是否小于业务耗时

我把所有configs/config.yaml里的timeout都设置为 10s 然后重启服务，还是会遇到

配置下客户端超时

shenqidebaozi commented 3 years ago

另外要确定是哪个地方的超时，对应的是哪个 上下文

xmsz commented 3 years ago

另外要确定是哪个地方的超时，对应的是哪个 上下文

ok 我再自己试试

noovertime7 commented 4 months ago

@shenqidebaozi @exuan 请问解决了吗，我也遇到了相同的问题，偶发性的，有的机器正常，有的机器失败，etcd版本3.5，kratos版本v2.6.1，etcd有报错{"level":"warn","ts":"2024-07-10T13:06:09.691572+0800","logger":"etcd-client","caller":"v3@v3.5.9/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000580fc0/58.56.21.14:2379","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"}， 我定位到了kratos etcd注册中心具体代码，不知道什么原因 image

noovertime7 commented 4 months ago

image @xmsz

kratos-ci-bot commented 4 months ago

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

image @xmsz

noovertime7 commented 4 months ago

找到问题了，端口被占用了，导致启动失败，顶层的context被取消了，奇怪的是报错在注册中心这里了，排查了好久

kratos-ci-bot commented 4 months ago

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

The problem was found. The port was occupied, causing the startup to fail. The top-level context was canceled. The strange thing is that the error was reported here in the registration center. It took a long time to troubleshoot.