ydb-platform / ydb-go-sdk

Pure Go native and database/sql driver for YDB
https://ydb.tech
Apache License 2.0
136 stars 71 forks source link

bug: topic writer CheckErrorRetryDecisionDefault does not seem to have the default behavior #1092

Closed holycheater closed 6 months ago

holycheater commented 6 months ago

Bug Report

YDB GO SDK version:

3.56.2

Environment

macOS, M1

Current behavior:

topicoptions.CheckErrorRetryDecisionDefault does not behave like default (when you don't set any retry-function)

The code below produces following test result:

            Error:          Received unexpected error:
                            ydb: writer is closed: ydb: failed to read grpc message from writer stream: connError{node_id:1,address:'localhost:2136'}: operation/SCHEME_ERROR (code = 400070, address = localhost:2136, issues = [{#500014 'status is not ok: tablet is dropped'}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:120)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/grpcwrapper/rawtopic/rawtopicwriter.(*StreamWriter).Recv(streamwriter.go:48)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/topic/topicwriterinternal.(*WriterReconnector).Write(writer_reconnector.go:214)`
            Test:           TestPipeDead

Related code:

package stuff

import (
    "bytes"
    "context"
    "testing"
    "time"

    "github.com/stretchr/testify/require"
    "github.com/ydb-platform/ydb-go-sdk/v3"
    "github.com/ydb-platform/ydb-go-sdk/v3/topic/topicoptions"
    "github.com/ydb-platform/ydb-go-sdk/v3/topic/topictypes"
    "github.com/ydb-platform/ydb-go-sdk/v3/topic/topicwriter"
)

func TestPipeDead(t *testing.T) {
    ctx, cancel := context.WithTimeout(context.Background(), time.Second*60)
    defer cancel()

    conn, err := ydb.Open(context.TODO(), "grpc://localhost:2136/local",
        ydb.WithInsecure(),
    )
    require.NoError(t, err)

    err = conn.Topic().Create(ctx, "some-test-topic",
        topicoptions.CreateWithSupportedCodecs(topictypes.CodecRaw, topictypes.CodecGzip, topictypes.CodecZstd),
    )
    require.NoError(t, err)

    w, err := conn.Topic().StartWriter("some-test-topic",
        topicoptions.WithWriterCheckRetryErrorFunction(
            func(errInfo topicoptions.CheckErrorRetryArgs) topicoptions.CheckErrorRetryResult {
                // Retry for all transport errors
                if ydb.IsTransportError(errInfo.Error) {
                    return topicoptions.CheckErrorRetryDecisionRetry
                }

                // and use default behavior for all other errors
                return topicoptions.CheckErrorRetryDecisionDefault
            }),
    )
    require.NoError(t, err)

    // important bit, recreate topic
    require.NoError(t, conn.Topic().Drop(ctx, "some-test-topic"))
    err = conn.Topic().Create(ctx, "some-test-topic",
        topicoptions.CreateWithSupportedCodecs(topictypes.CodecRaw, topictypes.CodecGzip, topictypes.CodecZstd),
    )
    require.NoError(t, err)
    time.Sleep(time.Second * 5)

    err = w.Write(ctx, topicwriter.Message{Data: bytes.NewBufferString("whatever")})
    require.NoError(t, err)
}

Other information:

Running ydb locally from a docker image, some relatively fresh nightly build

holycheater commented 6 months ago

hmm, sorry, it seems to be race-y while recreating topic, retrying depends whether we get "tablet is dropped" or "pipe is dead" error first.