datastore: Parallel Transaction on same kind with unique keys giving concurrent transaction error

mehulparmariitr commented 7 months ago

Client

datastore

Environment

Go executable on Linux

Go Environment

$ go version - go version go1.21.0 darwin/amd64 $ go env -

GO111MODULE='on' GOARCH='amd64' GOBIN='' GOCACHE='/Users/mehparmar/Library/Caches/go-build' GOENV='/Users/mehparmar/Library/Application Support/go/env' GOEXE='' GOEXPERIMENT='' GOFLAGS='' GOHOSTARCH='amd64' GOHOSTOS='darwin' GOINSECURE='' GOMODCACHE='/Users/mehparmar/go/pkg/mod' GONOPROXY='' GONOSUMDB='' GOOS='darwin' GOPATH='/Users/mehparmar/go' GOPRIVATE='' GOPROXY='https://proxy.golang.org,direct' GOROOT='/usr/local/go' GOSUMDB='sum.golang.org' GOTMPDIR='' GOTOOLCHAIN='auto' GOTOOLDIR='/usr/local/go/pkg/tool/darwin_amd64' GOVCS='' GOVERSION='go1.21.0' GCCGO='gccgo' GOAMD64='v1' AR='ar' CC='clang' CXX='clang++' CGO_ENABLED='1' GOMOD='/Users/mehparmar/Repos/mehparmar/abcserv/go.mod' GOWORK='' CGO_CFLAGS='-O2 -g' CGO_CPPFLAGS='' CGO_CXXFLAGS='-O2 -g' CGO_FFLAGS='-O2 -g' CGO_LDFLAGS='-O2 -g' PKG_CONFIG='pkg-config' GOGCCFLAGS='-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/6n/kcp2mfmx0s9_lqw9y30yqq500000gq/T/go-build1900003096=/tmp/go-build -gno-record-gcc-switches -fno-common'

Code

e.g.


func processBatch(ctx context.Context, logger *l.Logger, desQSp *gds.QuerySpecs, tableName string, batch []dataanalytics.AggregationEntity, wg *sync.WaitGroup) {
    defer wg.Done() // Signal that this goroutine is done

        //Calling GetMulti on keys passed here, then checking if its insert or update call.
    err, _, _, _ := dataanalytics.Upsert(ctx, logger, (*gds2.QuerySpecs)(desQSp), tableName, batch)
    if err != nil {
        // Handle the error appropriately (logging, retries, etc.)
        log.Printf("Error during Upsert: %v", err)
    }
}

// code in main function
// go routines to write data in parallel for unique keys
var wg sync.WaitGroup // To wait for goroutines to finish
        //At max 1000 keys can be queried from Google Datastore, Hence creating batches of 1000
        for i := 0; i < len(msgs); i += 1000 {
            end := i + 1000
            if end > len(msgs) {
                end = len(msgs)
            }
            batch := msgs[i:end]
            // Process the batch here
            //wg.Add(1)
            go processBatch(ctx, l.New("abc-services"), (*gds.QuerySpecs)(desQSp), preAggregationRequest.DestinationKind, batch, &wg)
        }

        wg.Wait() // Wait for all goroutines to complete

Expected behavior

All entities should be written. There are around 275,000

Actual behavior

2024/04/30 05:06:42 Error during Upsert: datastore: concurrent transaction UUID-NOT-COLLECTED API-NOT-COLLECTED class.Upsert tx.Commit: datastore: concurrent transaction 2024/04/30 05:06:43 Error during Upsert: datastore: concurrent transaction UUID-NOT-COLLECTED API-NOT-COLLECTED class.Upsert tx.Commit: rpc error: code = InvalidArgument desc = Invalid transaction. 2024/04/30 05:06:43 Error during Upsert: rpc error: code = InvalidArgument desc = Invalid transaction. UUID-NOT-COLLECTED API-NOT-COLLECTED class.Upsert tx.Commit: rpc error: code = InvalidArgument desc = Invalid transaction. 2024/04/30 05:06:43 Error during Upsert: rpc error: code = InvalidArgument desc = Invalid transaction.

Additional context

if keys are unique then why transaction is failing?

bhshkh commented 7 months ago

On which version of Datastore client library are you facing this issue?

mehulparmariitr commented 7 months ago

@bhshkh I see its 1.0.0.

But we have replaced also this lib in go.mod

replace cloud.google.com/go/datastore => github.com/googleapis/google-cloud-go/datastore v1.0.0

mehulparmariitr commented 7 months ago

Even with the latest version the error is there. Seems like a logical error. But if keys are unique then it shouldnt throw concurrent transaction error right?

go 1.12

require (
    cloud.google.com/go/bigquery v1.60.0
    cloud.google.com/go/datastore v1.16.0
    cloud.google.com/go/logging v1.9.0
    cloud.google.com/go/profiler v0.1.0
    cloud.google.com/go/pubsub v1.37.0
    cloud.google.com/go/storage v1.39.1
    github.com/dimfeld/httptreemux v5.0.1+incompatible
    github.com/dimfeld/httptreemux/v5 v5.0.2
    github.com/evanphx/json-patch v4.5.0+incompatible
    github.com/getlantern/deepcopy v0.0.0-20160317154340-7f45deb8130a
    github.com/go-redis/redis v6.15.7+incompatible
    github.com/gomodule/redigo v2.0.0+incompatible
    github.com/kelseyhightower/envconfig v1.4.0
    github.com/mitchellh/mapstructure v1.1.2
    github.com/oleiade/reflections v1.0.1
    github.com/openzipkin/zipkin-go v0.2.5
    github.com/pborman/uuid v1.2.1
    github.com/pkg/errors v0.9.1
    github.com/schwarmco/go-cartesian-product v0.0.0-20180515110546-d5ee747a6dc9
    github.com/signalfx/golib/v3 v3.3.19
    github.com/spf13/cobra v1.5.0
    github.com/stretchr/testify v1.9.0
    github.com/thedevsaddam/gojsonq v2.3.0+incompatible
    go.opencensus.io v0.24.0
    google.golang.org/api v0.176.1
    google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda
    gopkg.in/go-playground/assert.v1 v1.2.1 // indirect
    gopkg.in/go-playground/validator.v8 v8.18.2
    gopkg.in/yaml.v2 v2.4.0
)

replace cloud.google.com/go/datastore => github.com/googleapis/google-cloud-go/datastore v1.16.0

The error -

 UUID-NOT-COLLECTED API-NOT-COLLECTED abc.Upsert tx.Commit: datastore: concurrent transaction
2024/05/01 02:46:59 Error during Upsert: datastore: concurrent transaction
 UUID-NOT-COLLECTED API-NOT-COLLECTED abc.Upsert tx.Commit: datastore: concurrent transaction
2024/05/01 02:46:59 Error during Upsert: datastore: concurrent transaction
 UUID-NOT-COLLECTED API-NOT-COLLECTED abc.Upsert tx.Commit: rpc error: code = InvalidArgument desc = The referenced transaction has expired or is no longer valid.
2024/05/01 02:47:00 Error during Upsert: rpc error: code = InvalidArgument desc = The referenced transaction has expired or is no longer valid.

mehulparmariitr commented 7 months ago

This is the method I am using to upsert into DB -

// Upsert - insert in gds aggregation and update using transaction if entries previously exist with same MdInstanceHashKey it gets updated
func Upsert(ctx context.Context, logger *slog.Logger, qSp *gds.QuerySpecs, tableName string, msgs []AggregationEntity) (error, []*datastore.Key, []*datastore.Key, []*AggregationEntity) {
    ctx, span := trace.StartSpan(ctx, "internal.dataanalytics.Update")
    defer span.End()
    logger.Infof(ctx, "Started putting entries in GDS(aggregation) in %v table", tableName)

    client, err := gds.GetDatastoreClient(ctx, logger, qSp)
    if err != nil {
        logger.Errorf(ctx, "Failed to create GDS client: %v", err)
        return err, nil, nil, nil
    }

    // make keys array use MdInstanceHashKey column
    var keys []*datastore.Key
    for id := 0; id < len(msgs); id++ {
        newKey := datastore.NameKey(tableName, msgs[id].MdInstanceHashKey, nil)
        newKey.Namespace = qSp.Namespace
        keys = append(keys, newKey)
    }
    logger.Infof(ctx, "Length of keys to insert in data store are : %v", len(keys))
    tx, err := client.NewTransaction(ctx)
    if err != nil {
        logger.Errorf(ctx, "client.NewTransaction: %v", err)
        return err, nil, nil, nil
    }
    gdsEntry := make([]AggregationEntity, len(msgs))
    if err := tx.GetMulti(keys, gdsEntry); err == datastore.ErrNoSuchEntity {
        logger.Errorf(ctx, "error in update in GDS %v", err)
        return err, nil, nil, nil
    }

    var createKeys []*datastore.Key
    var updateKeys []*datastore.Key
    var createValues []*AggregationEntity
    // checking if entry with same key exists or not if it exists then update it
    for i := 0; i < len(msgs); i++ {

        if gdsEntry[i].GitTestRepoURL == "" {
            //create
            gdsEntry[i] = msgs[i]
            createKeys = append(createKeys, keys[i])
            createValues = append(createValues, &msgs[i])
        } else {
            //update
            if gdsEntry[i].Type == Pass {
                gdsEntry[i] = AggregateUpdateSum(logger, gdsEntry[i], msgs[i], Passed)
                updateKeys = append(updateKeys, keys[i])
            }
            if gdsEntry[i].Type == Fail {
                gdsEntry[i] = AggregateUpdateSum(logger, gdsEntry[i], msgs[i], Failed)
                updateKeys = append(updateKeys, keys[i])
            }
            if gdsEntry[i].Type == Skip {
                gdsEntry[i] = AggregateUpdateSum(logger, gdsEntry[i], msgs[i], Skipped)
                updateKeys = append(updateKeys, keys[i])
            }
            if gdsEntry[i].Type == MostCommonFailure {
                gdsEntry[i] = AggregateUpdateSum(logger, gdsEntry[i], msgs[i], Failed)
                updateKeys = append(updateKeys, keys[i])
            }
            if gdsEntry[i].Type == MostFailed {
                gdsEntry[i] = AggregateUpdateSum(logger, gdsEntry[i], msgs[i], Failed)
                updateKeys = append(updateKeys, keys[i])
            }
            if gdsEntry[i].Type == MostSkipped {
                gdsEntry[i] = AggregateUpdateSum(logger, gdsEntry[i], msgs[i], Skipped)
            }
            if gdsEntry[i].Type == MostDuration {
                gdsEntry[i] = AggregateUpdateMax(logger, gdsEntry[i], msgs[i], Duration)
                updateKeys = append(updateKeys, keys[i])
            }
        }
    }

    // batch operation to put in GDS using transaction
    if _, e := tx.PutMulti(keys, gdsEntry); e != nil {
        tx.Rollback()
        logger.Errorf(ctx, "tx.PutMulti: %v", e)
        //logger.Errorf(ctx, "gdsEntry1 are: %v", gdsEntry)
        //logger.Errorf(ctx, "keys1 are: %v", gdsEntry)
        return e, nil, nil, nil
    }
    if _, er := tx.Commit(); er != nil {
        tx.Rollback()
        logger.Errorf(ctx, "tx.Commit: %v", er) // error: concurrent transaction
        //logger.Errorf(ctx, "gdsEntry2 are: %v", gdsEntry)
        //logger.Errorf(ctx, "keys2 are: %v", gdsEntry)
        return er, nil, nil, nil
    }

    logger.Infof(ctx, "Entries stored successfully")
    return nil, createKeys, updateKeys, createValues
}

bhshkh commented 4 months ago

Looking into this

bhshkh commented 4 months ago

I'm unable to reproduce this issue. This is what I have tried:

type User struct {
    name   string
    height int
    age    int
}

func main() {
    ctx := context.Background()

    projectID := "my-project"
    databaseID := "database-01"
    client, err := datastore.NewClientWithDatabase(ctx, projectID, databaseID, opts...)
    if err != nil {
        fmt.Printf("Failed to create client: %v\n", err)
    }
    defer client.Close()

    issue10068(client, ctx)
}
func issue10068(client *datastore.Client, ctx context.Context) {
    batchSize := 50
    total := 400

    // Create users and keys
    users := []*User{}
    keys := []*datastore.Key{}
    for i := 0; i < total; i++ {
        name := fmt.Sprintf("user-%03d", i)
        keys = append(keys, datastore.NameKey("users", name, nil))
        users = append(users, &User{name: name, age: i, height: i})
    }

    var wg sync.WaitGroup
    numBatches := total / batchSize
    for currBatch := 0; currBatch < numBatches; currBatch++ {
        wg.Add(1)
        currBatchKeys := keys[batchSize*currBatch : batchSize*(currBatch+1)]
        currBatchUsers := users[batchSize*currBatch : batchSize*(currBatch+1)]
        go putMultiInTxn(currBatch, client, ctx, currBatchKeys, currBatchUsers, &wg)
    }
    wg.Wait()

}

func putMultiInTxn(currBatch int, client *datastore.Client, ctx context.Context, keys []*datastore.Key, users []*User, wg *sync.WaitGroup) {
    keysStr := ""
    for _, key := range keys {
        keysStr += key.Name + ", "
    }
    fmt.Printf("\n#%d: Keys: %s\n", currBatch, keysStr)
    defer wg.Done()
    txn, err := client.NewTransaction(ctx)
    if err != nil {
        fmt.Printf("client.NewTransaction: %v\n", err)
        return
    }
    _, err = txn.PutMulti(keys, users)
    if err != nil {
        fmt.Printf("txn.PutMulti: %v\n", err)
        return
    }
    _, err = txn.Commit()
    if err != nil {
        fmt.Printf("txn.Commit: %v\n", err)
        return
    }
}

outputs:

$ go run .

#7: Keys: user-350, user-351, user-352, user-353, user-354, user-355, user-356, user-357, user-358, user-359, user-360, user-361, user-362, user-363, user-364, user-365, user-366, user-367, user-368, user-369, user-370, user-371, user-372, user-373, user-374, user-375, user-376, user-377, user-378, user-379, user-380, user-381, user-382, user-383, user-384, user-385, user-386, user-387, user-388, user-389, user-390, user-391, user-392, user-393, user-394, user-395, user-396, user-397, user-398, user-399, 

#4: Keys: user-200, user-201, user-202, user-203, user-204, user-205, user-206, user-207, user-208, user-209, user-210, user-211, user-212, user-213, user-214, user-215, user-216, user-217, user-218, user-219, user-220, user-221, user-222, user-223, user-224, user-225, user-226, user-227, user-228, user-229, user-230, user-231, user-232, user-233, user-234, user-235, user-236, user-237, user-238, user-239, user-240, user-241, user-242, user-243, user-244, user-245, user-246, user-247, user-248, user-249, 

#3: Keys: user-150, user-151, user-152, user-153, user-154, user-155, user-156, user-157, user-158, user-159, user-160, user-161, user-162, user-163, user-164, user-165, user-166, user-167, user-168, user-169, user-170, user-171, user-172, user-173, user-174, user-175, user-176, user-177, user-178, user-179, user-180, user-181, user-182, user-183, user-184, user-185, user-186, user-187, user-188, user-189, user-190, user-191, user-192, user-193, user-194, user-195, user-196, user-197, user-198, user-199, 

#0: Keys: user-000, user-001, user-002, user-003, user-004, user-005, user-006, user-007, user-008, user-009, user-010, user-011, user-012, user-013, user-014, user-015, user-016, user-017, user-018, user-019, user-020, user-021, user-022, user-023, user-024, user-025, user-026, user-027, user-028, user-029, user-030, user-031, user-032, user-033, user-034, user-035, user-036, user-037, user-038, user-039, user-040, user-041, user-042, user-043, user-044, user-045, user-046, user-047, user-048, user-049, 

#1: Keys: user-050, user-051, user-052, user-053, user-054, user-055, user-056, user-057, user-058, user-059, user-060, user-061, user-062, user-063, user-064, user-065, user-066, user-067, user-068, user-069, user-070, user-071, user-072, user-073, user-074, user-075, user-076, user-077, user-078, user-079, user-080, user-081, user-082, user-083, user-084, user-085, user-086, user-087, user-088, user-089, user-090, user-091, user-092, user-093, user-094, user-095, user-096, user-097, user-098, user-099, 

#5: Keys: user-250, user-251, user-252, user-253, user-254, user-255, user-256, user-257, user-258, user-259, user-260, user-261, user-262, user-263, user-264, user-265, user-266, user-267, user-268, user-269, user-270, user-271, user-272, user-273, user-274, user-275, user-276, user-277, user-278, user-279, user-280, user-281, user-282, user-283, user-284, user-285, user-286, user-287, user-288, user-289, user-290, user-291, user-292, user-293, user-294, user-295, user-296, user-297, user-298, user-299, 

#6: Keys: user-300, user-301, user-302, user-303, user-304, user-305, user-306, user-307, user-308, user-309, user-310, user-311, user-312, user-313, user-314, user-315, user-316, user-317, user-318, user-319, user-320, user-321, user-322, user-323, user-324, user-325, user-326, user-327, user-328, user-329, user-330, user-331, user-332, user-333, user-334, user-335, user-336, user-337, user-338, user-339, user-340, user-341, user-342, user-343, user-344, user-345, user-346, user-347, user-348, user-349, 

#2: Keys: user-100, user-101, user-102, user-103, user-104, user-105, user-106, user-107, user-108, user-109, user-110, user-111, user-112, user-113, user-114, user-115, user-116, user-117, user-118, user-119, user-120, user-121, user-122, user-123, user-124, user-125, user-126, user-127, user-128, user-129, user-130, user-131, user-132, user-133, user-134, user-135, user-136, user-137, user-138, user-139, user-140, user-141, user-142, user-143, user-144, user-145, user-146, user-147, user-148, user-149, 
$

I have also tried by setting batchSize := 1000 and total := 1000000 but still do not see the error. Are you sure that the keys are unique? Can you please share a short snippet that I can run on my end to reproduce the issue?

bhshkh commented 3 months ago

Closing this issue as there has been no response from the reporter. Please re-open if issue is seen again

googleapis / google-cloud-go

datastore: Parallel Transaction on same kind with unique keys giving concurrent transaction error #10068