[Bug]: [GOSDK] No error is returned when inserting rows with extra fields than the schema

ThreadDao commented 4 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Environment

- Milvus version: master-20240528-b138ae74-amd64
- Deployment mode(standalone or cluster): 
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): go-sdk v2 (2.4.0-dev)
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

create collection with 2 fields: int64 + floatVec

insert row succ with int64, int32, floatVec fields

[2024/05/30 19:44:44.968 +08:00] [DEBUG] [testcases/insert_test.go:267] ["Row data"] ["row[0]"="{\"int32\":1,\"int64\":1,\"floatVec\":[0.7953272,0.97875476,0.71185696,0.80443656,0.6120793,0.9971814,0.6910342,0.60436314]}"]

If inserting column data with extra fields will return error: field int32 does not exist in collection xxx

case:

// test field name: pk, row json name: int64
func TestInsertRowMismatchFields(t *testing.T) {
ctx := hp.CreateContext(t, time.Second*common.DefaultTimeout)
mc := createDefaultMilvusClient(ctx, t)

cp := hp.NewCreateCollectionParams(hp.Int64Vec)
_, schema := hp.CollPrepare.CreateCollection(ctx, t, mc, cp, hp.TNewFieldsOption().TWithDim(8), hp.TNewSchemaOption())

// rows fields > schema fields
t.Log("https://github.com/milvus-io/milvus/issues/33487")
rowsMore := make([]interface{}, 0, 10)
for i := 1; i< 11; i++ {
    row := hp.BaseRow{
        Int64: int64(i),
        Int32: int32(i),
        FloatVec: common.GenFloatVector(8),
    }
    rowsMore = append(rowsMore, row)
}
log.Debug("Row data", zap.Any("row[0]", rowsMore[0]))
_, errInsert = mc.Insert(ctx, clientv2.NewRowBasedInsertOption(schema.CollectionName, rowsMore...))
common.CheckErr(t, errInsert, false, "")
}

Expected Behavior

same behavior between row and column !

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

yanliang567 commented 4 months ago

/unassign

please note that go sdk v2 commits should go to 2.4 branch as well

congqixia commented 4 months ago

@ThreadDao row based API will only fetch field data exists in schema. It looks like this behavior does not work for dynamic schema. Will fix that

xiaofan-luan commented 4 months ago

We need to discuss a meta cache policy. This will be very important for all SDKs, especially handle schema free collections.

All the sdk cache the schema by collection name -> collection meta
all the dml request carry a collection timestamp
if timestamp smaller than proxy cache, server should throw an error and client refresh the cache

xiaofan-luan commented 4 months ago

@czs007 will be working on this. Before we have this, we can remove all the client side check and serialize all data into the request. the server side check the field and map it to the right position

milvus-io / milvus