milvus-io / milvus-sdk-node

The Official Mivus node.js sdk(client)
https://milvus.io
Apache License 2.0
122 stars 37 forks source link

add transformers for bf16 and f16 data-type #303

Closed shanghaikid closed 5 months ago

shanghaikid commented 5 months ago

we shouldn't do the data transform by default for bf16 and f16 data type, although it doesn't exist in javascript. but we provide these helpers: f32ArrayToF16Bytes f16BytesToF32Array f32ArrayToBf16Bytes bf16BytesToF32Array

So that user can use in the insert, query, search api. like this

 const insert = await milvusClient.insert({
      collection_name: COLLECTION_NAME,
      data: data,
      transformers: {
        [DataType.BFloat16Vector]: f32ArrayToBf16Bytes,
      },
    });

const query = await milvusClient.query({
      collection_name: COLLECTION_NAME,
      filter: 'id > 0',
      output_fields: ['vector', 'id'],
      transformers: {
        [DataType.BFloat16Vector]: bf16BytesToF32Array,
      },
    });

const search = await milvusClient.search({
      vector: data[0].vector,
      collection_name: COLLECTION_NAME,
      output_fields: ['id', 'vector'],
      limit: 5,
      transformers: {
        [DataType.BFloat16Vector]: bf16BytesToF32Array,
      },
    });
sre-ci-robot commented 5 months ago

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: shanghaikid

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/milvus-io/milvus-sdk-node/blob/2.4/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
codecov[bot] commented 5 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 97.51%. Comparing base (e9fcf68) to head (44223e2).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## 2.4 #303 +/- ## ========================================== + Coverage 97.45% 97.51% +0.05% ========================================== Files 36 37 +1 Lines 2238 2249 +11 Branches 608 615 +7 ========================================== + Hits 2181 2193 +12 + Misses 52 51 -1 Partials 5 5 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.