Truncate strings exceeding max_length when inserting to Milvus - Githubissues

nv-morpheus / Morpheus

Morpheus SDK

Apache License 2.0

309 stars 119 forks source link

Truncate strings exceeding max_length when inserting to Milvus #1665

Closed dagardner-nv closed 2 months ago

dagardner-nv commented 2 months ago

Description

Adds new helper methods to morpheus.io.utils, cudf_string_cols_exceed_max_bytes and truncate_string_cols_by_bytes
When truncate_long_strings=True MilvusVectorDBResourceService will truncate all VARCHAR fields according to the schema's max_length
Add truncate_long_strings=True in config for vdb_upload pipeline
Set C++ mode to default for example LLM pipelines
Remove issues 1650 & 1651 from known_issues.md

Closes #1650 Closes #1651

By Submitting this PR I confirm:

I am familiar with the Contributing Guidelines.
When the PR is ready for review, new or existing tests cover these changes.
When the PR is ready for review, the documentation is up to date with these changes.

mdemoret-nv commented 2 months ago

/merge