vespa-engine / vespa

AI + Data, online. https://vespa.ai
https://vespa.ai
Apache License 2.0
5.49k stars 586 forks source link

Short form for indexed tensors representing binary data requires "values" #30738

Open jobergum opened 3 months ago

jobergum commented 3 months ago

From doc

Short form for indexed tensors representing binary data (with int8 cell value type): May use a string with a hex dump of the cell values:

"tensorfield": "FF00118022FE" can be used to represent the value tensor(x[6]):[-1,0,17,-128,34,-2].

But this fails

{ "put": "id:doc:doc::1", "fields": {"tensorfield":"FF00118022FE"}}
feed: got status 400 ({"pathId":"/document/v1/doc/doc/docid/1","message":"Error in document 'id:doc:doc::1' - could not parse field 'tensorfield' of type 'tensor<int8>(x[6])': Expected class 'com.yahoo.document.datatypes.TensorFieldValue', got 'java.lang.String'.: Expected class 'com.yahoo.document.datatypes.TensorFieldValue', got 'java.lang.String'."}) for put id:doc:doc::1: not retryable

However, the legacy format works.

{ "put": "id:doc:doc::1", "fields": {"tensorfield": { "values": "FF00118022FE"}}}
vespa document get id:doc:doc::1
{
    "pathId": "/document/v1/doc/doc/docid/1",
    "id": "id:doc:doc::1",
    "fields": {
        "tensorfield": {
            "type": "tensor<int8>(x[6])",
            "values": [
                -1,
                0,
                17,
                -128,
                34,
                -2
            ]
        }
    }
}
geirst commented 3 months ago

Please also add a system test.

geirst commented 1 month ago

The documentation should also be updated to include details on hex value support for vectors of type bfloat16, float, double. The bit representation of a single value should be part of these details (e.g. java.lang.Float.intBitsToFloat() and java.lang.Double.longBitsToDouble() is used).