Closed jiachengdb closed 1 month ago
This is true, and intended behavior. For this kind of low-level API it's hard to performantly do all the safety checks we do for high level APIs. Hence the warning we have in the docs:
This is an advanced API and doesn’t provide the same level of validation as the other APIs. For example, it’s the responsibility of the caller to ensure that the fragments are valid for the schema.
(From https://lancedb.github.io/lance/api/python/lance.html#lance.dataset.LanceDataset.commit)
Yeah, i think it would be nice to add a sanity check for the schema and reject the writes.
@jiachengdb for now we won't plan to address this issue at this API level. If this is a big pain-point, let's discuss together whether we can add something to the higher level lancedb api?
Here is a repro on pylance 0.13.0.
This will output
If I change the type of the "value" column to int64, the result is correct.
The bug is that
LanceFragment.create
neither checks the input data conforms to the target schema (it should fail if the schema doesn't match), nor does it coerce the input data. The current behavior is bad because it silently corrupts the data.