Open rok opened 1 year ago
@wjones127 not sure if this should be closed.
Are there any docs on how to write these types to a lance dataset? Specifically I'm trying to create a video column that's some sort of image array type.
I'm experimenting with doing this instead of just storing the video as bytes to save on decoding time in my training loop.
Hi @tonyf. In general, you can write an Apache Arrow extension array, and these can be written and read from Lance. A good reference for this would be Rok's changes for the image extension types:
After adding
ImageURIArray
,EncodedImageArray
andFixedShapeImageTensorArray
it is straightforward to add analogous types for video. NamelyVideoURIArray
,VideoEncodedArray
andFixedShapeVideoTensorArray
array. For decoder see TFs decode_webp.