TensorFlow's RaggedTensor has been around for a while and PyTorch's NestedTensor is new, but they should be conversion targets for Awkward Arrays. And then a tutorial should be written in the spirit of this one:
but using Awkward strings ↔ RaggedTensors instead of making np.array(["...", "...", ...], dtype=object). It's a great example of why that interface would be useful, particularly when paired with ak.str.* functions. (We can demonstrate the difference with a scaling test on some enormous dataset. What's an enormous text dataset? I wonder...)
I believe that this is an equivalent for PyTorch's NestedTensor:
I don't know if JAX has a ragged array interface, but if it does, let's get that, too. All of these converters are relatively easy, but we'd have to get TensorFlow, PyTorch, and JAX into at least one of the CI tests. TensorFlow and JAX can be hard to install.
Description of new feature
TensorFlow's RaggedTensor has been around for a while and PyTorch's NestedTensor is new, but they should be conversion targets for Awkward Arrays. And then a tutorial should be written in the spirit of this one:
https://www.tensorflow.org/text/tutorials/text_generation
but using Awkward strings ↔ RaggedTensors instead of making
np.array(["...", "...", ...], dtype=object)
. It's a great example of why that interface would be useful, particularly when paired withak.str.*
functions. (We can demonstrate the difference with a scaling test on some enormous dataset. What's an enormous text dataset? I wonder...)I believe that this is an equivalent for PyTorch's NestedTensor:
https://pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html
I don't know if JAX has a ragged array interface, but if it does, let's get that, too. All of these converters are relatively easy, but we'd have to get TensorFlow, PyTorch, and JAX into at least one of the CI tests. TensorFlow and JAX can be hard to install.