search

mmcdermott / nested_ragged_tensors

Utilities for efficiently working with, saving, and loading, collections of connected nested ragged tensors in PyTorch
MIT License
7 stars 1 forks source link

`flatten` does not work with `target_dim=1` #39

Closed mmcdermott closed 1 week ago

mmcdermott commented 1 week ago 
nested_ragged_tensors on  dev [$!] via  v3.11.9 via  NRT 
❯ pytest --doctest-modules -s --ignore=docs 
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.11.9, pytest-8.3.2, pluggy-1.5.0
rootdir: /home/mmd/Projects/nested_ragged_tensors
configfile: pyproject.toml
plugins: hydra-core-1.3.2, cov-5.0.0
collected 25 items                                                                                                                                                                           

sample_dataset_builder/src/sample_dataset_builder/dataset_generator.py .....
src/nested_ragged_tensors/ragged_numpy.py .........F..........

========================================================================================== FAILURES ==========================================================================================
______________________________________________________ [doctest] nested_ragged_tensors.ragged_numpy.JointNestedRaggedTensorDict.flatten ______________________________________________________
937                    [3, 3, 2, 2, 0, 0, 0]], dtype=uint8)
938             >>> dense_dict['val']
939             array([[1. , 0.2, 0. , 3.1, 0. , 1. , 2.2],
940                    [3. , 3.3, 2. , 0. , 0. , 0. , 0. ]], dtype=float32)
941             >>> J = JointNestedRaggedTensorDict({
942             ...     "ts": [1,         2,      3],
943             ...     "id": [[1, 2, 3], [3, 4], [1, 2]],
944             ...     "val": [[1.0, 0.2, 0.], [3.1, 0.], [1., 2.2]],
945             ... })
946             >>> dense_dict = J.flatten().to_dense()
UNEXPECTED EXCEPTION: KeyError('dim0/bounds')
Traceback (most recent call last):
  File "/home/mmd/mambaforge/envs/NRT/lib/python3.11/doctest.py", line 1355, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest nested_ragged_tensors.ragged_numpy.JointNestedRaggedTensorDict.flatten[12]>", line 1, in <module>
  File "/home/mmd/Projects/nested_ragged_tensors/src/nested_ragged_tensors/ragged_numpy.py", line 974, in flatten
    prev_bounds = self.tensors[f"dim{target_dim-1}/bounds"]
                  ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'dim0/bounds'
/home/mmd/Projects/nested_ragged_tensors/src/nested_ragged_tensors/ragged_numpy.py:946: UnexpectedException
================================================================================== short test summary info ===================================================================================
FAILED src/nested_ragged_tensors/ragged_numpy.py::nested_ragged_tensors.ragged_numpy.JointNestedRaggedTensorDict.flatten
================================================================================ 1 failed, 24 passed in 8.76s ================================================================================
mmcdermott commented 1 week ago

FYI @Oufattole