can't read a branch

[ivukotic]

there is a strange issue when trying to read a branch from an ATLAS physlite file. I use uproot 5.2.1 and awkward 2.5.2.

Here the smallest reproducible example:

import uproot
import awkward as ak
fn='root://xcache.af.uchicago.edu:1094//'
fn+='root://dcgftp.usatlas.bnl.gov:1094//pnfs/usatlas.bnl.gov/LOCALGROUPDISK/rucio/data18_13TeV/04/9a/DAOD_PHYSLITE.34857549._000001.pool.root.1'
with uproot.open(fn) as f:
    tree=f['CollectionTree;1']
    a=tree["PrimaryVerticesAuxDyn.trackParticleLinks"].array()

this file is already cached so you should be able to access it without authentication.

this is the result:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[3], line 7
      5 with uproot.open(fn) as f:
      6     tree=f['CollectionTree;1']
----> 7     a=tree["PrimaryVerticesAuxDyn.trackParticleLinks"].array()
      8     print(a.to_list()[:2])

File /venv/lib/python3.9/site-packages/uproot/behaviors/TBranch.py:1815, in TBranch.array(self, interpretation, entry_start, entry_stop, decompression_executor, interpretation_executor, array_cache, library, ak_add_doc)
   1812                 ranges_or_baskets.append((branch, basket_num, range_or_basket))
   1814 interp_options = {"ak_add_doc": ak_add_doc}
-> 1815 _ranges_or_baskets_to_arrays(
   1816     self,
   1817     ranges_or_baskets,
   1818     branchid_interpretation,
   1819     entry_start,
   1820     entry_stop,
   1821     decompression_executor,
   1822     interpretation_executor,
   1823     library,
   1824     arrays,
   1825     False,
   1826     interp_options,
   1827 )
   1829 _fix_asgrouped(
   1830     arrays,
   1831     expression_context,
   (...)
   1835     ak_add_doc,
   1836 )
   1838 if array_cache is not None:

File /venv/lib/python3.9/site-packages/uproot/behaviors/TBranch.py:3142, in _ranges_or_baskets_to_arrays(hasbranches, ranges_or_baskets, branchid_interpretation, entry_start, entry_stop, decompression_executor, interpretation_executor, library, arrays, update_ranges_or_baskets, interp_options)
   3139     pass
   3141 elif isinstance(obj, tuple) and len(obj) == 3:
-> 3142     uproot.source.futures.delayed_raise(*obj)
   3144 else:
   3145     raise AssertionError(obj)

File /venv/lib/python3.9/site-packages/uproot/source/futures.py:38, in delayed_raise(exception_class, exception_value, traceback)
     34 def delayed_raise(exception_class, exception_value, traceback):
     35     """
     36     Raise an exception from a background thread on the main thread.
     37     """
---> 38     raise exception_value.with_traceback(traceback)

File /venv/lib/python3.9/site-packages/uproot/behaviors/TBranch.py:3084, in _ranges_or_baskets_to_arrays.<locals>.basket_to_array(basket)
   3081 context = dict(branch.context)
   3082 context["forth"] = forth_context[branch.cache_key]
-> 3084 basket_arrays[basket.basket_num] = interpretation.basket_array(
   3085     basket.data,
   3086     basket.byte_offsets,
   3087     basket,
   3088     branch,
   3089     context,
   3090     basket.member("fKeylen"),
   3091     library,
   3092     interp_options,
   3093 )
   3094 if basket.num_entries != len(basket_arrays[basket.basket_num]):
   3095     raise ValueError(
   3096         """basket {} in tree/branch {} has the wrong number of entries """
   3097         """(expected {}, obtained {}) when interpreted as {}
   (...)
   3105         )
   3106     )

File /venv/lib/python3.9/site-packages/uproot/interpretation/objects.py:145, in AsObjects.basket_array(self, data, byte_offsets, basket, branch, context, cursor_offset, library, options)
    134 assert basket.byte_offsets is not None
    136 if self._forth and (
    137     isinstance(
    138         library,
   (...)
    143     )
    144 ):
--> 145     output = self.basket_array_forth(
    146         data,
    147         byte_offsets,
    148         basket,
    149         branch,
    150         context,
    151         cursor_offset,
    152         library,
    153         options,
    154     )
    156 else:
    157     output = ObjectArray(
    158         self._model, branch, context, byte_offsets, data, cursor_offset
    159     ).to_numpy()

File /venv/lib/python3.9/site-packages/uproot/interpretation/objects.py:231, in AsObjects.basket_array_forth(self, data, byte_offsets, basket, branch, context, cursor_offset, library, options)
    229 # this thread tries to do it!
    230 try:
--> 231     output = self._discover_forth(
    232         data, byte_offsets, branch, context, cursor_offset
    233     )
    234 except CannotBeForth:
    235     self._forth = False

File /venv/lib/python3.9/site-packages/uproot/interpretation/objects.py:390, in AsObjects._discover_forth(self, data, byte_offsets, branch, context, cursor_offset)
    383 chunk = uproot.source.chunk.Chunk.wrap(
    384     branch.file.source, data[byte_offsets[i] : byte_offsets[i + 1]]
    385 )
    386 cursor = uproot.source.cursor.Cursor(
    387     0, origin=-(byte_offsets[i] + cursor_offset)
    388 )
--> 390 context["forth"].gen.reset_active_node()
    391 output[i] = self._model.read(
    392     chunk, cursor, context, branch.file, branch.file.detached, branch
    393 )
    395 derived_form = context["forth"].gen.model.derive_form()

AttributeError: 'NoneType' object has no attribute 'reset_active_node'

[jpivarski]

The first thing I noticed is that this TBranch includes an ElementLink data type.

>>> tree["PrimaryVerticesAuxDyn.trackParticleLinks"].typename
'std::vector<std::vector<ElementLink<DataVector<xAOD::TrackParticle_v1>>>>'

I wonder if it's related to #951.

The error message is wrong: if it is the case that we can't deserialize it, it should be a DeserializationError. This error comes from thinking the context["forth"].gen is still active when it's not. I'll check that.

Meanwhile, we can circumvent Uproot's attempt to use AwkwardForth by loading it with library="np".

>>> tree["PrimaryVerticesAuxDyn.trackParticleLinks"].array(library="np")
array([<STLVector [[]] at 0x751ef3eb3910>,
       <STLVector [[]] at 0x751ef3eb38b0>,
       <STLVector [[]] at 0x751ef3eb3130>, ...,
       <STLVector [[]] at 0x751ef3569ed0>,
       <STLVector [[]] at 0x751ef3569f90>,
       <STLVector [[]] at 0x751ef356a050>], dtype=object)

It worked! Okay, so this is a TBranch that we can deserialize, but possibly not with AwkwardForth (and that line with context["forth"].gen is insufficiently guarded).

[Superharz]

I observed the same error with Physlite files. In my case, it happens with parentLinks and childLinks in the truth records. For some information (e.g. taus) both branches can be read. For other particles (e.g. muons) the childLinks always fail. The most interesting are the parentLinks. I observed that if I try to read this branch multiple times it most of the times results in the AttributeError: 'NoneType' object has no attribute 'reset_active_node' error but sometimes it does not throw an error and then the parentLinks are filled correctly. I don't need the links at the moment. So my workaround is simply to not read these branches.

[jpivarski]

It fails in cases in which whole TBaskets consist of empty lists. The AwkwardForth-discovery process is in a state in which the Forth code hasn't been generated yet because it hasn't seen a full example datum, but it hasn't given up yet because it might still find a full datum. This was tested in our small (mostly single-TBucket) test files, but the cases you've seen, @ivukotic and @Superharz, are in this state when transitioning from one TBasket to the next. The indicator of this state is when context["forth"].vm doesn't exist at startup or is equal to None after a TBasket, and the latter state wasn't correctly checked.

But, fortunately, the data are readable. It's not related to #951.

[Superharz]

Thank you for the explanation. But why does it sometimes succeeds with reading a file and most of the times not while reading the exact same file? I should mention that I test this in a Jupyter Notebook by simply re-runnig the cell to read the branch until it does not throw an error. So maybe it could be some IPython stuff.

[jpivarski]

That... does not make sense. Unless maybe you're using an interpretation_executor to read the TBaskets with threads, in which case, there could be a race condition? If read sequentially without threads (the default), this ought to be deterministic.

[Superharz]

This is all I am doing right now:

with uproot.open(path + filename) as f:
    f[tree]["TruthBottomAuxDyn.parentLinks"].array()

It sometimes works, but most of the times does not work. This behavior also stays the same if I restart the Python kernel after each try.

[jpivarski]

Well, I can't think of anything non-deterministic in this process, but it is a complex process.

The variable in question is an attribute of a thread-local variable so that it behaves properly if multithreading is involved (though most of the time, it's not). Maybe this isn't as deterministic as I think it is?

Actually, the TBaskets can arrive in any order—the server is not obliged to send them in file-order if it doesn't want to, and then Uproot would deal with them in the order they're received. That's a source of non-determinism. I don't think it can apply to local files, even though we're getting them through fsspec now.

But anyway, the point may be moot, since the fix has been merged into main. If you pip install -e . from Uproot in the main branch, you shouldn't see the issue at all. Is that the case?

[Superharz]

I tested it. First: Sometimes it works:

>>> with uproot.open(path + filename) as f:
...     f[tree]["TruthBottomAuxDyn.childLinks"].array()
...
<Array [[], [], [], [], ..., [], [], [], []] type='40000 * var * var * stru...'>

But sometimes this non-deterministic error happens with the exact same input file. However, the error message is now different from before:

>>> with uproot.open(path + filename) as f:
...     f[tree]["TruthBottomAuxDyn.childLinks"].array()
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\uproot\behaviors\TBranch.py", line 1815, in array
    _ranges_or_baskets_to_arrays(
  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\uproot\behaviors\TBranch.py", line 3142, in _ranges_or_baskets_to_arrays
    uproot.source.futures.delayed_raise(*obj)
  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\uproot\source\futures.py", line 38, in delayed_raise
    raise exception_value.with_traceback(traceback)
  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\uproot\behaviors\TBranch.py", line 3111, in basket_to_array
    arrays[branch.cache_key] = interpretation.final_array(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\uproot\interpretation\objects.py", line 475, in final_array
    output = numpy.concatenate(trimmed)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\awkward\highlevel.py", line 1527, in __array_function__
    return ak._connect.numpy.array_function(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\awkward\_connect\numpy.py", line 102, in array_function
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\awkward\_connect\numpy.py", line 142, in ensure_valid_args
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\awkward\_dispatch.py", line 62, in dispatch
    next(gen_or_result)
  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\awkward\operations\ak_concatenate.py", line 66, in concatenate
    return _impl(arrays, axis, mergebool, highlevel, behavior, attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\awkward\operations\ak_concatenate.py", line 114, in _impl
    content_or_others = ensure_same_backend(
                        ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\awkward\operations\ak_concatenate.py", line 116, in <genexpr>
    ctx.unwrap(
  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\awkward\_layout.py", line 146, in unwrap
    return to_layout_impl(
           ^^^^^^^^^^^^^^^
  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\awkward\operations\ak_to_layout.py", line 177, in _impl
    promoted_layout = ak.operations.from_numpy(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\awkward\_dispatch.py", line 39, in dispatch
    gen_or_result = func(*args, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\awkward\operations\ak_from_numpy.py", line 55, in from_numpy
    from_arraylib(array, regulararray, recordarray),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\awkward\_layout.py", line 347, in from_arraylib
    raise TypeError("Awkward Array does not support arrays with object dtypes.")
TypeError: Awkward Array does not support arrays with object dtypes.

This error occurred while calling

    ak.concatenate(
        [<Array [[], [], [], [], ..., [], [], [], []] type='45 * var * var * ...
    )

[jpivarski]

Open this as a new issue. I think the non-deterministic part might be something unrelated to the first issue. If you can provide an example file, that would help a lot.

I think it's going wrong here:

  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\uproot\behaviors\TBranch.py", line 3111, in basket_to_array
    arrays[branch.cache_key] = interpretation.final_array(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\flori\miniforge3\envs\test\Lib\site-packages\uproot\interpretation\objects.py", line 475, in final_array
    output = numpy.concatenate(trimmed)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^

In the subsequent output, it's trying to make an Awkward Array out of a NumPy array with dtype=object, which isn't allowed. On the line above, I wonder if the list of arrays, trimmed, accidentally has a mix of Awkward Arrays and NumPy dtype=object arrays. (The latter need an additional step to be turned into Awkward Arrays.)

[Superharz]

I opened the issue in #1101