spacetelescope / roman_datamodels

Datamodel support for the roman calibration pipeline
https://roman-datamodels.readthedocs.io
Other
7 stars 20 forks source link

`DataModel.to_flat_dict` includes arrays when using `include_arrays=False` #366

Open braingram opened 1 month ago

braingram commented 1 month ago

If I open a roman file (one of the regression test files for romancal) and use to_flat_dict(include_arrays=False) arrays are included in the flat dictionary.

>>> import roman_datamodels as rdm
>>> m = rdm.open("test_bigdata/roman-pipeline/dev/WFI/image/r0000101001001001001_01101_0001_WFI01_cal.asdf")
>>> fd = m.to_flat_dict(include_arrays=False)
>>> fd["roman.meta.dq"]
array([[1048576, 1048576, 1048576, ..., 1048576, 1048576, 1048576],
       [      0,       0,       0, ...,       4,       0,       0],
       [1048576, 1048576, 1048576, ..., 1048576, 1048576, 1048576],
       ...,
       [      0,       0,       0, ...,       0,       0,       0],
       [      0,       0,       0, ...,       0,       0,       0],
       [1048576, 1048576, 1048576, ..., 1048576, 1048576, 1048576]],
      dtype=uint32)
schlafly commented 1 month ago

Poking a little more at this one, it looks like the issue is that roman.dq is an asdf.tags.core.ndarray.NDArrayType rather than a np.ndarray. https://github.com/spacetelescope/roman_datamodels/blob/911485a5913e61a42dc0b79c83325eadec310d3a/src/roman_datamodels/datamodels/_core.py#L301

We should probably think about alternatives to isinstance(...) checks as these kinds of issues pop up rather frequently.

In [8]: cal.to_flat_dict()['roman.dq']
Out[8]: 
array([[1048576, 1048576, 1048576, ..., 1048576, 1048576, 1048576],
       [      0,       0,       0, ...,       4,       0,       0],
       [1048576, 1048576, 1048576, ..., 1048576, 1048576, 1048576],
       ...,
       [      0,       0,       0, ...,       0,       0,       0],
       [      0,       0,       0, ...,       0,       0,       0],
       [1048576, 1048576, 1048576, ..., 1048576, 1048576, 1048576]],
      dtype=uint32)

In [9]: type(cal.to_flat_dict()['roman.dq'])
Out[9]: asdf.tags.core.ndarray.NDArrayType

In [10]: isinstance(cal.to_flat_dict()['roman.dq'], np.ndarray)
Out[10]: False