Array Type insertion

[lukecampbell]

What's the interface for array types being set?

[caseybryant]

I'm not sure what you mean.

[lukecampbell]

I'm trying to use set_parameter_values but one of the fields is an array type and this error is raised, so I'm assuming I'm just using the incorrect interface.

   ----- exception: param_array and alignment_array must have the same number of elements -----
tion/science_granule_ingestion_worker.py:459    self.add_granule(stream_id, rdt)
tion/science_granule_ingestion_worker.py:622    self.insert_values(coverage, rdt, stream_id)
tion/science_granule_ingestion_worker.py:530    np_dict[k] = NumpyParameterData(k, value, time_array)
e-model/coverage_model/parameter_data.py:103    raise ValueError("param_array and alignment_array must have the same number of elements")

[caseybryant]

This error is thrown if, in science_granule_ingestion_worker.py:530, value.size != time_array.size.

We are trying to ensure data alignment across a write. In this case, it appears you’re telling us there are time_array.size data points, but you provided value.size data points. We don’t know how to resolve alignment (i.e. at what time each index in value was observed).

By the way, the alignment_array (time_array) argument is a convenience option to provide visibility into how the parameter data aligns. It is not persisted, nor is it required.
To provide the least amount of fragmenting on disk and fewest writes to disk/db, the optimal method is to create a dictionary of NumpyParameterData objects and submit the dictionary to set_parameter_values. This method will validate that the alignment parameter, coverage.temporal_parameter_name, is in the dictionary, and that all instances of NumpyPameterData contain the same number of elements for alignment.

[lukecampbell]

I made a test to show what I mean.

Array types are potentially ragged arrays, the test only shows a proper array of arrays.

[caseybryant]

I fixed the problem that existed in the test. What do you mean by ragged arrays? Something like this? np.array([ [0,1], [2,3,4,5] ])

[lukecampbell]

Yes.

If you try to run the above, numpy won't even allow it. I don't remember how Chris dealt with it but ragged arrays is something that the sensors produce.

[caseybryant]

numpy allows it, but the dtype is object. The resulting array is: shape - (2,) dtype - object object - list of ints

If you want to store it as a numpy array, you have to parse through every element, convert, calculate the slice sizes, write an numpy array, and store the array of slice sizes and data.

[lukecampbell]

I'm not exactly sure how we dealt with it before but I think it was converted to a full numpy array where the inner dimension was the size of the largest array and where the other arrays lacked numbers there was a fill value.

This was transparent to the client though.

[caseybryant]

Commit 9b5e26bee2292eb00ecd5f7e533325ab40386bf0 allows the interface to work, but there is some messiness under the covers that still needs to be sorted out.

[lukecampbell]

K, I'll give it a shot shortly.

I'll also make a new issue for ragged array support and close this out.