dimosped
commented
6 years ago
def append(self, arctic_lib, version, symbol, item, previous_version, dtype=None, dirty_append=True):
collection = arctic_lib.get_top_level_collection()
if previous_version.get('shape', [-1]) != [-1, ] + list(item.shape)[1:]:
raise UnhandledDtypeException()
if not dtype:
dtype = item.dtype
if previous_version['up_to'] == 0:
dtype = dtype
elif len(item) == 0:
dtype = self._dtype(previous_version['dtype'])
else:
dtype = self._promote_types(dtype, previous_version['dtype'])
item = item.astype(dtype)
if str(dtype) != previous_version['dtype']:
logger.debug('Converting %s from %s to %s' % (symbol, previous_version['dtype'], str(dtype)))
if item.dtype.hasobject:
raise UnhandledDtypeException()
version['dtype'] = str(dtype)
version['dtype_metadata'] = dict(dtype.metadata or {})
version['type'] = self.TYPE
> old_arr = self._do_read(collection, previous_version, symbol).astype(dtype)
E ValueError: structures must have the same size
../../../arctic/store/_ndarray_store.py:240: ValueError
================= 1 failed, 1 pytest-warnings in 1.68 seconds ==================
Process finished with exit code 0
bmoscon
Frankkkkk
CrazyAlvaro
Arctic Version
v1.56.0
Arctic Store
VersionStore, TickStore
Description of problem and/or code sample that reproduces the issue
We use astype() to convert to a a dtype which can be a superset of the existing numpy array: https://github.com/manahl/arctic/blob/master/arctic/store/_ndarray_store.py#L240
In this case, using Numpy==1.14.0 prohibits the conversion between dtypes of non equal sizes due to this change: https://github.com/numpy/numpy/commit/7412b85a1b4920e5b72ab694e51b11e4cb615866#diff-d3108a3599564ada8ce08fe45f539350R2674
An example test case to replicate the issue: https://github.com/manahl/arctic/blob/master/tests/integration/store/test_ndarray_store_append.py#L193