Adios Modules Does not Work

ax3l commented 9 years ago

The Adios module does not work due to several problems.

(adios 1.9.0 backend with python pip version 1.8.0.post2 and pyDive 1.2.2)

Minor:

the documentation is outdated. Module name is pyDive.adios not pyDive.ad

Blocker:

calling partPosx = pyDive.adios.open(adiosName, '/data/0/particles/i/globalCellIdx/x') on a variable leads to an error in the node = variable_path_iter.next() call in open::update_tree() in pyDive/arrays/ad_ndarray.py: output (StopIteration)
probably caused due to the fact that adios variables (and attributes) are currently not nicely iterate-able as in h5py (e.g., attributes are pure strings with the full path as a key; ... not sure what causes the problem here with the variable)
I tried "assuming" the input (arguments: ({}, '/data/0/particles/i/globalCellIdx/x', <listiterator object at 0x7fd4cd1a3350>, 'x')) are "node == leaf" but then it failed again in open_variable with:

[...]
  File "[...]/lib/python2.7/site-packages/pyDive/arrays/ad_ndarray.py", line 69, in open_variable
    .format(result.name, filename, variable_path), targets=result.target_ranks)
  File "<string>", line 2, in execute
  File "[...]/lib/python2.7/site-packages/IPython/parallel/client/view.py", line 55, in sync_results
[...]

this line is called with the arguments ('dist_array0', '<myAdiosName>.bp', '/data/0/particles/i/globalCellIdx/x') ~~while outside of the views shape[0]=217167841L and offset[0]=undefined (at least before executing the view).~~ the values of print(target_shapes, target_offset_vectors) are:

([[3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393248], [3393217L]],

[[0], [3393248], [6786496], [10179744], [13572992], [16966240], [20359488], [23752736], [27145984], [30539232], [33932480], [37325728], [40718976], [44112224], [47505472], [50898720], [54291968], [57685216], [61078464], [64471712], [67864960], [71258208], [74651456], [78044704], [81437952], [84831200], [88224448], [91617696], [95010944], [98404192], [101797440], [105190688], [108583936], [111977184], [115370432], [118763680], [122156928], [125550176], [128943424], [132336672], [135729920], [139123168], [142516416], [145909664], [149302912], [152696160], [156089408], [159482656], [162875904], [166269152], [169662400], [173055648], [176448896], [179842144], [183235392], [186628640], [190021888], [193415136], [196808384], [200201632], [203594880], [206988128], [210381376], [213774624]])

ax3l commented 9 years ago

@Heikman did you find anything?

heikoburau commented 9 years ago

Can this issue be closed?

ax3l commented 9 years ago

need to cross check, trying to provide feedback soon, thanks! :sparkles:

heikoburau commented 8 years ago

Status? The adios test passes.

ax3l commented 8 years ago

Seems like it is still failing in the CI branch with segfaults: https://travis-ci.org/ComputationalRadiationPhysics/pyDive/builds/110949805

ComputationalRadiationPhysics / pyDive

Adios Modules Does not Work #17