zincware / MDSuite

A post-processing engine for particle simulations
https://mdsuite.readthedocs.io/
Eclipse Public License 2.0
36 stars 7 forks source link

RDF issue that occasionally happens in the CI #504

Open PythonFZ opened 2 years ago

PythonFZ commented 2 years ago
traj_file = '/tmp/pytest-of-runner/pytest-0/NaCl_gk_i_q.lammpstraj'
true_values = {'Cl_Cl': {'left': 0.2372193090684762, 'min_pomf': 5.4953035102048034e-11, 'pomf': [inf, inf, inf, inf, inf, inf, ...]...20018507094386176, 0.0030027760641579263, 0.004003701418877235, 0.005004626773596544, 0.006005552128315853, ...], ...}}
tmp_path = PosixPath('/tmp/pytest-of-runner/pytest-0/test_experiment1')

    def test_experiment(traj_file, true_values, tmp_path):
        """Test the pomf called from the experiment class"""
        os.chdir(tmp_path)
        project = mds.Project()
        project.add_experiment(
            "NaCl", simulation_data=traj_file, timestep=0.002, temperature=1400
        )

>       computation = project.experiments["NaCl"].run.PotentialOfMeanForce(plot=False)

/home/runner/work/MDSuite/MDSuite/CI/integration_tests/calculators/test_potential_of_mean_force.py:75: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/runner/work/MDSuite/MDSuite/mdsuite/calculators/calculator.py:125: in inner
    cls.run_analysis()
/home/runner/work/MDSuite/MDSuite/mdsuite/calculators/calculator.py:321: in run_analysis
    self.run_calculator()
/home/runner/work/MDSuite/MDSuite/mdsuite/calculators/potential_of_mean_force.py:236: in run_calculator
    calculations = self.experiment.run.RadialDistributionFunction(plot=False)
/home/runner/work/MDSuite/MDSuite/mdsuite/calculators/calculator.py:125: in inner
    cls.run_analysis()
/home/runner/work/MDSuite/MDSuite/mdsuite/calculators/calculator.py:321: in run_analysis
    self.run_calculator()
/home/runner/work/MDSuite/MDSuite/mdsuite/calculators/radial_distribution_function.py:854: in run_calculator
    enumerate(batch_ds), ncols=70, disable=batch_tqm, total=self.n_batches
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/tqdm/std.py:1180: in __iter__
    for obj in iterable:
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:836: in __next__
    return self._next_internal()
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:822: in _next_internal
    output_shapes=self._flat_output_shapes)
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/tensorflow/python/ops/gen_dataset_ops.py:2923: in iterator_get_next
    _ops.raise_from_not_ok_status(e, name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

e = _NotOkStatusException(), name = None

    def raise_from_not_ok_status(e, name):
      e.message += (" name: " + name if name is not None else "")
>     raise core._status_to_exception(e) from None  # pylint: disable=protected-access
E     tensorflow.python.framework.errors_impl.UnknownError: IndexError: list index out of range
E     Traceback (most recent call last):
E     
E       File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 271, in __call__
E         ret = func(*args)
E     
E       File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 642, in wrapper
E         return func(*args, **kwargs)
E     
E       File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1004, in generator_py_func
E         values = next(generator_state.get_iterator(iterator_id))
E     
E       File "/home/runner/work/MDSuite/MDSuite/mdsuite/database/data_manager.py", line 204, in generator
E         select_slice = np.s_[self.atom_selection, loop_array[batch]]
E     
E     IndexError: list index out of range
E     
E     
E        [[{{node PyFunc}}]] [Op:IteratorGetNext]

/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:7186: UnknownError

see also https://github.com/zincware/MDSuite/runs/5309604368?check_suite_focus=true

SamTov commented 2 years ago

So I have tried again in #493 to resolve this by essentially further ensuring that there can be NO remainder in the batch but there could be a better way to avoid this. I would change the priority of this until we see it arise again but not close it until we can test this property. Perhaps in #476 this can actually be tested correctly by forcing a remainder in the batch.