Database generation has less time steps than file

[Fratorhe]

Describe the bug When I generate the database in a per-atom configuration, the number of time-steps read is a bit smaller. In this case, I did 20.000 steps, and the database loaded 19.987. I guess it may be related to the batching because it happened only in that case and the batching took 23 steps (19.987+23=20.000)...

To Reproduce Steps to reproduce the behavior:

1. Go to tutorials\walkthrough_notebooks\Liquid_Argon
2. Run the LAMMPS case, a file called gk_data.lmp_traj of ~3.5Gb will be created. Running time should be ~7 min.
3. from_traj and run the conductivity_gk.py.
4. check the hdf5 database to see the problem.

Expected behavior Size database wrt input file is inconsistent.

Screenshots

Desktop (please complete the following information):

• OS: [e.g. iOS] Windows 10
• Version [e.g. 22]

Additional context Python 3.7.4

[SamTov]

This is usually because it gets cut by the batching, although in this case it could be something smaller. In the case of batching we would need to add some additional methods to look for excess trajectories and put them in. Perhaps @PythonFZ can discuss some TF stuff.

[PythonFZ]

EDIT: We should be able to use tf.data.Dataset with its batching method to generate batches with different sizes. Usually the last batch would be smaller than the batches before.

[SamTov]

Yes this is what I was thinking

[SamTov]

Sorry I did not mean to close.

[SamTov]

@Fratorhe please check the latest push and see if this has been solved. I did not have any good data to check this quickly.

[SamTov]

Can we close this issue? @Fratorhe

[Fratorhe]

Sorry, I am finishing with the GK thermal and viscosity. I will test it in one hour or so. I will close it if it works.

[SamTov]

Perfect

[Fratorhe]

working!!