Open anseongpark opened 1 year ago
Could you please check the configuration of the failed task? and repeat manually the fp calculation to check what happened.
I transfer the issue to dpdata. It looks to me that there is at least a bug in dpdata. When the energy is not found, there should be an error, or dpdata should ignore the frame. None
should not be appended, and I remember NumPy has disallowed such usage.
Summary
Sometimes, energy.raw file at iteration.0000x/02.fp/ doesn't have all the task's energy. For example, when implementing 02.fp step with 30 tasks, other files such as coord.raw, box.raw have all 30 task's data. However, energy.raw sometimes has less than 30, which abrupt the process. From the raw code, I think I found the way that dpgen reads the energy data. After the first principle calculation of a task is finished, DPGEN searches for a specific key word (a specific sentence that has energy value in it) and save the energy data. However, I found out that some of my first principle calculation ended without such specific key word (sentence), so that the DPGEN could not find the task's energy value. Repeating the job several times, it rarely successfully calculated all the tasks so that it proceeds the iteration. I wonder why this happen, really appreciate your help. Thank you.
DPGEN Version and Platform 14 DeepModeling 15 ------------ 16 Version: 0.10.6 17 Date: Nov-01-2022 18 Path: /home/Software/python-3.8.3/lib/python3.8/site-packages/dpgen 19 20 Dependency 21 ------------ 22 numpy 1.23.3 /home/Software/python-3.8.3/lib/python3.8/site-packages/numpy 23 dpdata 0.2.10 /home/Software/python-3.8.3/lib/python3.8/site-packages/dpdata 24 pymatgen 2019.6.5 /home/Software/python-3.8.3/lib/python3.8/site-packages/pymatgen 25 monty 2022.9.9 /home/Software/python-3.8.3/lib/python3.8/site-packages/monty 26 ase 3.22.1 /home/Software/python-3.8.3/lib/python3.8/site-packages/ase 27 paramiko 2.11.0 /home/Software/python-3.8.3/lib/python3.8/site-packages/paramiko 28 custodian 2022.5.26 /home/Software/python-3.8.3/lib/python3.8/site-packages/custodian
error 127 Traceback (most recent call last): 128 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/numpy/lib/npyio.py", line 1565, in savetxt 129 v = format % tuple(row) + newline 130 TypeError: must be real number, not NoneType 131 132 The above exception was the direct cause of the following exception: 133 134 Traceback (most recent call last): 135 File "/home/Software/python-3.8.3/bin/dpgen", line 8, in
136 sys.exit(main())
137 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpgen/main.py", line 185, in main
138 args.func(args)
139 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpgen/generator/run.py", line 3642, in gen_run
140 run_iter (args.PARAM, args.MACHINE)
141 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpgen/generator/run.py", line 3631, in run_iter
142 post_fp (ii, jdata)
143 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpgen/generator/run.py", line 3508, in post_fp
144 post_fp_vasp(iter_index, jdata)
145 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpgen/generator/run.py", line 3151, in post_fp_vasp
146 all_sys.to_deepmd_raw(sys_data_path)
147 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpdata/system.py", line 1476, in to_format
148 return self.to_fmt_obj(ff(), *args, kwargs)
149 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpdata/system.py", line 1087, in to_fmt_obj
150 return fmtobj.to_labeled_system(self.data, *args, *kwargs)
151 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpdata/format.py", line 77, in to_labeled_system
152 return self.to_system(data, args, kwargs)
153 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpdata/plugins/deepmd.py", line 22, in to_system
154 dpdata.deepmd.raw.dump(file_name, data)
155 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpdata/deepmd/raw.py", line 76, in dump
156 np.savetxt(os.path.join(folder, 'energy.raw'), np.reshape(data['energies'], [nframes, 1]))
157 File "<__array_function__ internals>", line 180, in savetxt
158 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/numpy/lib/npyio.py", line 1567, in savetxt
159 raise TypeError("Mismatch between array dtype ('%s') and "
160 TypeError: Mismatch between array dtype ('object') and format specifier ('%.18e')