demo_abacus with ABACUS 3.1

[NanomagLab]

Hello.

I'm trying to follow the "demo_abacus" with ABACUS 3.1.

I've made raw data using ABACUS 3.1.

After that, I tried the deeph-preprocess.

deeph-preprocess --config preprocess.ini

And it occur :

User config name: ['preprocess.ini']
Found 400 directories to preprocess
Preprocessing No. 1/400 [09:00:00<?]...Output subdirectories: OUT.ABACUS
Traceback (most recent call last):
  File "/home/whal1235/miniconda3/bin/deeph-preprocess", line 8, in <module>
    sys.exit(main())
  File "/home/whal1235/miniconda3/lib/python3.9/site-packages/deeph/scripts/preprocess.py", line 147, in main
    worker(index)
  File "/home/whal1235/miniconda3/lib/python3.9/site-packages/deeph/scripts/preprocess.py", line 118, in worker
    abacus_parse(abspath, os.path.abspath(relpath), 'OUT.' + abacus_suffix)
  File "/home/whal1235/miniconda3/lib/python3.9/site-packages/deeph/preprocess/abacus_get_data.py", line 246, in abacus_parse
    hamiltonian_dict, tmp = parse_matrix(
  File "/home/whal1235/miniconda3/lib/python3.9/site-packages/deeph/preprocess/abacus_get_data.py", line 204, in parse_matrix
    num_element = int(line1[3])
ValueError: invalid literal for int() with base 10: 'H(R):'

I think it occur from output file format difference between ABACUS 2.XX and ABACUS 3.1.

This is an example of ABACUS 3.1 output file.

OUT.ABACUS.zip

Can you fix it? Or do you recomend me to install ABACUS 2.xx?

Thank you.

[mzjb]

Thank you for your report. I fixed it in this commit.

[NanomagLab]

Thank you for your prompt response. It is greatly appreciated.

Could I continue this issue until I finish my demo_abacus in ABACUS 3.1?

I will close this issue once I have completed all the necessary steps. Thank you

[NanomagLab]

Hi!

I have other issues.

I've tried to preprocess my ABACUS output dataset.

deeph-preprocess --config preprocess.ini

It progress well.

And then I tried the deeph-train process.

deeph-train --config train.ini

It starts to making graph data.

At the saving progress. The program was killed

Finish processing 500 structures, have cost **** seconds
Kiiled

It is a memory problem. My memory (128 GB) is not enough.

Another problem is that the graph data is too large to be carried by GPU memory.

The graph data file of only 50 samples is 6.92 GB, which means that 500 samples may be about 70 GB.

I realize that my processed data is much larger than the dataset used in the paper.

Hear is my processed data (./processed/0/, from ABACUS 3.1).

0.zip

Is it something wrong? Or, is it normal in DeepH-pack with ABACUS?

Is it possible to follow the tutorial using OpenMX or ABACUS from a desktop computer?

Thank you.

[mzjb]

@NanomagLab

Hi there,

DeepH code requires approximately twice the memory of your graph data file, which in your case is around 140GB. However, the GPU memory required during training is much lower, typically around 10GB (when batch size equals 1).

In our paper, we used 450 graphene supercells with 72 atoms each, and the corresponding graph data file is ~20GB. As for the reason why your graph file is so large (70GB), it may be due to the large cutoff radius you are using. You can try using a basis set with smaller radius for the DFT calculation, or you can set additional truncation when generating the graph (such as setting radius = 6.0 for the configuration file of deeph-preprocess).

Additionally, we have just released the code for DeepH-E3 (https://arxiv.org/abs/2210.13955; accepted by Nat. Commun.) as an open-source project on Github (https://github.com/Xiaoxun-Gong/DeepH-E3). This method is an upgraded version (performs better in terms of accuracy, training speed, and memory consumption) of DeepH, and the code for DeepH-E3 uses the same data format as DeepH. DeepH needs a lot of memory because it needs to store the orientation relative to the local coordinates, but DeepH-E3 does not need it. For datasets of graphene supercells, the graph file is ~20 GB for DeepH and ~2 GB for DeepH-E3. The DeepH-E3 code will be integrated into the current DeepH-pack repository in the future.

[NanomagLab]

Thank you!

I will try a small radius like you said.

And your new paper is very interesting. I think I should read it.

I'm always so thankful for your kind response.