Excessive Memory Usage in `band_cal_parallel.py` for Large Systems

[newplay]

Dear YangZhong,

I am writing to you regarding a memory issue I am encountering while calculating the band structure of a system with 3903 atoms using band_cal_parallel.py. With parameters set to max_orb = 26 and k-points num = 90, the calculation unexpectedly requires more than 2TB of RAM, exceeding our server's limit and causing the node to shut down.

I am puzzled by this large memory requirement. Could you please shed some light on the diagonalization method employed in the script and how it contributes to the memory usage?

Since my primary interest lies in the electronic structure near the Fermi level, would it be possible to modify the script to calculate and store only the bands within a range of ±100 bands around the Fermi energy? This could potentially help reduce the memory footprint of the calculation.

Thank you for your time and guidance.

Sincerely,

TzuChing

[QuantumLab-ZY]

Hi, I would like to check the parameters in your band_cal_parallel.yaml file.

[newplay]

Hi, I would like to check the parameters in your band_cal_parallel.yaml file.

Hi, thanks for your response. Here is my configuration file for band_cal_parallel.py:

filename: MoS2_WSe2_hetero
structure_name: MoS2_WSe2_hetero
graph_data_path: /home5/zjlin/ML_work/HamGNN/Bilayer_TMD/work_dir/openmx_test/transform_moire_MoS2_WSe2/HamGNN_result/heter_moire/graph/graph_data.npz
hamiltonian_path: /home5/zjlin/ML_work/HamGNN/Bilayer_TMD/work_dir/openmx_test/transform_moire_MoS2_WSe2/HamGNN_result/heter_moire/band/version_0/prediction_hamiltonian.npy # The path to the prediction_hamiltonian.npy file. If null, then the Hamiltonian in graph_data.npz will be used instead.

# If k_path and label are null, the k_path will be set automatically.
k_path: [[0., 0., 0.], [0.5, 0., 0.0], [0.333333, 0.333333, 0.], [0., 0., 0.]] 
label: ['$Gamma$', '$M$', '$K$', '$Gamma$']
nk: 90 # The total number of k points in all k paths
auto_mode: False

nao_max: 26
num_wfns: 10 # Export the wave functions in the interval of [VBM-num_wfn, VBM+num_wfn]
save_dir: /home5/zjlin/ML_work/HamGNN/Bilayer_TMD/work_dir/openmx_test/transform_moire_MoS2_WSe2/HamGNN_result/heter_moire/band
soc_switch: False
Ham_type: openmx # openmx or abacus

The pbs_job.sh script is as follows:

#! /bin/bash
#PBS -N HamGNN_hetero_MoS2_WSe2
#PBS -e HamGNN_hetero_MoS2_WSe2.err
#PBS -o HamGNN_hetero_MoS2_WSe2.log
#PBS -V
#PBS -l nodes=1:ppn=32,walltime=9999:00:00

cd $PBS_O_WORKDIR
source activate
conda deactivate
conda activate HamGNN
mpirun -np 32 band_cal_parallel --config band_cal_parallel.yaml

The POSCAR file is attached: POSCAR file

I used the model of MoW(SeS)2_model we discussed before. Because the graph_data.npz and prediction_hamiltonian.npy files are too large, I assume if you evaluate the model by yourself, you will get the same results. Best Regards TzuChing

[QuantumLab-ZY]

Hi, TzuChing

It's too difficult to diagonalize the Hamiltonian matrix of a structure with up to 3903 atoms using only 32 cores. You can try to find more computing resources (at least 64 cores and enough memory). If you really can't find more computing resources, you can put the relevant structure and Hamiltonian matrix into the Zenodo database, and I will help you with diagonalization.

Best wishes, Yang Zhong

[newplay]

Hi, TzuChing

It's too difficult to diagonalize the Hamiltonian matrix of a structure with up to 3903 atoms using only 32 cores. You can try to find more computing resources (at least 64 cores and enough memory). If you really can't find more computing resources, you can put the relevant structure and Hamiltonian matrix into the Zenodo database, and I will help you with diagonalization.

Best wishes, Yang Zhong Hi, Yang Zhong,

Thank you very much for your suggestion and willingness to help with the diagonalization of the Hamiltonian matrix for my structure. I truly appreciate your advice on seeking more computing resources.

However, I don't want to waste your time and resources on my project. I will attempt to simplify and approximate the problem as much as possible on my end.

Again, thank you for your generous offer and support.

Best wishes, TzuChing