xichenpan / ARLDM

Official Pytorch Implementation of Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Models
https://arxiv.org/abs/2211.10950
MIT License
182 stars 28 forks source link

stuck by hydra when running on slurm cluster #7

Closed bibisbar closed 1 year ago

bibisbar commented 1 year ago

Hi, when I tried to run this program on our slurm cluster, I can only get the seed output Global seed set to 1 and nothing else. After checking, I think the problem may be the hydra decorator. Seems like the main function did not even be executed. Do you have some experience or recommendations regarding this problem? It would be really helpful and thanks in advance:)

xichenpan commented 1 year ago

Hi @bibisbar, thansk for you attention! This is a good question, and thanks for open an issue to point it out. I guess it is a hydra issue. I have used pytorch lightning in slurm cluster and it worked fine. To disable hydra, you need to change https://github.com/Flash-321/ARLDM/blob/81523af705a1895e4aa827dc7482501b4be664c8/main.py#L453-L454 into

def main():

as for the config, you may now setup a argparser or maybe use

from omegaconf import OmegaConf
args= OmegaConf.load('config.yaml')

to load yaml config, so that you do not need to modify the code.