Too many levels of symbolic links for last.ckpt

[fealke]

Hello, I was just trying to run all steps of the given prolificdreamer commands of the Colab. The first one worked as intended:

# Train
!python launch.py --config configs/prolificdreamer.yaml --train --gpu 0 system.prompt_processor.prompt="$prompt" data.width=64 data.height=64 data.batch_size=1

which resulted in ... Test results saved to outputs/prolificdreamer/a_detailed_green_fern_in_a_wooden_pot@20231108-074143/save

Now, in the refinment part, I used this command:

# Geometry Refinment
!python launch.py --config configs/prolificdreamer-geometry.yaml --train --gpu 0 system.prompt_processor.prompt="$prompt" system.geometry_convert_from=/content/threestudio/outputs/prolificdreamer/a_detailed_green_fern_in_a_wooden_pot@20231108-074143/ckpts/last.ckpt

and got the following issue:

2023-11-08 07:56:31.530199: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-11-08 07:56:31.530253: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-11-08 07:56:31.530293: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-11-08 07:56:33.287855: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Seed set to 0
[INFO] Initializing geometry from a given checkpoint ...
Traceback (most recent call last):
  File "/content/threestudio/launch.py", line 237, in <module>
    main(args, extras)
  File "/content/threestudio/launch.py", line 105, in main
    system: BaseSystem = threestudio.find(cfg.system_type)(
  File "/content/threestudio/threestudio/systems/base.py", line 45, in __init__
    self.configure()
  File "/content/threestudio/threestudio/systems/prolificdreamer.py", line 25, in configure
    super().configure()
  File "/content/threestudio/threestudio/systems/base.py", line 266, in configure
    state_dict, epoch, global_step = load_module_weights(
  File "/content/threestudio/threestudio/utils/misc.py", line 40, in load_module_weights
    ckpt = torch.load(path, map_location=map_location)
  File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 986, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 435, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 416, in __init__
    super().__init__(open(name, mode))
OSError: [Errno 40] Too many levels of symbolic links: '/content/threestudio/outputs/prolificdreamer/a_detailed_green_fern_in_a_wooden_pot@20231108-074143/ckpts/last.ckpt'

Does anyone know how to solve this problem in Colab? I would be very grateful for any help. Many thanks in advance!

[bennyguo]

Hi! Could you please check the lightning version by pip list | grep lightning?

[fealke]

This is the output of the command

lightning                        2.0.0
lightning-cloud                  0.5.52
lightning-utilities              0.9.0
pytorch-lightning                2.1.1

[bennyguo]

@fealke Uninstall these four libraries and reinstall lightning by pip install lightning==2.0.0 should solve the problem.

[bond007alex]

pip install lightning==2.0.0 even after a cache purge just reinstalls the 4 packages in colabs

[fealke]

Unfortunately, even after removing all packages and reinstalling lightning==2.0.0, the error remains that last.ckpt cannot be accessed due to symbolic links.