/data/anaconda3/envs/codef/lib/python3.10/site-packages/lightning_utilities/core/imports.py:13: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
/data/anaconda3/envs/codef/lib/python3.10/site-packages/lightning_fabric/init.py:36: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('lightning_fabric').
Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packagesimport("pkg_resources").declare_namespace(name)
/data/anaconda3/envs/codef/lib/python3.10/site-packages/torchmetrics/utilities/imports.py:24: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
_PYTHON_LOWER_3_8 = LooseVersion(_PYTHON_VERSION) < LooseVersion("3.8")
/data/anaconda3/envs/codef/lib/python3.10/site-packages/torchmetrics/utilities/imports.py:24: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
_PYTHON_LOWER_3_8 = LooseVersion(_PYTHON_VERSION) < LooseVersion("3.8")
/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/init.py:36: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('pytorch_lightning').
Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packagesimport("pkg_resources").declare_namespace(name)
ModelCheckpoint(save_last=True, save_top_k=-1, monitor=None) will duplicate the last checkpoint saved.
/data/anaconda3/envs/codef/lib/python3.10/site-packages/lightning_fabric/connector.py:562: UserWarning: 16 is supported for historical reasons but its usage is discouraged. Please set your precision to 16-mixed instead!
rank_zero_warn(
Using 16bit Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
all_sequences/scene_0/scene_0/00000.png
all_sequences/scene_0/scene_0/00001.png
all_sequences/scene_0/scene_0/00002.png
all_sequences/scene_0/scene_0/00003.png
all_sequences/scene_0/scene_0/00004.png
all_sequences/scene_0/scene_0/00005.png
all_sequences/scene_0/scene_0/00006.png
all_sequences/scene_0/scene_0/00007.png
all_sequences/scene_0/scene_0/00008.png
all_sequences/scene_0/scene_0/00009.png
all_sequences/scene_0/scene_0/00010.png
all_sequences/scene_0/scene_0/00011.png
all_sequences/scene_0/scene_0/00012.png
all_sequences/scene_0/scene_0/00013.png
all_sequences/scene_0/scene_0/00014.png
all_sequences/scene_0/scene_0/00015.png
all_sequences/scene_0/scene_0/00016.png
all_sequences/scene_0/scene_0/00017.png
all_sequences/scene_0/scene_0/00018.png
all_sequences/scene_0/scene_0/00019.png
all_sequences/scene_0/scene_0/00020.png
all_sequences/scene_0/scene_0/00021.png
all_sequences/scene_0/scene_0/00022.png
all_sequences/scene_0/scene_0/00023.png
all_sequences/scene_0/scene_0/00024.png
all_sequences/scene_0/scene_0/00025.png
all_sequences/scene_0/scene_0/00026.png
all_sequences/scene_0/scene_0/00027.png
all_sequences/scene_0/scene_0/00028.png
all_sequences/scene_0/scene_0/00029.png
all_sequences/scene_0/scene_0/00030.png
all_sequences/scene_0/scene_0/00031.png
all_sequences/scene_0/scene_0/00032.png
all_sequences/scene_0/scene_0/00033.png
all_sequences/scene_0/scene_0/00034.png
all_sequences/scene_0/scene_0/00035.png
all_sequences/scene_0/scene_0/00036.png
all_sequences/scene_0/scene_0/00037.png
all_sequences/scene_0/scene_0/00038.png
all_sequences/scene_0/scene_0/00039.png
all_sequences/scene_0/scene_0/00040.png
all_sequences/scene_0/scene_0/00041.png
all_sequences/scene_0/scene_0/00042.png
all_sequences/scene_0/scene_0/00043.png
all_sequences/scene_0/scene_0/00044.png
all_sequences/scene_0/scene_0/00045.png
all_sequences/scene_0/scene_0/00046.png
all_sequences/scene_0/scene_0/00047.png
all_sequences/scene_0/scene_0/00048.png
all_sequences/scene_0/scene_0/00049.png
all_sequences/scene_0/scene_0/00050.png
all_sequences/scene_0/scene_0/00051.png
all_sequences/scene_0/scene_0/00052.png
all_sequences/scene_0/scene_0/00053.png
all_sequences/scene_0/scene_0/00054.png
all_sequences/scene_0/scene_0/00055.png
all_sequences/scene_0/scene_0/00056.png
all_sequences/scene_0/scene_0/00057.png
all_sequences/scene_0/scene_0/00058.png
all_sequences/scene_0/scene_0/00059.png
all_sequences/scene_0/scene_0/00060.png
all_sequences/scene_0/scene_0/00061.png
all_sequences/scene_0/scene_0/00062.png
all_sequences/scene_0/scene_0/00063.png
all_sequences/scene_0/scene_0/00064.png
all_sequences/scene_0/scene_0/00065.png
all_sequences/scene_0/scene_0/00066.png
all_sequences/scene_0/scene_0/00067.png
all_sequences/scene_0/scene_0/00068.png
all_sequences/scene_0/scene_0/00069.png
all_sequences/scene_0/scene_0/00070.png
all_sequences/scene_0/scene_0/00071.png
all_sequences/scene_0/scene_0/00072.png
all_sequences/scene_0/scene_0/00073.png
all_sequences/scene_0/scene_0/00074.png
all_sequences/scene_0/scene_0/00075.png
all_sequences/scene_0/scene_0/00076.png
all_sequences/scene_0/scene_0/00077.png
all_sequences/scene_0/scene_0/00078.png
all_sequences/scene_0/scene_0/00079.png
all_sequences/scene_0/scene_0/00080.png
all_sequences/scene_0/scene_0/00081.png
all_sequences/scene_0/scene_0/00082.png
all_sequences/scene_0/scene_0/00083.png
all_sequences/scene_0/scene_0/00084.png
all_sequences/scene_0/scene_0/00085.png
all_sequences/scene_0/scene_0/00086.png
all_sequences/scene_0/scene_0/00087.png
all_sequences/scene_0/scene_0/00088.png
all_sequences/scene_0/scene_0/00089.png
all_sequences/scene_0/scene_0/00090.png
all_sequences/scene_0/scene_0/00091.png
all_sequences/scene_0/scene_0/00092.png
all_sequences/scene_0/scene_0/00093.png
all_sequences/scene_0/scene_0/00094.png
all_sequences/scene_0/scene_0/00095.png
all_sequences/scene_0/scene_0/00096.png
all_sequences/scene_0/scene_0/00097.png
all_sequences/scene_0/scene_0/00098.png
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
distributed_backend=nccl
All distributed processes registered. Starting with 1 processes
You are using a CUDA device ('NVIDIA A10') that has Tensor Cores. To properly utilize them, you should set torch.set_float32_matmul_precision('medium' | 'high') which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
/data/anaconda3/envs/codef/lib/python3.10/site-packages/torch/utils/tensorboard/init.py:4: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if not hasattr(tensorboard, "version") or LooseVersion(
/data/anaconda3/envs/codef/lib/python3.10/site-packages/torch/utils/tensorboard/init.py:6: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
) < LooseVersion("1.15"):
Testing DataLoader 0: 1%|? | 1/99 [00:00<00:16, 5.84it/s]/data/anaconda3/envs/codef/lib/python3.10/site-packages/skvideo/io/ffmpeg.py:466: DeprecationWarning: tostring() is deprecated. Use tobytes() instead.
self._proc.stdin.write(vid.tostring())
Traceback (most recent call last):
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/skvideo/io/ffmpeg.py", line 466, in writeFrame
self._proc.stdin.write(vid.tostring())
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/CoDeF/train.py", line 557, in
main(hparams)
File "/data/CoDeF/train.py", line 550, in main
trainer.test(system, dataloaders=system.test_dataloader())
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 706, in test
return call._call_and_handle_interrupt(
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, kwargs)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 92, in launch
return function(*args, *kwargs)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 749, in _test_impl
results = self._run(model, ckpt_path=ckpt_path)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 935, in _run
results = self._run_stage()
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 971, in _run_stage
return self._evaluation_loop.run()
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/loops/utilities.py", line 177, in _decorator
return loop_run(self, args, kwargs)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 115, in run
self._evaluation_step(batch, batch_idx, dataloader_idx)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 375, in _evaluation_step
output = call._call_strategy_hook(trainer, hook_name, step_kwargs.values())
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 288, in _call_strategy_hook
output = fn(args, *kwargs)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 348, in test_step
return self.model.test_step(args, **kwargs)
File "/data/CoDeF/train.py", line 489, in test_step
self.video_visualizer.add(img)
File "/data/CoDeF/utils/video_visualizer.py", line 95, in add
self.video.writeFrame(frame)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/skvideo/io/ffmpeg.py", line 471, in writeFrame
raise IOError(msg)
OSError: [Errno 32] Broken pipe
/data/anaconda3/envs/codef/lib/python3.10/site-packages/lightning_utilities/core/imports.py:13: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html import pkg_resources /data/anaconda3/envs/codef/lib/python3.10/site-packages/lightning_fabric/init.py:36: DeprecationWarning: Deprecated call to
pkg_resources.declare_namespace('lightning_fabric')
. Implementing implicit namespace packages (as specified in PEP 420) is preferred topkg_resources.declare_namespace
. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages import("pkg_resources").declare_namespace(name) /data/anaconda3/envs/codef/lib/python3.10/site-packages/torchmetrics/utilities/imports.py:24: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. _PYTHON_LOWER_3_8 = LooseVersion(_PYTHON_VERSION) < LooseVersion("3.8") /data/anaconda3/envs/codef/lib/python3.10/site-packages/torchmetrics/utilities/imports.py:24: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. _PYTHON_LOWER_3_8 = LooseVersion(_PYTHON_VERSION) < LooseVersion("3.8") /data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/init.py:36: DeprecationWarning: Deprecated call topkg_resources.declare_namespace('pytorch_lightning')
. Implementing implicit namespace packages (as specified in PEP 420) is preferred topkg_resources.declare_namespace
. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages import("pkg_resources").declare_namespace(name) ModelCheckpoint(save_last=True, save_top_k=-1, monitor=None) will duplicate the last checkpoint saved. /data/anaconda3/envs/codef/lib/python3.10/site-packages/lightning_fabric/connector.py:562: UserWarning: 16 is supported for historical reasons but its usage is discouraged. Please set your precision to 16-mixed instead! rank_zero_warn( Using 16bit Automatic Mixed Precision (AMP) GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores IPU available: False, using: 0 IPUs HPU available: False, using: 0 HPUs all_sequences/scene_0/scene_0/00000.png all_sequences/scene_0/scene_0/00001.png all_sequences/scene_0/scene_0/00002.png all_sequences/scene_0/scene_0/00003.png all_sequences/scene_0/scene_0/00004.png all_sequences/scene_0/scene_0/00005.png all_sequences/scene_0/scene_0/00006.png all_sequences/scene_0/scene_0/00007.png all_sequences/scene_0/scene_0/00008.png all_sequences/scene_0/scene_0/00009.png all_sequences/scene_0/scene_0/00010.png all_sequences/scene_0/scene_0/00011.png all_sequences/scene_0/scene_0/00012.png all_sequences/scene_0/scene_0/00013.png all_sequences/scene_0/scene_0/00014.png all_sequences/scene_0/scene_0/00015.png all_sequences/scene_0/scene_0/00016.png all_sequences/scene_0/scene_0/00017.png all_sequences/scene_0/scene_0/00018.png all_sequences/scene_0/scene_0/00019.png all_sequences/scene_0/scene_0/00020.png all_sequences/scene_0/scene_0/00021.png all_sequences/scene_0/scene_0/00022.png all_sequences/scene_0/scene_0/00023.png all_sequences/scene_0/scene_0/00024.png all_sequences/scene_0/scene_0/00025.png all_sequences/scene_0/scene_0/00026.png all_sequences/scene_0/scene_0/00027.png all_sequences/scene_0/scene_0/00028.png all_sequences/scene_0/scene_0/00029.png all_sequences/scene_0/scene_0/00030.png all_sequences/scene_0/scene_0/00031.png all_sequences/scene_0/scene_0/00032.png all_sequences/scene_0/scene_0/00033.png all_sequences/scene_0/scene_0/00034.png all_sequences/scene_0/scene_0/00035.png all_sequences/scene_0/scene_0/00036.png all_sequences/scene_0/scene_0/00037.png all_sequences/scene_0/scene_0/00038.png all_sequences/scene_0/scene_0/00039.png all_sequences/scene_0/scene_0/00040.png all_sequences/scene_0/scene_0/00041.png all_sequences/scene_0/scene_0/00042.png all_sequences/scene_0/scene_0/00043.png all_sequences/scene_0/scene_0/00044.png all_sequences/scene_0/scene_0/00045.png all_sequences/scene_0/scene_0/00046.png all_sequences/scene_0/scene_0/00047.png all_sequences/scene_0/scene_0/00048.png all_sequences/scene_0/scene_0/00049.png all_sequences/scene_0/scene_0/00050.png all_sequences/scene_0/scene_0/00051.png all_sequences/scene_0/scene_0/00052.png all_sequences/scene_0/scene_0/00053.png all_sequences/scene_0/scene_0/00054.png all_sequences/scene_0/scene_0/00055.png all_sequences/scene_0/scene_0/00056.png all_sequences/scene_0/scene_0/00057.png all_sequences/scene_0/scene_0/00058.png all_sequences/scene_0/scene_0/00059.png all_sequences/scene_0/scene_0/00060.png all_sequences/scene_0/scene_0/00061.png all_sequences/scene_0/scene_0/00062.png all_sequences/scene_0/scene_0/00063.png all_sequences/scene_0/scene_0/00064.png all_sequences/scene_0/scene_0/00065.png all_sequences/scene_0/scene_0/00066.png all_sequences/scene_0/scene_0/00067.png all_sequences/scene_0/scene_0/00068.png all_sequences/scene_0/scene_0/00069.png all_sequences/scene_0/scene_0/00070.png all_sequences/scene_0/scene_0/00071.png all_sequences/scene_0/scene_0/00072.png all_sequences/scene_0/scene_0/00073.png all_sequences/scene_0/scene_0/00074.png all_sequences/scene_0/scene_0/00075.png all_sequences/scene_0/scene_0/00076.png all_sequences/scene_0/scene_0/00077.png all_sequences/scene_0/scene_0/00078.png all_sequences/scene_0/scene_0/00079.png all_sequences/scene_0/scene_0/00080.png all_sequences/scene_0/scene_0/00081.png all_sequences/scene_0/scene_0/00082.png all_sequences/scene_0/scene_0/00083.png all_sequences/scene_0/scene_0/00084.png all_sequences/scene_0/scene_0/00085.png all_sequences/scene_0/scene_0/00086.png all_sequences/scene_0/scene_0/00087.png all_sequences/scene_0/scene_0/00088.png all_sequences/scene_0/scene_0/00089.png all_sequences/scene_0/scene_0/00090.png all_sequences/scene_0/scene_0/00091.png all_sequences/scene_0/scene_0/00092.png all_sequences/scene_0/scene_0/00093.png all_sequences/scene_0/scene_0/00094.png all_sequences/scene_0/scene_0/00095.png all_sequences/scene_0/scene_0/00096.png all_sequences/scene_0/scene_0/00097.png all_sequences/scene_0/scene_0/00098.png Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1distributed_backend=nccl All distributed processes registered. Starting with 1 processes
You are using a CUDA device ('NVIDIA A10') that has Tensor Cores. To properly utilize them, you should set
torch.set_float32_matmul_precision('medium' | 'high')
which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] /data/anaconda3/envs/codef/lib/python3.10/site-packages/torch/utils/tensorboard/init.py:4: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. if not hasattr(tensorboard, "version") or LooseVersion( /data/anaconda3/envs/codef/lib/python3.10/site-packages/torch/utils/tensorboard/init.py:6: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. ) < LooseVersion("1.15"): Testing DataLoader 0: 1%|? | 1/99 [00:00<00:16, 5.84it/s]/data/anaconda3/envs/codef/lib/python3.10/site-packages/skvideo/io/ffmpeg.py:466: DeprecationWarning: tostring() is deprecated. Use tobytes() instead. self._proc.stdin.write(vid.tostring()) Traceback (most recent call last): File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/skvideo/io/ffmpeg.py", line 466, in writeFrame self._proc.stdin.write(vid.tostring()) BrokenPipeError: [Errno 32] Broken pipeDuring handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/data/CoDeF/train.py", line 557, in
main(hparams)
File "/data/CoDeF/train.py", line 550, in main
trainer.test(system, dataloaders=system.test_dataloader())
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 706, in test
return call._call_and_handle_interrupt(
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, kwargs)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 92, in launch
return function(*args, *kwargs)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 749, in _test_impl
results = self._run(model, ckpt_path=ckpt_path)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 935, in _run
results = self._run_stage()
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 971, in _run_stage
return self._evaluation_loop.run()
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/loops/utilities.py", line 177, in _decorator
return loop_run(self, args, kwargs)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 115, in run
self._evaluation_step(batch, batch_idx, dataloader_idx)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 375, in _evaluation_step
output = call._call_strategy_hook(trainer, hook_name, step_kwargs.values())
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 288, in _call_strategy_hook
output = fn(args, *kwargs)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 348, in test_step
return self.model.test_step(args, **kwargs)
File "/data/CoDeF/train.py", line 489, in test_step
self.video_visualizer.add(img)
File "/data/CoDeF/utils/video_visualizer.py", line 95, in add
self.video.writeFrame(frame)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/skvideo/io/ffmpeg.py", line 471, in writeFrame
raise IOError(msg)
OSError: [Errno 32] Broken pipe
FFMPEG COMMAND: /usr/local/bin/ffmpeg -y -f rawvideo -pix_fmt rgb24 -s 540x960 -i - -r 30.00 -s 540x960 -vcodec libx264 -crf 1 -pix_fmt yuv420p /data/CoDeF/results/all_sequences/scene_0/base/scene_0_base.mp4
FFMPEG STDERR OUTPUT:
Testing DataLoader 0: 1%| | 1/99 [00:00<00:42, 2.31it/s]
太难了,全是报错,已经装一周了还没装好