moduleNotFoundError: No module named 'distutils.util

[dho799]

Getting the following error when training w/ runpod.io using Joepenna dreambooth Jupyter notebook:

Global seed set to 23 Running on GPUs 0, Loading model from model.ckpt LatentDiffusion: Running in eps-prediction mode DiffusionWrapper has 859.52 M params. making attention of type 'vanilla' with 512 in_channels Working with z of shape (1, 4, 64, 64) = 16384 dimensions. making attention of type 'vanilla' with 512 in_channels Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/transformers/utils/import_utils.py", line 1110, in _get_module return importlib.import_module("." + module_name, self.name) File "/usr/lib/python3.10/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "/usr/local/lib/python3.10/dist-packages/transformers/models/clip/modeling_clip.py", line 27, in from ...modeling_utils import PreTrainedModel File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 83, in from accelerate import version as accelerate_version File "/usr/local/lib/python3.10/dist-packages/accelerate/init.py", line 7, in from .accelerator import Accelerator File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 29, in from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state File "/usr/local/lib/python3.10/dist-packages/accelerate/checkpointing.py", line 24, in from .utils import ( File "/usr/local/lib/python3.10/dist-packages/accelerate/utils/init.py", line 6, in from .dataclasses import ( File "/usr/local/lib/python3.10/dist-packages/accelerate/utils/dataclasses.py", line 29, in from distutils.util import strtobool ModuleNotFoundError: No module named 'distutils.util'

Global seed set to 23 Running on GPUs 0, Loading model from model.ckpt LatentDiffusion: Running in eps-prediction mode DiffusionWrapper has 859.52 M params. making attention of type 'vanilla' with 512 in_channels Working with z of shape (1, 4, 64, 64) = 16384 dimensions. making attention of type 'vanilla' with 512 in_channels Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/transformers/utils/import_utils.py", line 1110, in _get_module return importlib.import_module("." + module_name, self.name) File "/usr/lib/python3.10/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "/usr/local/lib/python3.10/dist-packages/transformers/models/clip/modeling_clip.py", line 27, in from ...modeling_utils import PreTrainedModel File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 83, in from accelerate import version as accelerate_version File "/usr/local/lib/python3.10/dist-packages/accelerate/init.py", line 7, in from .accelerator import Accelerator File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 29, in from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state File "/usr/local/lib/python3.10/dist-packages/accelerate/checkpointing.py", line 24, in from .utils import ( File "/usr/local/lib/python3.10/dist-packages/accelerate/utils/init.py", line 6, in from .dataclasses import ( File "/usr/local/lib/python3.10/dist-packages/accelerate/utils/dataclasses.py", line 29, in from distutils.util import strtobool ModuleNotFoundError: No module named 'distutils.util'

[dho799]

Anyone know how to resolve this? Been trying for days to fix with no success

[SkylarkingMonkey]

It's because of changes that RunPod has made. If you look at your pod it probably says runpod/pytorch:3.10-1.13.1-116 in upper left of the pod cell. click on the 3 horizontal lines and select the 'edit pod' option. Then in the docker name where it says runpod/pytorch:3.10-1.13.1-116, delete the numbers so it just says runpod/pytorch, save, and then restart your pod and reinstall all the environment requirements in the Jupiter notebook (second cell). All should work after that.