ImportError: cannot import name 'split_torch_state_dict_into_shards' from 'huggingface_hub'

[ylchan87]

Problem

spinning up docker with commit 7fe9562 (latest commit as of 8Jul2024) fails with:

Traceback (most recent call last):
  File "/workspace/instantmesh/app.py", line 12, in <module>
    from diffusers import DiffusionPipeline, EulerAncestralDiscreteScheduler
  File "/workspace/miniconda3/envs/instantmesh/lib/python3.10/site-packages/diffusers/__init__.py", line 3, in <module>
    from .configuration_utils import ConfigMixin
  File "/workspace/miniconda3/envs/instantmesh/lib/python3.10/site-packages/diffusers/configuration_utils.py", line 34, in <module>
    from .utils import (
  File "/workspace/miniconda3/envs/instantmesh/lib/python3.10/site-packages/diffusers/utils/__init__.py", line 21, in <module>
    from .accelerate_utils import apply_forward_hook
  File "/workspace/miniconda3/envs/instantmesh/lib/python3.10/site-packages/diffusers/utils/accelerate_utils.py", line 24, in <module>
    import accelerate
  File "/workspace/miniconda3/envs/instantmesh/lib/python3.10/site-packages/accelerate/__init__.py", line 16, in <module>
    from .accelerator import Accelerator
  File "/workspace/miniconda3/envs/instantmesh/lib/python3.10/site-packages/accelerate/accelerator.py", line 34, in <module>
    from huggingface_hub import split_torch_state_dict_into_shards
ImportError: cannot import name 'split_torch_state_dict_into_shards' from 'huggingface_hub' (/workspace/miniconda3/envs/instantmesh/lib/python3.10/site-packages/huggingface_hub/__init__.py)

Quick fix

lock package accelarate to 0.31.0 in requirements.txt

i.e.

$ git diff requirements.txt
diff --git a/requirements.txt b/requirements.txt
index aa68bbd..7b606cd 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -5,7 +5,7 @@ einops
 omegaconf
 torchmetrics
 webdataset
-accelerate
+accelerate==0.31.0
 tensorboard
 PyMCubes

[Utkarsh4430]

The error you're encountering is due to the split_torch_state_dict_into_shards function not being available in huggingface-hub version 0.20.3. This function is included starting from version 0.23.0.

To resolve this issue, update the huggingface-hub library to version 0.23.0 or later: