ERROR: Could not build wheels for xformers, which is required to install pyproject.toml-based projects

[zha11037]

Thanks for your work!

There are some questions when I tried training the model.

When I installed xformers in third_party by using "pip install ."or"pip install -vvv .", I met this error： However, by using this command, I can run the demo successfully:

python demo/run.py \
    resume_from_checkpoint=pretrained/SDv1.5mv-rawbox_2023-09-07_18-39_224x400 \
    ++runner.enable_xformers_memory_efficient_attention=false

Next step, I tried training the model without xformers, I used command as folllows:

accelerate launch  --mixed_precision fp16 --gpu_ids all --num_processes 8 tools/train.py ++runner.enable_xformers_memory_efficient_attention=false\
  +exp=224x400

But then I met the error that: ValueError: Using torch.compile requires PyTorch 2.0 or higher. Tracback:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /data/nfs/zhongda-shuhe/wzh/zsy/MagicDrive/tools/train.py:114 in <module>                        │
│                                                                                                  │
│   111                                                                                            │
│   112                                                                                            │
│   113 if __name__ == "__main__":                                                                 │
│ ❱ 114 │   main()                                                                                 │
│   115                                                                                            │
│                                                                                                  │
│ /opt/conda/envs/zmagic2/lib/python3.8/site-packages/hydra/main.py:90 in decorated_main           │
│                                                                                                  │
│    87 │   │   │   │   else:                                                                      │
│    88 │   │   │   │   │   # no return value from run_hydra() as it may sometime actually run t   │
│    89 │   │   │   │   │   # multiple times (--multirun)                                          │
│ ❱  90 │   │   │   │   │   _run_hydra(                                                            │
│    91 │   │   │   │   │   │   args=args,                                                         │
│    92 │   │   │   │   │   │   args_parser=args_parser,                                           │
│    93 │   │   │   │   │   │   task_function=task_function,                                       │
│                                                                                                  │
│ /opt/conda/envs/zmagic2/lib/python3.8/site-packages/hydra/_internal/utils.py:394 in _run_hydra   │
│                                                                                                  │
│   391 │   │                                                                                      │
│   392 │   │   if args.run or args.multirun:                                                      │
│   393 │   │   │   run_mode = hydra.get_mode(config_name=config_name, overrides=overrides)        │
│ ❱ 394 │   │   │   _run_app(                                                                      │
│   395 │   │   │   │   run=args.run,                                                              │
│   396 │   │   │   │   multirun=args.multirun,                                                    │
│   397 │   │   │   │   mode=run_mode,                                                             │
│                                                                                                  │
│ /opt/conda/envs/zmagic2/lib/python3.8/site-packages/hydra/_internal/utils.py:457 in _run_app     │
│                                                                                                  │
│   454 │   │   │   overrides.extend(["hydra.mode=MULTIRUN"])                                      │
│   455 │                                                                                          │
│   456 │   if mode == RunMode.RUN:                                                                │
│ ❱ 457 │   │   run_and_report(                                                                    │
│   458 │   │   │   lambda: hydra.run(                                                             │
│   459 │   │   │   │   config_name=config_name,                                                   │
│   460 │   │   │   │   task_function=task_function,                                               │
│                                                                                                  │
│ /opt/conda/envs/zmagic2/lib/python3.8/site-packages/hydra/_internal/utils.py:222 in              │
│ run_and_report                                                                                   │
│                                                                                                  │
│   219 │   │   return func()                                                                      │
│   220 │   except Exception as ex:                                                                │
│   221 │   │   if _is_env_set("HYDRA_FULL_ERROR") or is_under_debugger():                         │
│ ❱ 222 │   │   │   raise ex                                                                       │
│   223 │   │   else:                                                                              │
│   224 │   │   │   try:                                                                           │
│   225 │   │   │   │   if isinstance(ex, CompactHydraException):                                  │
│                                                                                                  │
│ /opt/conda/envs/zmagic2/lib/python3.8/site-packages/hydra/_internal/utils.py:219 in              │
│ run_and_report                                                                                   │
│                                                                                                  │
│   216                                                                                            │
│   217 def run_and_report(func: Any) -> Any:                                                      │
│   218 │   try:                                                                                   │
│ ❱ 219 │   │   return func()                                                                      │
│   220 │   except Exception as ex:                                                                │
│   221 │   │   if _is_env_set("HYDRA_FULL_ERROR") or is_under_debugger():                         │
│   222 │   │   │   raise ex                                                                       │
│                                                                                                  │
│ /opt/conda/envs/zmagic2/lib/python3.8/site-packages/hydra/_internal/utils.py:458 in <lambda>     │
│                                                                                                  │
│   455 │                                                                                          │
│   456 │   if mode == RunMode.RUN:                                                                │
│   457 │   │   run_and_report(                                                                    │
│ ❱ 458 │   │   │   lambda: hydra.run(                                                             │
│   459 │   │   │   │   config_name=config_name,                                                   │
│   460 │   │   │   │   task_function=task_function,                                               │
│   461 │   │   │   │   overrides=overrides,                                                       │
│                                                                                                  │
│ /opt/conda/envs/zmagic2/lib/python3.8/site-packages/hydra/_internal/hydra.py:132 in run          │
│                                                                                                  │
│   129 │   │   callbacks.on_run_end(config=cfg, config_name=config_name, job_return=ret)          │
│   130 │   │                                                                                      │
│   131 │   │   # access the result to trigger an exception in case the job failed.                │
│ ❱ 132 │   │   _ = ret.return_value                                                               │
│   133 │   │                                                                                      │
│   134 │   │   return ret                                                                         │
│   135                                                                                            │
│                                                                                                  │
│ /opt/conda/envs/zmagic2/lib/python3.8/site-packages/hydra/core/utils.py:260 in return_value      │
│                                                                                                  │
│   257 │   │   │   sys.stderr.write(                                                              │
│   258 │   │   │   │   f"Error executing job with overrides: {self.overrides}" + os.linesep       │
│   259 │   │   │   )                                                                              │
│ ❱ 260 │   │   │   raise self._return_value                                                       │
│   261 │                                                                                          │
│   262 │   @return_value.setter                                                                   │
│   263 │   def return_value(self, value: Any) -> None:                                            │
│                                                                                                  │
│ /opt/conda/envs/zmagic2/lib/python3.8/site-packages/hydra/core/utils.py:186 in run_job           │
│                                                                                                  │
│   183 │   │   with env_override(hydra_cfg.hydra.job.env_set):                                    │
│   184 │   │   │   callbacks.on_job_start(config=config, task_function=task_function)             │
│   185 │   │   │   try:                                                                           │
│ ❱ 186 │   │   │   │   ret.return_value = task_function(task_cfg)                                 │
│   187 │   │   │   │   ret.status = JobStatus.COMPLETED                                           │
│   188 │   │   │   except Exception as e:                                                         │
│   189 │   │   │   │   ret.return_value = e                                                       │
│                                                                                                  │
│ /data/nfs/zhongda-shuhe/wzh/zsy/MagicDrive/tools/train.py:95 in main                             │
│                                                                                                  │
│    92 │   runner_cls = load_module(cfg.model.runner_module)                                      │
│    93 │   runner = runner_cls(cfg, accelerator, train_dataset, val_dataset)                      │
│    94 │   runner.set_optimizer_scheduler()                                                       │
│ ❱  95 │   runner.prepare_device()                                                                │
│    96 │                                                                                          │
│    97 │   # tracker                                                                              │
│    98 │   logging.debug("Current config:\n" + OmegaConf.to_yaml(cfg, resolve=True))              │
│                                                                                                  │
│ /data/nfs/zhongda-shuhe/wzh/zsy/MagicDrive/./magicdrive/runner/multiview_runner.py:183 in        │
│ prepare_device                                                                                   │
│                                                                                                  │
│   180 │   │   │   self.train_dataloader,                                                         │
│   181 │   │   │   self.lr_scheduler,                                                             │
│   182 │   │   )                                                                                  │
│ ❱ 183 │   │   ddp_modules = self.accelerator.prepare(*ddp_modules)                               │
│   184 │   │   (                                                                                  │
│   185 │   │   │   self.controlnet_unet,                                                          │
│   186 │   │   │   self.optimizer,                                                                │
│                                                                                                  │
│ /opt/conda/envs/zmagic2/lib/python3.8/site-packages/accelerate/accelerator.py:1182 in prepare    │
│                                                                                                  │
│   1179 │   │   elif self.distributed_type == DistributedType.MEGATRON_LM:                        │
│   1180 │   │   │   result = self._prepare_megatron_lm(*args)                                     │
│   1181 │   │   else:                                                                             │
│ ❱ 1182 │   │   │   result = tuple(                                                               │
│   1183 │   │   │   │   self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d i  │
│   1184 │   │   │   )                                                                             │
│   1185 │   │   │   result = tuple(self._prepare_one(obj, device_placement=d) for obj, d in zip(  │
│                                                                                                  │
│ /opt/conda/envs/zmagic2/lib/python3.8/site-packages/accelerate/accelerator.py:1183 in <genexpr>  │
│                                                                                                  │
│   1180 │   │   │   result = self._prepare_megatron_lm(*args)                                     │
│   1181 │   │   else:                                                                             │
│   1182 │   │   │   result = tuple(                                                               │
│ ❱ 1183 │   │   │   │   self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d i  │
│   1184 │   │   │   )                                                                             │
│   1185 │   │   │   result = tuple(self._prepare_one(obj, device_placement=d) for obj, d in zip(  │
│   1186                                                                                           │
│                                                                                                  │
│ /opt/conda/envs/zmagic2/lib/python3.8/site-packages/accelerate/accelerator.py:1022 in            │
│ _prepare_one                                                                                     │
│                                                                                                  │
│   1019 │   │   │   if isinstance(obj, torch.utils.data.DataLoader):                              │
│   1020 │   │   │   │   return self.prepare_data_loader(obj, device_placement=device_placement)   │
│   1021 │   │   │   elif isinstance(obj, torch.nn.Module):                                        │
│ ❱ 1022 │   │   │   │   return self.prepare_model(obj, device_placement=device_placement)         │
│   1023 │   │   │   elif isinstance(obj, torch.optim.Optimizer):                                  │
│   1024 │   │   │   │   optimizer = self.prepare_optimizer(obj, device_placement=device_placemen  │
│   1025 │   │   │   │   return optimizer                                                          │
│                                                                                                  │
│ /opt/conda/envs/zmagic2/lib/python3.8/site-packages/accelerate/accelerator.py:1344 in            │
│ prepare_model                                                                                    │
│                                                                                                  │
│   1341 │   │   # torch.compile should be called last.                                            │
│   1342 │   │   if self.state.dynamo_plugin.backend != DynamoBackend.NO:                          │
│   1343 │   │   │   if not is_torch_version(">=", "2.0"):                                         │
│ ❱ 1344 │   │   │   │   raise ValueError("Using `torch.compile` requires PyTorch 2.0 or higher."  │
│   1345 │   │   │   model = torch.compile(model, **self.state.dynamo_plugin.to_kwargs())          │
│   1346 │   │   return model                                                                      │
│   1347                                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Using `torch.compile` requires PyTorch 2.0 or higher.

Did I run a wrong command? Or how can I fix the qestion about instaling xformers?

[flymin]

Did you config the accelerate? You may enable some unsupported function in your configuration file.

For xformers, could you check the results of this function? I do not understand why it returns “-.d20240803”, even without the leading “+”.

[zha11037]

Thanks for your quick rely! I 've sovled the question by accelerating my config and redownloading xformers!