Colab Notebook for training a custom model

[niazipan]

I've been trying to set up a Google Colab notebook to train on a custom dataset. Custom notebook here: https://colab.research.google.com/drive/1CqbxgNHUY56j96yC3flyCuSNCkR5XWPl (although you'd need your own custom data to test)

I get the following error when running the main.py script (last cell of the notebook):

Global seed set to 23
Running on GPUs 0,
Working with z of shape (1, 256, 16, 16) = 65536 dimensions.
loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth
VQLPIPSWithDiscriminator running with hinge loss.
/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loggers/test_tube.py:105: LightningDeprecationWarning: The TestTubeLogger is deprecated since v1.5 and will be removed in v1.7. We recommend switching to the `pytorch_lightning.loggers.TensorBoardLogger` as an alternative.
  "The TestTubeLogger is deprecated since v1.5 and will be removed in v1.7. We recommend switching to the"
/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/connectors/callback_connector.py:148: LightningDeprecationWarning: Setting `Trainer(checkpoint_callback=<pytorch_lightning.callbacks.model_checkpoint.ModelCheckpoint object at 0x7f2494cf7bd0>)` is deprecated in v1.5 and will be removed in v1.7. Please consider using `Trainer(enable_checkpointing=<pytorch_lightning.callbacks.model_checkpoint.ModelCheckpoint object at 0x7f2494cf7bd0>)`.
  f"Setting `Trainer(checkpoint_callback={checkpoint_callback})` is deprecated in v1.5 and will "
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
accumulate_grad_batches = 1
Setting learning rate to 2.25e-05 = 1 (accumulate_grad_batches) * 1 (num_gpus) * 5 (batchsize) * 4.50e-06 (base_lr)
/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/configuration_validator.py:292: LightningDeprecationWarning: Base `Callback.on_train_batch_end` hook signature has changed in v1.5. The `dataloader_idx` argument will be removed in v1.7.
  f"Base `Callback.{hook}` hook signature has changed in v1.5."
/usr/local/lib/python3.7/dist-packages/pytorch_lightning/core/datamodule.py:470: LightningDeprecationWarning: DataModule.setup has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.setup.
  f"DataModule.{name} has already been called, so it will not be called again. "
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Project config
Summoning checkpoint.
Traceback (most recent call last):
  File "main.py", line 565, in <module>
    trainer.fit(model, data)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 741, in fit
    self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 685, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 777, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1199, in _run
    self._dispatch()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1279, in _dispatch
    self.training_type_plugin.start_training(self)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 202, in start_training
    self._results = trainer.run_stage()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1289, in run_stage
    return self._run_train()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1306, in _run_train
    self._pre_training_routine()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1301, in _pre_training_routine
    self.call_hook("on_pretrain_routine_start")
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1495, in call_hook
    callback_fx(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/callback_hook.py", line 148, in on_pretrain_routine_start
    callback.on_pretrain_routine_start(self, self.lightning_module)
  File "/content/drive/MyDrive/colab-taming-transformers/taming-transformers/main.py", line 197, in on_pretrain_routine_start
    print(self.config.pretty())
  File "/usr/local/lib/python3.7/dist-packages/omegaconf/dictconfig.py", line 354, in __getattr__
    key=key, value=None, cause=e, type_override=ConfigAttributeError
  File "/usr/local/lib/python3.7/dist-packages/omegaconf/base.py", line 196, in _format_and_raise
    type_override=type_override,
  File "/usr/local/lib/python3.7/dist-packages/omegaconf/_utils.py", line 821, in format_and_raise
    _raise(ex, cause)
  File "/usr/local/lib/python3.7/dist-packages/omegaconf/_utils.py", line 719, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set end OC_CAUSE=1 for full backtrace
  File "/usr/local/lib/python3.7/dist-packages/omegaconf/dictconfig.py", line 351, in __getattr__
    return self._get_impl(key=key, default_value=_DEFAULT_MARKER_)
  File "/usr/local/lib/python3.7/dist-packages/omegaconf/dictconfig.py", line 438, in _get_impl
    node = self._get_node(key=key, throw_on_missing_key=True)
  File "/usr/local/lib/python3.7/dist-packages/omegaconf/dictconfig.py", line 470, in _get_node
    raise ConfigKeyError(f"Missing key {key}")
omegaconf.errors.ConfigAttributeError: Missing key pretty
    full_key: setup_callback.params.config.pretty
    object_type=dict

I've already had to install dependancies manually becuase of the issues with conda on Google Colab.

I've also removed references to lightning_config.logger in the main.py script, as per the fix here: https://github.com/CompVis/taming-transformers/issues/72#issuecomment-875757912

Any ideas how to get it up and running?

Thanks

m

[wolframalexa]

This happened to me too, and I was able to fix it by commenting out the lines that used "pretty". Not the most elegant solution, but it worked.

[Zhtuzki]

What are inputs and outputs in your procedure