Closed chakri-rvk closed 1 year ago
What happens if you set datamodule.ims_per_batch=2
? ims_per_batch
is the global batch size irrespective of the number of devices. This value is divided by trainer.devices
to get the number of images on each device.
Also, can you put a code diff somewhere or, ideally, push your branch somewhere?
Here is the git diff output
I just changed the COCO_TorchvisionFasterRCNN_Adv.yaml file and the changed the model from _fasterrcnn_resnet50fpn to _fasterrcnn_resnet50_fpnv2
I ran the command with datamodule.ims_per_batch=2 as you suggested and I encounter the same problem - the progress bar just freezes.
However, if the change the model back to _fasterrcnn_resnet50fpn, I see the iterations progress. However, after 50 batches, the adversarial training does not occur - the progress bar continues but the gain goes to nan (but this time, the progress bar does not freeze, and the I can terminate the run using Ctrl+C). However, with the same learning rate parameter and using single GPU, I do not encounter such issues (even after 300 batches).
Can you run even run this on 1 GPU without DDP? Can you run it on 1 GPU with DDP?
I was able to reproduce this issue locally.
@mzweilin: I'm not sure what is going on. I don't see this issue on #103. Also running CIFAR10_CNN_Adv
with DDP on 2 GPUs fails:
$ CUDA_VISIBLE_DEVICES=2,3 python -m mart experiment=CIFAR10_CNN_Adv trainer=ddp fit=true trainer.devices=2 datamodule.world_size=2
...
Traceback (most recent call last):
File "/home/ccorneli/Projects/MART/mart/__main__.py", line 56, in main
metric_dict, _ = lightning(cfg)
File "/home/ccorneli/Projects/MART/mart/utils/utils.py", line 58, in wrap
raise ex
File "/home/ccorneli/Projects/MART/mart/utils/utils.py", line 55, in wrap
metric_dict, object_dict = task_func(cfg=cfg)
File "/home/ccorneli/Projects/MART/mart/tasks/lightning.py", line 75, in lightning
trainer.fit(model=model, datamodule=datamodule, ckpt_path=ckpt_path)
File "/home/ccorneli/Projects/MART/venv/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 770, in fit
self._call_and_handle_interrupt(
File "/home/ccorneli/Projects/MART/venv/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 723, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/ccorneli/Projects/MART/venv/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 811, in _fit_impl
results = self._run(model, ckpt_path=self.ckpt_path)
File "/home/ccorneli/Projects/MART/venv/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1217, in _run
self.strategy.setup(self)
File "/home/ccorneli/Projects/MART/venv/lib/python3.9/site-packages/pytorch_lightning/strategies/ddp.py", line 176, in setup
self.configure_ddp()
File "/home/ccorneli/Projects/MART/venv/lib/python3.9/site-packages/pytorch_lightning/strategies/ddp.py", line 299, in configure_ddp
self.model = self._setup_model(LightningDistributedModule(self.model))
File "/home/ccorneli/Projects/MART/venv/lib/python3.9/site-packages/pytorch_lightning/strategies/ddp.py", line 192, in _setup_model
return DistributedDataParallel(module=model, device_ids=device_ids, **self._ddp_kwargs)
File "/home/ccorneli/Projects/MART/venv/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 657, in __init__
_sync_module_states(
File "/home/ccorneli/Projects/MART/venv/lib/python3.9/site-packages/torch/distributed/utils.py", line 134, in _sync_module_states
module_states.append(buffer.detach())
File "/home/ccorneli/Projects/MART/venv/lib/python3.9/site-packages/torch/nn/parameter.py", line 144, in __torch_function__
raise ValueError( ValueError: Attempted to use an uninitialized parameter in <method 'detach' of 'torch._C._TensorBase' objects>. This error happens when you are using a `LazyModule` or explicitly manipulating `torch.nn.parameter.UninitializedBuffer` objects. When using LazyModules Call `forward` with a dummy batch to
initialize the parameters before calling torch functions
Can you run even run this on 1 GPU without DDP? Can you run it on 1 GPU with DDP?
Yes. I can run both the combinations you suggested (I continued the runs for over 100 batches and then terminated). Here are the commands
CUDA_VISIBLE_DEVICES=1 python -m mart experiment=COCO_TorchvisionFasterRCNN_Adv task_name=COCO_TorchvisionFasterRCNNv2_AdvTrain_Trial1 trainer=gpu fit=true
CUDA_VISIBLE_DEVICES=1 python -m mart experiment=COCO_TorchvisionFasterRCNN_Adv task_name=COCO_TorchvisionFasterRCNNv2_AdvTrain_Trial1 trainer=gpu fit=true trainer=ddp trainer.devices=1
I also tried adding datamodule.ims_per_batch=4
and the following command works.
CUDA_VISIBLE_DEVICES=1 python -m mart experiment=COCO_TorchvisionFasterRCNN_Adv task_name=COCO_TorchvisionFasterRCNNv2_AdvTrain_Trial1 trainer=gpu fit=true datamodule.ims_per_batch=4 trainer=ddp trainer.devices=1
And for all these runs, the code base is same states in my previous post. PFA the diff status for reference (I re-ran the command)
The moment I increase to GPUs, it falls apart. @dxoigmn Thank you for testing and reproducing the error on CIFAR10
I think if you delete these lines: https://github.com/IntelLabs/MART/blob/eced15bdad18b6683190997590e2500a332b03e7/mart/attack/perturber/perturber.py#L51 https://github.com/IntelLabs/MART/blob/eced15bdad18b6683190997590e2500a332b03e7/mart/attack/perturber/perturber.py#L70
Everything should work correctly again.
@dxoigmn I commented the 2 lines you pointed, but I ran into AttributeError: 'Perturber' object has no attribute 'perturbation'.
I ran the following command
CUDA_VISIBLE_DEVICES=1,2 python -m mart experiment=COCO_TorchvisionFasterRCNN_Adv task_name=COCO_TorchvisionFasterRCNNv2_AdvTrain_Trial1 trainer=gpu fit=true trainer=ddp trainer.devices=2
Find the log below
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [1,2]
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [1,2]
| Name | Type | Params
--------------------------------------------------------
0 | model | SequentialDict | 43.7 M
1 | training_metrics | MAP | 0
2 | validation_metrics | MAP | 0
3 | test_metrics | MetricCollection | 0
--------------------------------------------------------
43.5 M Trainable params
225 K Non-trainable params
43.7 M Total params
174.849 Total estimated model params size (MB)
Sanity Checking: 0it [00:00, ?it/s]/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:240: PossibleUserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 256 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
rank_zero_warn(
/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:240: PossibleUserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 256 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
rank_zero_warn(
Epoch 0: 0%| | 0/15411 [00:00<?, ?it/s][2023-03-30 20:04:56,116][mart.utils.utils][ERROR] -
Traceback (most recent call last):
File "/raid/vravilla3/MART_v2/MART/mart/utils/utils.py", line 55, in wrap
metric_dict, object_dict = task_func(cfg=cfg)
File "/raid/vravilla3/MART_v2/MART/mart/tasks/lightning.py", line 75, in lightning
trainer.fit(model=model, datamodule=datamodule, ckpt_path=ckpt_path)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 770, in fit
self._call_and_handle_interrupt(
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt
return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 93, in launch
return function(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 811, in _fit_impl
results = self._run(model, ckpt_path=self.ckpt_path)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1236, in _run
results = self._run_stage()
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1323, in _run_stage
return self._run_train()
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1353, in _run_train
self.fit_loop.run()
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 266, in advance
self._outputs = self.epoch_loop.run(self._data_fetcher)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 208, in advance
batch_output = self.batch_loop.run(batch, batch_idx)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 88, in advance
outputs = self.optimizer_loop.run(split_batch, optimizers, batch_idx)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 203, in advance
result = self._run_optimization(
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 256, in _run_optimization
self._optimizer_step(optimizer, opt_idx, batch_idx, closure)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 369, in _optimizer_step
self.trainer._call_lightning_module_hook(
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1595, in _call_lightning_module_hook
output = fn(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/core/lightning.py", line 1646, in optimizer_step
optimizer.step(closure=optimizer_closure)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/core/optimizer.py", line 168, in step
step_output = self._strategy.optimizer_step(self._optimizer, self._optimizer_idx, closure, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/strategies/ddp.py", line 286, in optimizer_step
optimizer_output = super().optimizer_step(optimizer, opt_idx, closure, model, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/strategies/strategy.py", line 193, in optimizer_step
return self.precision_plugin.optimizer_step(model, optimizer, opt_idx, closure, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 155, in optimizer_step
return optimizer.step(closure=closure, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/optim/lr_scheduler.py", line 65, in wrapper
return wrapped(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/optim/optimizer.py", line 113, in wrapper
return func(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/optim/sgd.py", line 125, in step
loss = closure()
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 140, in _wrap_closure
closure_result = closure()
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 148, in __call__
self._result = self.closure(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 134, in closure
step_output = self._step_fn()
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 427, in _training_step
training_step_output = self.trainer._call_strategy_hook("training_step", *step_kwargs.values())
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1765, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/strategies/ddp.py", line 349, in training_step
return self.model(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1008, in forward
output = self._run_ddp_forward(*inputs, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 969, in _run_ddp_forward
return module_to_run(*inputs[0], **kwargs[0])
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/overrides/base.py", line 82, in forward
output = self.module.training_step(*inputs, **kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/models/modular.py", line 103, in training_step
output = self(input=input, target=target, model=self.model, step="training")
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/models/modular.py", line 95, in forward
return self.model(**kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/nn/nn.py", line 115, in forward
output = module(step=step, sequence=sequence, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/attack/adversary.py", line 333, in forward
self.attacker.fit(input=input, target=target, model=model, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 12, in decorate_autocast
return func(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 12, in decorate_autocast
return func(*args, **kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/attack/adversary.py", line 191, in fit
self.on_run_start(adversary=self, input=input, target=target, model=model, **kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/attack/adversary.py", line 151, in on_run_start
super().on_run_start(
File "/raid/vravilla3/MART_v2/MART/mart/attack/adversary.py", line 34, in on_run_start
callback.on_run_start(**kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/attack/perturber/batch.py", line 66, in on_run_start
perturber.on_run_start(
File "/raid/vravilla3/MART_v2/MART/mart/attack/perturber/perturber.py", line 74, in on_run_start
self.perturbation.register_hook(self.gradient_modifier)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1207, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'Perturber' object has no attribute 'perturbation'
[2023-03-30 20:04:56,118][mart.utils.utils][INFO] - Closing loggers...
Error executing job with overrides: ['experiment=COCO_TorchvisionFasterRCNN_Adv', 'task_name=COCO_TorchvisionFasterRCNNv2_AdvTrain_Trial1', 'trainer=gpu', 'fit=true', 'trainer=ddp', 'trainer.devices=2', 'datamodule.ims_per_batch=4']
Error executing job with overrides: ['experiment=COCO_TorchvisionFasterRCNN_Adv', 'task_name=COCO_TorchvisionFasterRCNNv2_AdvTrain_Trial1', 'trainer=gpu', 'fit=true', 'trainer=ddp', 'trainer.devices=2', 'datamodule.ims_per_batch=4']
Traceback (most recent call last):
File "/raid/vravilla3/MART_v2/MART/mart/__main__.py", line 56, in main
metric_dict, _ = lightning(cfg)
File "/raid/vravilla3/MART_v2/MART/mart/utils/utils.py", line 58, in wrap
raise ex
File "/raid/vravilla3/MART_v2/MART/mart/utils/utils.py", line 55, in wrap
metric_dict, object_dict = task_func(cfg=cfg)
File "/raid/vravilla3/MART_v2/MART/mart/tasks/lightning.py", line 75, in lightning
trainer.fit(model=model, datamodule=datamodule, ckpt_path=ckpt_path)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 770, in fit
self._call_and_handle_interrupt(
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 723, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 811, in _fit_impl
results = self._run(model, ckpt_path=self.ckpt_path)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1236, in _run
results = self._run_stage()
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1323, in _run_stage
return self._run_train()
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1353, in _run_train
self.fit_loop.run()
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 266, in advance
self._outputs = self.epoch_loop.run(self._data_fetcher)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 208, in advance
batch_output = self.batch_loop.run(batch, batch_idx)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 88, in advance
outputs = self.optimizer_loop.run(split_batch, optimizers, batch_idx)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 203, in advance
result = self._run_optimization(
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 256, in _run_optimization
self._optimizer_step(optimizer, opt_idx, batch_idx, closure)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 369, in _optimizer_step
self.trainer._call_lightning_module_hook(
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1595, in _call_lightning_module_hook
output = fn(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/core/lightning.py", line 1646, in optimizer_step
optimizer.step(closure=optimizer_closure)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/core/optimizer.py", line 168, in step
step_output = self._strategy.optimizer_step(self._optimizer, self._optimizer_idx, closure, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/strategies/ddp.py", line 286, in optimizer_step
optimizer_output = super().optimizer_step(optimizer, opt_idx, closure, model, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/strategies/strategy.py", line 193, in optimizer_step
return self.precision_plugin.optimizer_step(model, optimizer, opt_idx, closure, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 155, in optimizer_step
return optimizer.step(closure=closure, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/optim/lr_scheduler.py", line 65, in wrapper
return wrapped(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/optim/optimizer.py", line 113, in wrapper
return func(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/optim/sgd.py", line 125, in step
loss = closure()
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 140, in _wrap_closure
closure_result = closure()
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 148, in __call__
self._result = self.closure(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 134, in closure
step_output = self._step_fn()
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 427, in _training_step
training_step_output = self.trainer._call_strategy_hook("training_step", *step_kwargs.values())
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1765, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/strategies/ddp.py", line 349, in training_step
return self.model(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1008, in forward
output = self._run_ddp_forward(*inputs, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 969, in _run_ddp_forward
return module_to_run(*inputs[0], **kwargs[0])
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/overrides/base.py", line 82, in forward
output = self.module.training_step(*inputs, **kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/models/modular.py", line 103, in training_step
output = self(input=input, target=target, model=self.model, step="training")
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/models/modular.py", line 95, in forward
return self.model(**kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/nn/nn.py", line 115, in forward
output = module(step=step, sequence=sequence, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/attack/adversary.py", line 333, in forward
self.attacker.fit(input=input, target=target, model=model, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 12, in decorate_autocast
return func(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 12, in decorate_autocast
return func(*args, **kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/attack/adversary.py", line 191, in fit
self.on_run_start(adversary=self, input=input, target=target, model=model, **kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/attack/adversary.py", line 151, in on_run_start
super().on_run_start(
File "/raid/vravilla3/MART_v2/MART/mart/attack/adversary.py", line 34, in on_run_start
callback.on_run_start(**kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/attack/perturber/batch.py", line 66, in on_run_start
perturber.on_run_start(
File "/raid/vravilla3/MART_v2/MART/mart/attack/perturber/perturber.py", line 74, in on_run_start
self.perturbation.register_hook(self.gradient_modifier)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1207, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'Perturber' object has no attribute 'perturbation'
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Traceback (most recent call last):
File "/raid/vravilla3/MART_v2/MART/mart/__main__.py", line 56, in main
metric_dict, _ = lightning(cfg)
File "/raid/vravilla3/MART_v2/MART/mart/utils/utils.py", line 58, in wrap
raise ex
File "/raid/vravilla3/MART_v2/MART/mart/utils/utils.py", line 55, in wrap
metric_dict, object_dict = task_func(cfg=cfg)
File "/raid/vravilla3/MART_v2/MART/mart/tasks/lightning.py", line 75, in lightning
trainer.fit(model=model, datamodule=datamodule, ckpt_path=ckpt_path)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 770, in fit
self._call_and_handle_interrupt(
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt
return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 93, in launch
return function(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 811, in _fit_impl
results = self._run(model, ckpt_path=self.ckpt_path)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1236, in _run
results = self._run_stage()
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1323, in _run_stage
return self._run_train()
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1353, in _run_train
self.fit_loop.run()
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 266, in advance
self._outputs = self.epoch_loop.run(self._data_fetcher)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 208, in advance
batch_output = self.batch_loop.run(batch, batch_idx)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 88, in advance
outputs = self.optimizer_loop.run(split_batch, optimizers, batch_idx)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 203, in advance
result = self._run_optimization(
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 256, in _run_optimization
self._optimizer_step(optimizer, opt_idx, batch_idx, closure)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 369, in _optimizer_step
self.trainer._call_lightning_module_hook(
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1595, in _call_lightning_module_hook
output = fn(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/core/lightning.py", line 1646, in optimizer_step
optimizer.step(closure=optimizer_closure)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/core/optimizer.py", line 168, in step
step_output = self._strategy.optimizer_step(self._optimizer, self._optimizer_idx, closure, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/strategies/ddp.py", line 286, in optimizer_step
optimizer_output = super().optimizer_step(optimizer, opt_idx, closure, model, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/strategies/strategy.py", line 193, in optimizer_step
return self.precision_plugin.optimizer_step(model, optimizer, opt_idx, closure, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 155, in optimizer_step
return optimizer.step(closure=closure, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/optim/lr_scheduler.py", line 65, in wrapper
return wrapped(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/optim/optimizer.py", line 113, in wrapper
return func(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/optim/sgd.py", line 125, in step
loss = closure()
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 140, in _wrap_closure
closure_result = closure()
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 148, in __call__
self._result = self.closure(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 134, in closure
step_output = self._step_fn()
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 427, in _training_step
training_step_output = self.trainer._call_strategy_hook("training_step", *step_kwargs.values())
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1765, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/strategies/ddp.py", line 349, in training_step
return self.model(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1008, in forward
output = self._run_ddp_forward(*inputs, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 969, in _run_ddp_forward
return module_to_run(*inputs[0], **kwargs[0])
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/pytorch_lightning/overrides/base.py", line 82, in forward
output = self.module.training_step(*inputs, **kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/models/modular.py", line 103, in training_step
output = self(input=input, target=target, model=self.model, step="training")
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/models/modular.py", line 95, in forward
return self.model(**kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/nn/nn.py", line 115, in forward
output = module(step=step, sequence=sequence, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/attack/adversary.py", line 333, in forward
self.attacker.fit(input=input, target=target, model=model, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 12, in decorate_autocast
return func(*args, **kwargs)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 12, in decorate_autocast
return func(*args, **kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/attack/adversary.py", line 191, in fit
self.on_run_start(adversary=self, input=input, target=target, model=model, **kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/attack/adversary.py", line 151, in on_run_start
super().on_run_start(
File "/raid/vravilla3/MART_v2/MART/mart/attack/adversary.py", line 34, in on_run_start
callback.on_run_start(**kwargs)
File "/raid/vravilla3/MART_v2/MART/mart/attack/perturber/batch.py", line 66, in on_run_start
perturber.on_run_start(
File "/raid/vravilla3/MART_v2/MART/mart/attack/perturber/perturber.py", line 74, in on_run_start
self.perturbation.register_hook(self.gradient_modifier)
File "/nethome/vravilla3/miniconda3/envs/gardv3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1207, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'Perturber' object has no attribute 'perturbation'
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Epoch 0: 0%| | 0/15411 [00:06<?, ?it/s]
Add self.perturbation = None to the init func
@dxoigmn I added self.perturbation = None. It gave AttributeError: 'NoneType' object has no attribute 'register_hook'
Here is the git diff
Then, I tried adding self.gradientmodifier = None to the init func. I then got this error **AttributeError: 'NoneType' object has no attribute 'fill'**
Here is the snippet of the error
For both these runs, I used this command
CUDA_VISIBLE_DEVICES=1,2 python -m mart experiment=COCO_TorchvisionFasterRCNN_Adv task_name=COCO_TorchvisionFasterRCNNv2_AdvTrain_Trial1 trainer=gpu fit=true trainer=ddp trainer.devices=2 datamodule.ims_per_batch=4
@dxoigmn I tried the following changes (set self.perturbation = None in the init function, commented the two lines you mentioned and then added self..perturbation back in a later line) and I could get the code to run for 1 GPU. But for multiple GPU, I still face the same problem - the progress bar freezes after 1 batch. Based on this experiment, I think the two lines you suggested to comment out might not be the cause of this problem (cos even after that I am facing the same problem). What do you think?
PFA the changes I made to perturber.py file
Update: With the changes (shown below) to the perturber.py file, I could get the runs going (for specific batch_sizes). @dxoigmn Does setting self.perturbation = perturbation
make sense in terms of adv training?
Here is the command used
CUDA_VISIBLE_DEVICES=2,3 python -m mart experiment=COCO_TorchvisionFasterRCNN_Adv task_name=COCO_TorchvisionFasterRCNNv2_AdvTrain_Trial1 fit=true trainer=ddp trainer.devices=2 datamodule.ims_per_batch=6
However, if I reduced datamodule.ims_per_batch=4
, I encounter the same "freeze in progress bar" problem when using 2 GPUs (I do not encounter this problem if trainer.devices=1
)
Could you please help me fix this?
I tried the following commands to use 2 GPUs python -m mart experiment=COCO_TorchvisionFasterRCNN_Adv task_name=COCO_TorchvisionFasterRCNNv2_AdvTrain_Trial1 trainer=gpu fit=true datamodule.ims_per_batch=4 trainer=ddp trainer.devices=2
I also tried CUDA_VISIBLE_DEVICES=4,5 python -m mart experiment=COCO_TorchvisionFasterRCNN_Adv task_name=COCO_TorchvisionFasterRCNNv2_AdvTrain_Trial1 trainer=gpu fit=true datamodule.ims_per_batch=4 trainer=ddp trainer.devices=2
And here is snippet of the COCO_TorchvisionFasterRCNN_Adv.yaml file![image](https://user-images.githubusercontent.com/63456911/228673814-6d03b103-4242-4a99-b781-704340045074.png)
For both these commands, I see the progress bar vary from 1 to 5 twice (updated once every 1.5 sec or so). After that, I do not see any progress. PFA a screenshot of the messages I see before the progress bar "freezes".
Also, once the progress bar stops to update, I am not able to terminate/interrupt the run using the typical Ctrl+C command (I had to kill the process based on pid).
Could you please help me fix this.