AlexMRuch
commented
3 years ago Follow-up: the freezing with ddp isn't just with the learning rate optimizer, it also happens with the main training function:
# Stop training, when loss metric does not improve on validation set
early_stop_callback = EarlyStopping(
monitor = "val_loss",
min_delta = 1e-4,
patience = 10,
verbose = False,
mode = "min"
)
lr_logger = LearningRateLogger() # log the learning rate
logger = TensorBoardLogger("lightning_logs") # log to tensorboard
# Update trainer
trainer = pl.Trainer(
max_epochs = 500,
gpus = [0,1],
distributed_backend = 'ddp', # https://pytorch-lightning.readthedocs.io/en/latest/multi_gpu.html#distributed-modes
weights_summary = "top",
gradient_clip_val = 0.1,
early_stop_callback = early_stop_callback,
#limit_train_batches = 20, # comment in for training, running validation every 20 batches
#fast_dev_run=True, # comment in to check that network or dataset has no serious bugs
callbacks = [lr_logger],
logger = logger
)
# Update model
tft = TemporalFusionTransformer.from_dataset(
training,
learning_rate = res.suggestion(), #use res.suggestion() or manual input from lr_finder above (e.g., 0.03)
hidden_size = 16, # biggest influence network size
attention_head_size = 1,
dropout = 0.1,
hidden_continuous_size = 8,
output_size = 7, # QuantileLoss has 7 quantiles by default
loss=QuantileLoss(),
log_interval = 10, # log example every 10 batches
reduce_on_plateau_patience = 4, # reduce learning automatically
)
print(f"Number of parameters in transformer model: {tft.size()/1e3:.1f}k")
# Train model
trainer.fit(
tft,
train_dataloader = train_dataloader,
val_dataloaders = val_dataloader
)Also, when I use ddp in that context, only 1 of my 2 GPUs is recognized by the system, even when setting gpus = 2 or doing gpus = [0,1]:
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
CUDA_VISIBLE_DEVICES: [0]
Number of parameters in transformer model: 23.4k
jdb78
^^^ Once I run 
Note that when I run it here, my terminal running jupyter lab does not throw the same 
dempseyryan
When I initialize my TFT trainer to use multiple GPUs
The library is able to recognize that I used both GPUs
However, when I try to find the optimal learning rate
I get an
AttributeError: Can't pickle local object '_apply_to_outputs.<locals>.decorator_fn.<locals>.new_func'error with the following trace:Any idea what may be triggering this? My guess is that because I'm not distributing across multiple machines, the pickle is getting messed up. That's fine and just indicates I misunderstood that setting for
distributed_backend, but moving on, I hit errors with the otherdistributed_backendsettings as well.Following https://pytorch-lightning.readthedocs.io/en/latest/multi_gpu.html#distributed-modes, when I hard-code
distributed_backendtoddp2, I get this traceand when I hard-code
distributed_backendtodp(which is what I would expect to work most readily), I getWhen I use
ddp(as recommended for pytorch, given the speedup), the pipeline freezes and runningwatch nvidia-smifrom the terminal just shows the GPUs aren't moving and aren't loading any memory for processing.This error is thrown using the same setup as I had in #85, which I got working on a single GPU but now that I'm doing multivariate time series across all 50 states I'd really like to use both my GPUs to speed up the runtime.
Thanks!