Open isaacmg opened 1 year ago
ncalls tottime percall cumtime percall filename:lineno(function)
4224/1 0.068 0.000 76.199 76.199 {built-in method builtins.exec}
1 0.000 0.000 76.199 76.199 trainer.py:1(<module>)
1 0.003 0.003 72.211 72.211 trainer.py:173(main)
1 0.001 0.001 72.207 72.207 trainer.py:81(train_function)
1 0.000 0.000 57.669 57.669 pytorch_training.py:73(train_transformer_style)
66173/1709 1.299 0.000 36.352 0.021 module.py:1188(_call_impl)
816 0.204 0.000 36.102 0.044 dsanet.py:333(forward)
411 0.878 0.002 31.533 0.077 linear_regression.py:54(simple_decode)
1 0.035 0.035 29.468 29.468 pytorch_training.py:343(torch_single_train)
2 2.582 1.291 28.099 14.049 pytorch_training.py:452(compute_validation)
3264 0.290 0.000 22.240 0.007 dsanet.py:112(forward)
816 0.033 0.000 21.326 0.026 dsanet.py:179(forward)
816 0.479 0.001 14.274 0.017 dsanet.py:241(forward)
1 0.000 0.000 13.037 13.037 trainer.py:16(handle_model_evaluation1)
1 0.000 0.000 12.613 12.613 evaluator.py:70(evaluate_model)
1 0.000 0.000 12.392 12.392 evaluator.py:194(infer_on_torch_model)
1 0.000 0.000 12.294 12.294 evaluator.py:383(generate_predictions)
1 0.000 0.000 12.293 12.293 evaluator.py:519(generate_decoded_predictions)
3264 1.231 0.000 12.179 0.004 dsanet.py:54(forward)
590 0.107 0.000 11.401 0.019 __init__.py:1(<module>)
72 0.001 0.000 10.570 0.147 _tensor.py:429(backward)
72 0.001 0.000 10.569 0.147 __init__.py:103(backward)
72 10.461 0.145 10.566 0.147 {method 'run_backward' of 'torch._C._EngineBase' objects}
72 0.092 0.001 10.008 0.139 optimizer.py:135(wrapper)
72 0.003 0.000 9.909 0.138 optimizer.py:19(_use_grad)
72 0.118 0.002 9.905 0.138 adam.py:168(step)
72 0.001 0.000 9.774 0.136 adam.py:257(adam)
72 4.426 0.061 9.770 0.136 adam.py:319(_single_tensor_adam)
3264 1.274 0.000 9.543 0.003 dsanet.py:93(forward)
1632 0.006 0.000 9.237 0.006 conv.py:462(forward)
1632 0.096 0.000 9.228 0.006 conv.py:454(_conv_forward)
1632 9.132 0.006 9.132 0.006 {built-in method torch.conv2d}
6528 0.200 0.000 5.595 0.001 conv.py:312(forward)
6528 0.108 0.000 5.294 0.001 conv.py:304(_conv_forward)
6528 5.186 0.001 5.186 0.001 {built-in method torch.conv1d}
17952 0.669 0.000 4.373 0.000 linear.py:113(forward)
5065/1554 0.021 0.000 4.244 0.003 <frozen importlib._bootstrap>:1022(_find_and_load)
4116/710 0.015 0.000 4.212 0.006 <frozen importlib._bootstrap>:987(_find_and_load_unlocked)
3899/668 0.015 0.000 4.135 0.006 <frozen importlib._bootstrap>:664(_load_unlocked)
3598/667 0.008 0.000 4.114 0.006 <frozen importlib._bootstrap_external>:877(exec_module)
5111/667 0.003 0.000 4.035 0.006 <frozen importlib._bootstrap>:233(_call_with_frames_removed)
485 0.011 0.000 3.703 0.008 dataloader.py:623(__next__)
17952 3.679 0.000 3.679 0.000 {built-in method torch._C._nn.linear}
485 0.006 0.000 3.652 0.008 dataloader.py:1286(_next_data)
482 0.002 0.000 3.306 0.007 dataloader.py:1253(_get_data)
482 0.002 0.000 3.305 0.007 dataloader.py:1107(_try_get_data)
482 0.006 0.000 3.303 0.007 queues.py:98(get)
892 0.459 0.001 3.197 0.004 pytorch_training.py:275(compute_loss)
1307/133 0.003 0.000 3.094 0.023 {built-in method builtins.__import__}
3264 0.469 0.000 3.029 0.001 dsanet.py:17(forward)
4592/2041 0.009 0.000 2.836 0.001 <frozen importlib._bootstrap>:1053(_handle_fromlist)
410 0.004 0.000 2.397 0.006 pytorch_training.py:229(handle_scaling)
821 0.282 0.000 2.392 0.003 pytorch_loaders.py:123(inverse_scale)
6528 0.203 0.000 2.385 0.000 normalization.py:189(forward)
72 0.016 0.000 2.202 0.031 optimizer.py:246(zero_grad)
5688 2.179 0.000 2.179 0.000 {method 'zero_' of 'torch._C._TensorBase' objects}
So looking things over the speed bottlenecks on functions are generally where you would expect. For instance, on simple_decode
the model(src)
line takes up 90% of the compute. In PyTorch 2.0 we could possibly use torch.compile() however this raises the question how would that influence confidence intervals using dropout. Would we still get different values?
Using torch.compile
in simple_decode
421 ms ± 11.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Not using torch.compile
in simple_decode
266 ms ± 18 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
This was for generating 400 time steps into the future without a GPU. For a longer sequence maybe that would be faster? In general though I don't think the torch.compile is worth the potential problems with losing the confidence interval that might come with it.
Timer unit: 1e-09 s
Total time: 0.0194293 s
File: <ipython-input-22-fdd33412dbaf>
Function: __getitem__ at line 102
Line # Hits Time Per Hit % Time Line Contents
==============================================================
102 @profiler
103 def __getitem__(self, idx: int):
104 2 4655940.0 2327970.0 24.0 rows = self.df.iloc[idx: self.forecast_history + idx]
105 2 4402.0 2201.0 0.0 targs_idx_start = self.forecast_history + idx
106 2 6252.0 3126.0 0.0 if self.no_scale:
107 2 222211.0 111105.5 1.1 targ_rows = self.unscaled_df.iloc[targs_idx_start: self.forecast_length + targs_idx_start]
108 else:
109 targ_rows = self.df.iloc[
110 targs_idx_start: self.forecast_length + targs_idx_start
111 ]
112 2 13969710.0 6984855.0 71.9 src_data = rows.to_numpy()
113 2 430522.0 215261.0 2.2 src_data = torch.from_numpy(src_data).float()
114 2 90531.0 45265.5 0.5 trg_dat = targ_rows.to_numpy()
115 2 48803.0 24401.5 0.3 trg_dat = torch.from_numpy(trg_dat).float()
116 2 931.0 465.5 0.0 return src_data, trg_dat
Line # Hits Time Per Hit % Time Line Contents
==============================================================
347 @profile
348 def torch_single_train(model: PyTorchForecast,
349 opt: optim.Optimizer,
350 criterion: Type[torch.nn.modules.loss._Loss],
351 data_loader: DataLoader,
352 takes_target: bool,
353 meta_data_model: PyTorchForecast,
354 meta_data_model_representation: torch.Tensor,
355 meta_loss=None,
356 multi_targets=1,
357 forward_params: Dict = {}) -> float:
358 """Function that performs training of a single model. Runs through one epoch of the data.
359
360 :param model: The PyTorchForecast model that is trained
361 :type model: PyTorchForecast
362 :param opt: The optimizer to use in the code
363 :type opt: optim.Optimizer
364 :param criterion: [description]
365 :type criterion: Type[torch.nn.modules.loss._Loss]
366 :param data_loader: [description]
367 :type data_loader: DataLoader
368 :param takes_target: A boolean that indicates whether the model takes the target during training
369 :type takes_target: bool
370 :param meta_data_model: If supplied a model that handles meta-data else None.
371 :type meta_data_model: PyTorchForecast
372 :param meta_data_model_representation: [description]
373 :type meta_data_model_representation: torch.Tensor
374 :param meta_loss: [description], defaults to None
375 :type meta_loss: [type], optional
376 :param multi_targets: [description], defaults to 1
377 :type multi_targets: int, optional
378 :param forward_params: [description], defaults to {}
379 :type forward_params: Dict, optional
380 :raises ValueError: [description]
381 :return: [description]
382 :rtype: float
383 """
384
385 1 1544.0 1544.0 0.0 probablistic = None
386 1 2490.0 2490.0 0.0 if "probabilistic" in model.params["model_params"]:
387 probablistic = True
388 1 257081.0 257081.0 0.0 print('running torch_single_train')
389 1 364.0 364.0 0.0 i = 0
390 1 205.0 205.0 0.0 output_std = None
391 1 215.0 215.0 0.0 mulit_targets_copy = multi_targets
392 1 375.0 375.0 0.0 running_loss = 0.0
393 72 306151827.0 4252108.7 48.1 for src, trg in data_loader:
394 72 5728173.0 79558.0 0.9 opt.zero_grad()
395 72 30887.0 429.0 0.0 if meta_data_model:
396 representation = meta_data_model.model.generate_representation(meta_data_model_representation)
397 forward_params["meta_data"] = representation
398 if meta_loss:
399 output = meta_data_model.model(meta_data_model_representation)
400 met_loss = compute_loss(meta_data_model_representation, output, torch.rand(2, 3, 2), meta_loss, None)
401 met_loss.backward()
402 72 25828.0 358.7 0.0 if takes_target:
403 forward_params["t"] = trg
404 72 127133.0 1765.7 0.0 elif "TemporalLoader" == model.params["dataset_params"]["class"]:
405 forward_params["x_mark_enc"] = src[1].to(model.device)
406 forward_params["x_dec"] = trg[1].to(model.device)
407 forward_params["x_mark_dec"] = trg[0].to(model.device)
408 src = src[0]
409 pred_len = model.model.pred_len
410 trg = trg[0]
411 trg[:, -pred_len:, :] = torch.zeros_like(trg[:, -pred_len:, :].long()).float().to(model.device)
412 # Assign to avoid other if statement
413 72 40084.0 556.7 0.0 elif "SeriesIDLoader" == model.params["dataset_params"]["class"]:
414 pass
415 72 304913.0 4234.9 0.0 src = src.to(model.device)
416 72 498787.0 6927.6 0.1 trg = trg.to(model.device)
417 72 80243336.0 1114490.8 12.6 output = model.model(src, **forward_params)
418 72 691238.0 9600.5 0.1 if hasattr(model.model, "pred_len"):
419 multi_targets = mulit_targets_copy
420 pred_len = model.model.pred_len
421 output = output[:, :, 0:multi_targets]
422 labels = trg[:, -pred_len:, 0:multi_targets]
423 multi_targets = False
424 72 102195.0 1419.4 0.0 if model.params["dataset_params"]["class"] == "GeneralClassificationLoader":
425 labels = trg
426 72 43360.0 602.2 0.0 elif multi_targets == 1:
427 72 2079566.0 28882.9 0.3 labels = trg[:, :, 0]
428 elif multi_targets > 1:
429 labels = trg[:, :, 0:multi_targets]
430 72 31689.0 440.1 0.0 if probablistic:
431 output1 = output
432 output = output.mean
433 output_std = output1.stddev
434 72 67559.0 938.3 0.0 if type(criterion) == list:
435 loss = multi_crit(criterion, output, labels, None)
436 else:
437 72 11777639.0 163578.3 1.9 loss = compute_loss(labels, output, src, criterion, None, probablistic, output_std, m=multi_targets)
438 72 1583531.0 21993.5 0.2 if loss > 100:
439 print("Warning: high loss detected")
440 72 176070293.0 2445420.7 27.7 loss.backward()
441 72 47679070.0 662209.3 7.5 opt.step()
442 72 2301492.0 31965.2 0.4 if torch.isnan(loss) or loss == float('inf'):
443 raise ValueError("Error infinite or NaN loss detected. Try normalizing data or performing interpolation")
444 72 141336.0 1963.0 0.0 running_loss += loss.item()
445 72 58359.0 810.5 0.0 i += 1
446 1 354967.0 354967.0 0.1 print("The running loss is: ")
447 1 99189.0 99189.0 0.0 print(running_loss)
448 1 90326.0 90326.0 0.0 print("The number of items in train is: " + str(i))
449 1 1506.0 1506.0 0.0 total_loss = running_loss / float(i)
Patch coverage: 75.00
% and project coverage change: -11.20
:warning:
Comparison is base (
e68c1db
) 75.79% compared to head (4752f22
) 64.59%.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.
The goal of this PR is add automatic optimization/time calculation metrics. Also maybe do some optimization of some the code and find out where it does not run efficiently.