search

NVlabs / stylegan3

Official PyTorch implementation of StyleGAN3
Other
6.43k stars 1.13k forks source link

stuck at "Training for 25000 kimg..." #49

Open zjgt opened 3 years ago

zjgt commented 3 years ago

Describe the bug Previously stuck at relu step (see issue #48 ), now stuck at the "Training for 25000 kimg step. The reason is probably ninja (version 1.10.2), the GPU stopped being engaged (from P2 to P8), possible due to subprocess spawning error. The key error message seems to be: LINK : fatal error LNK1104: cannot open file 'bias_act_plugin.pyd' ninja: build stopped: subcommand failed.

To Reproduce (stylegan3) C:\Users\fsn20\PycharmProjects\stylegan3>python train.py --outdir=results --cfg=stylegan3-r --data=dataset --gpus=2 --batch=16 --gamma=8.2

Training options: { "G_kwargs": { "class_name": "training.networks_stylegan3.Generator", "z_dim": 512, "w_dim": 512, "mapping_kwargs": { "num_layers": 2 }, "channel_base": 65536, "channel_max": 1024, "magnitude_ema_beta": 0.9994456359721023, "conv_kernel": 1, "use_radial_filters": true }, "D_kwargs": { "class_name": "training.networks_stylegan2.Discriminator", "block_kwargs": { "freeze_layers": 0 }, "mapping_kwargs": {}, "epilogue_kwargs": { "mbstd_group_size": 4 }, "channel_base": 32768, "channel_max": 512 }, "G_opt_kwargs": { "class_name": "torch.optim.Adam", "betas": [ 0, 0.99 ], "eps": 1e-08, "lr": 0.0025 }, "D_opt_kwargs": { "class_name": "torch.optim.Adam", "betas": [ 0, 0.99 ], "eps": 1e-08, "lr": 0.002 }, "loss_kwargs": { "class_name": "training.loss.StyleGAN2Loss", "r1_gamma": 8.2, "blur_init_sigma": 10, "blur_fade_kimg": 100.0 }, "data_loader_kwargs": { "pin_memory": true, "prefetch_factor": 2, "num_workers": 3 }, "training_set_kwargs": { "class_name": "training.dataset.ImageFolderDataset", "path": "dataset", "use_labels": false, "max_size": 92, "xflip": false, "resolution": 512, "random_seed": 0 }, "num_gpus": 2, "batch_size": 16, "batch_gpu": 8, "metrics": [ "fid50k_full" ], "total_kimg": 25000, "kimg_per_tick": 4, "image_snapshot_ticks": 50, "network_snapshot_ticks": 50, "random_seed": 0, "ema_kimg": 5.0, "augment_kwargs": { "class_name": "training.augment.AugmentPipe", "xflip": 1, "rotate90": 1, "xint": 1, "scale": 1, "rotate": 1, "aniso": 1, "xfrac": 1, "brightness": 1, "contrast": 1, "lumaflip": 1, "hue": 1, "saturation": 1 }, "ada_target": 0.6, "run_dir": "results\00005-stylegan3-r-dataset-gpus2-batch16-gamma8.2" }

Output directory: results\00005-stylegan3-r-dataset-gpus2-batch16-gamma8.2 Number of GPUs: 2 Batch size: 16 images Training duration: 25000 kimg Dataset path: dataset Dataset size: 92 images Dataset resolution: 512 Dataset labels: False Dataset x-flips: False

Creating output directory... Launching processes... Loading training set...

Num images: 92 Image shape: [3, 512, 512] Label shape: [0]

Constructing networks... Setting up PyTorch plugin "bias_act_plugin"... Done. Setting up PyTorch plugin "filtered_lrelu_plugin"... Done.

Generator Parameters Buffers Output shape Datatype

mapping.fc0 262656 - [8, 512] float32 mapping.fc1 262656 - [8, 512] float32 mapping - 512 [8, 16, 512] float32 synthesis.input.affine 2052 - [8, 4] float32 synthesis.input 1048576 3081 [8, 1024, 36, 36] float32 synthesis.L0_36_1024.affine 525312 - [8, 1024] float32 synthesis.L0_36_1024 1049600 157 [8, 1024, 36, 36] float32 synthesis.L1_36_1024.affine 525312 - [8, 1024] float32 synthesis.L1_36_1024 1049600 157 [8, 1024, 36, 36] float32 synthesis.L2_52_1024.affine 525312 - [8, 1024] float32 synthesis.L2_52_1024 1049600 169 [8, 1024, 52, 52] float32 synthesis.L3_52_1024.affine 525312 - [8, 1024] float32 synthesis.L3_52_1024 1049600 157 [8, 1024, 52, 52] float32 synthesis.L4_84_1024.affine 525312 - [8, 1024] float32 synthesis.L4_84_1024 1049600 169 [8, 1024, 84, 84] float16 synthesis.L5_84_1024.affine 525312 - [8, 1024] float32 synthesis.L5_84_1024 1049600 157 [8, 1024, 84, 84] float16 synthesis.L6_148_1024.affine 525312 - [8, 1024] float32 synthesis.L6_148_1024 1049600 169 [8, 1024, 148, 148] float16 synthesis.L7_148_967.affine 525312 - [8, 1024] float32 synthesis.L7_148_967 991175 157 [8, 967, 148, 148] float16 synthesis.L8_276_645.affine 496071 - [8, 967] float32 synthesis.L8_276_645 624360 169 [8, 645, 276, 276] float16 synthesis.L9_276_431.affine 330885 - [8, 645] float32 synthesis.L9_276_431 278426 157 [8, 431, 276, 276] float16 synthesis.L10_532_287.affine 221103 - [8, 431] float32 synthesis.L10_532_287 123984 169 [8, 287, 532, 532] float16 synthesis.L11_532_192.affine 147231 - [8, 287] float32 synthesis.L11_532_192 55296 157 [8, 192, 532, 532] float16 synthesis.L12_532_128.affine 98496 - [8, 192] float32 synthesis.L12_532_128 24704 25 [8, 128, 532, 532] float16 synthesis.L13_512_128.affine 65664 - [8, 128] float32 synthesis.L13_512_128 16512 25 [8, 128, 512, 512] float16 synthesis.L14_512_3.affine 65664 - [8, 128] float32 synthesis.L14_512_3 387 1 [8, 3, 512, 512] float16 synthesis - - [8, 3, 512, 512] float32

Total 16665594 5588 - -

Setting up PyTorch plugin "upfirdn2d_plugin"... Done.

Discriminator Parameters Buffers Output shape Datatype

b512.fromrgb 256 16 [8, 64, 512, 512] float16 b512.skip 8192 16 [8, 128, 256, 256] float16 b512.conv0 36928 16 [8, 64, 512, 512] float16 b512.conv1 73856 16 [8, 128, 256, 256] float16 b512 - 16 [8, 128, 256, 256] float16 b256.skip 32768 16 [8, 256, 128, 128] float16 b256.conv0 147584 16 [8, 128, 256, 256] float16 b256.conv1 295168 16 [8, 256, 128, 128] float16 b256 - 16 [8, 256, 128, 128] float16 b128.skip 131072 16 [8, 512, 64, 64] float16 b128.conv0 590080 16 [8, 256, 128, 128] float16 b128.conv1 1180160 16 [8, 512, 64, 64] float16 b128 - 16 [8, 512, 64, 64] float16 b64.skip 262144 16 [8, 512, 32, 32] float16 b64.conv0 2359808 16 [8, 512, 64, 64] float16 b64.conv1 2359808 16 [8, 512, 32, 32] float16 b64 - 16 [8, 512, 32, 32] float16 b32.skip 262144 16 [8, 512, 16, 16] float32 b32.conv0 2359808 16 [8, 512, 32, 32] float32 b32.conv1 2359808 16 [8, 512, 16, 16] float32 b32 - 16 [8, 512, 16, 16] float32 b16.skip 262144 16 [8, 512, 8, 8] float32 b16.conv0 2359808 16 [8, 512, 16, 16] float32 b16.conv1 2359808 16 [8, 512, 8, 8] float32 b16 - 16 [8, 512, 8, 8] float32 b8.skip 262144 16 [8, 512, 4, 4] float32 b8.conv0 2359808 16 [8, 512, 8, 8] float32 b8.conv1 2359808 16 [8, 512, 4, 4] float32 b8 - 16 [8, 512, 4, 4] float32 b4.mbstd - - [8, 513, 4, 4] float32 b4.conv 2364416 16 [8, 512, 4, 4] float32 b4.fc 4194816 - [8, 512] float32 b4.out 513 - [8, 1] float32

Total 28982849 480 - -

Setting up augmentation... Distributing across 2 GPUs... Setting up training phases... Exporting sample images... Initializing logs... Training for 25000 kimg...

Aborted! Process SpawnProcess-2: Traceback (most recent call last): File "C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\utils\cpp_extension.py", line 1666, in _run_ninja_build subprocess.run( File "C:\Users\fsn20\miniconda3\envs\stylegan3\lib\subprocess.py", line 528, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\multiprocessing\spawn.py", line 59, in _wrap fn(i, *args) File "C:\Users\fsn20\PycharmProjects\stylegan3\train.py", line 47, in subprocess_fn training_loop.training_loop(rank=rank, c) File "C:\Users\fsn20\PycharmProjects\stylegan3\training\training_loop.py", line 278, in training_loop loss.accumulate_gradients(phase=phase.name, real_img=real_img, real_c=real_c, gen_z=gen_z, gen_c=gen_c, gain=phase.interval, cur_nimg=cur_nimg) File "C:\Users\fsn20\PycharmProjects\stylegan3\training\loss.py", line 74, in accumulate_gradients gen_img, _gen_ws = self.run_G(gen_z, gen_c) File "C:\Users\fsn20\PycharmProjects\stylegan3\training\loss.py", line 43, in run_G ws = self.G.mapping(z, c, update_emas=update_emas) File "C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl return forward_call(*input, *kwargs) File "C:\Users\fsn20\PycharmProjects\stylegan3\training\networks_stylegan3.py", line 151, in forward x = getattr(self, f'fc{idx}')(x) File "C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl return forward_call(input, kwargs) File "C:\Users\fsn20\PycharmProjects\stylegan3\training\networks_stylegan3.py", line 100, in forward x = bias_act.bias_act(x, b, act=self.activation) File "C:\Users\fsn20\PycharmProjects\stylegan3\torch_utils\ops\bias_act.py", line 84, in bias_act if impl == 'cuda' and x.device.type == 'cuda' and _init(): File "C:\Users\fsn20\PycharmProjects\stylegan3\torch_utils\ops\bias_act.py", line 41, in _init _plugin = custom_ops.get_plugin( File "C:\Users\fsn20\PycharmProjects\stylegan3\torch_utils\custom_ops.py", line 136, in get_plugin torch.utils.cpp_extension.load(name=module_name, build_directory=cached_build_dir, File "C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\utils\cpp_extension.py", line 1080, in load return _jit_compile( File "C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\utils\cpp_extension.py", line 1293, in _jit_compile _write_ninja_file_and_build_library( File "C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\utils\cpp_extension.py", line 1405, in _write_ninja_file_and_build_library _run_ninja_build( File "C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\utils\cpp_extension.py", line 1682, in _run_ninja_build raise RuntimeError(message) from e RuntimeError: Error building extension 'bias_act_plugin': [1/2] cl /showIncludes -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\Users\fsn20\miniconda 3\envs\stylegan3\lib\site-packages\torch\include -IC:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\fsn20\miniconda3\env s\stylegan3\lib\site-packages\torch\include\TH -IC:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUD A\v11.4\include" -IC:\Users\fsn20\miniconda3\envs\stylegan3\Include -D_GLIBCXX_USE_CXX11_ABI=0 /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc -c C:\Users\fsn20 \AppData\Local\torch_extensions\torch_extensions\Cache\bias_act_plugin\3cb576a0039689487cfba59279dd6d46-nvidia-geforce-rtx-3090\bias_act.cpp /Fobias_act.o Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30133 for x64 Copyright (C) Microsoft Corporation. All rights reserved.

C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=at::Tensor ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T

' being compiled with [ T=at::Tensor ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=at::Tensor ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<at::Tens or>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\ATen/core/TensorBody.h(734): note: see reference to class template instantiation 'c10::optional<at::Tenso r>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=at::Tensor ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=at::Generator ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=at::Generator ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=at::Generator ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<at::Gene rator>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\ATen/core/TensorBody.h(800): note: see reference to class template instantiation 'c10::optional<at::Gener ator>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=at::Generator ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=c10::impl::InlineDeviceGuard ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=c10::impl::InlineDeviceGuard ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=c10::impl::InlineDeviceGuard ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<c10::imp l::InlineDeviceGuard>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/core/impl/InlineDeviceGuard.h(427): note: see reference to class template instantiation 'c10::optiona l<c10::impl::InlineDeviceGuard>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/core/DeviceGuard.h(178): note: see reference to class template instantiation 'c10::impl::InlineOption alDeviceGuard' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=c10::impl::InlineDeviceGuard ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=std::string ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=std::string ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=std::string ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<std::str ing>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\ATen/core/jit_type_base.h(107): note: see reference to class template instantiation 'c10::optional<std::s tring>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=std::string ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=std::vector<c10::ShapeSymbol,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=std::vector<c10::ShapeSymbol,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=std::vector<c10::ShapeSymbol,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<std::vec tor<c10::ShapeSymbol,std::allocator>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\ATen/core/jit_type.h(351): note: see reference to class template instantiation 'c10::optional<std::vector <c10::ShapeSymbol,std::allocator>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=std::vector<c10::ShapeSymbol,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=std::vector<c10::optional,std::allocator<c10::optional>> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=std::vector<c10::optional,std::allocator<c10::optional>> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=std::vector<c10::optional,std::allocator<c10::optional>> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<std::vec tor<c10::optional,std::allocator<c10::optional>>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\ATen/core/jit_type.h(425): note: see reference to class template instantiation 'c10::optional<std::vector <c10::optional,std::allocator<c10::optional>>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\ATen/core/jit_type.h(664): note: see reference to class template instantiation 'c10::VaryingShape<c10::St ride>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=std::vector<c10::optional,std::allocator<c10::optional>> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=std::vector<c10::optional,std::allocator<c10::optional>> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=std::vector<c10::optional,std::allocator<c10::optional>> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=std::vector<c10::optional,std::allocator<c10::optional>> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<std::vec tor<c10::optional,std::allocator<c10::optional>>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\ATen/core/jit_type.h(425): note: see reference to class template instantiation 'c10::optional<std::vector <c10::optional,std::allocator<c10::optional>>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\ATen/core/jit_type.h(470): note: see reference to class template instantiation 'c10::VaryingShape<int64_t ' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=std::vector<c10::optional,std::allocator<c10::optional>> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=std::vector<int64_t,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=std::vector<int64_t,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=std::vector<int64_t,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<std::vec tor<int64_t,std::allocator>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\ATen/core/jit_type.h(568): note: see reference to class template instantiation 'c10::optional<std::vector <int64_t,std::allocator>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=std::vector<int64_t,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=c10::QualifiedName ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=c10::QualifiedName ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=c10::QualifiedName ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<c10::Qua lifiedName>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\ATen/core/jit_type.h(903): note: see reference to class template instantiation 'c10::optional<c10::Qualif iedName>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=c10::QualifiedName ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=c10::impl::InlineStreamGuard ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=c10::impl::InlineStreamGuard ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=c10::impl::InlineStreamGuard ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<c10::imp l::InlineStreamGuard>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/core/impl/InlineStreamGuard.h(196): note: see reference to class template instantiation 'c10::optiona l<c10::impl::InlineStreamGuard>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/core/StreamGuard.h(139): note: see reference to class template instantiation 'c10::impl::InlineOption alStreamGuard' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=c10::impl::InlineStreamGuard ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=c10::impl::VirtualGuardImpl ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=c10::impl::VirtualGuardImpl ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=c10::impl::VirtualGuardImpl ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<c10::imp l::VirtualGuardImpl>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/core/impl/InlineStreamGuard.h(231): note: see reference to class template instantiation 'c10::optiona l' being compiled with [ T=c10::impl::VirtualGuardImpl ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/core/StreamGuard.h(162): note: see reference to class template instantiation 'c10::impl::InlineMultiS treamGuard' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=c10::impl::VirtualGuardImpl ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=std::vector<std::reference_wrapper,std::allocator<std::reference_wrapper>> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=std::vector<std::reference_wrapper,std::allocator<std::reference_wrapper>> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=std::vector<std::reference_wrapper,std::allocator<std::reference_wrapper>> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<std::vec tor<std::reference_wrapper,std::allocator<std::reference_wrapper>>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\ATen/core/ivalue_inl.h(362): note: see reference to class template instantiation 'c10::optional<std::vect or<std::reference_wrapper,std::allocator<std::reference_wrapper>>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=std::vector<std::reference_wrapper,std::allocator<std::reference_wrapper>> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=c10::OperatorName ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=c10::OperatorName ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=c10::OperatorName ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<c10::Ope ratorName>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\ATen/record_function.h(306): note: see reference to class template instantiation 'c10::optional<c10::Oper atorName>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=c10::OperatorName ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=at::DimVector ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=at::DimVector ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=at::DimVector ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<at::DimV ector>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\ATen/TensorIterator.h(616): note: see reference to class template instantiation 'c10::optional<at::DimVec tor>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=at::DimVector ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=torch::autograd::ViewInfo ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=torch::autograd::ViewInfo ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=torch::autograd::ViewInfo ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<torch::a utograd::ViewInfo>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch/csrc/autograd/variable.h(529): note: see reference to class template instantiation 'c10::optional<t orch::autograd::ViewInfo>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=torch::autograd::ViewInfo ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=c10::IValue ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=c10::IValue ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=c10::IValue ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<c10::IVa lue>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\ATen/core/function_schema.h(115): note: see reference to class template instantiation 'c10::optional<c10: :IValue>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=c10::IValue ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=c10::AliasInfo ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=c10::AliasInfo ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=c10::AliasInfo ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<c10::Ali asInfo>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\ATen/core/function_schema.h(118): note: see reference to class template instantiation 'c10::optional<c10: :AliasInfo>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=c10::AliasInfo ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=std::tuple<std::string,size_t,size_t> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=std::tuple<std::string,size_t,size_t> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=std::tuple<std::string,size_t,size_t> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<std::tup le<std::string,size_t,size_t>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch/csrc/jit/frontend/source_range.h(146): note: see reference to class template instantiation 'c10::op tional<std::tuple<std::string,size_t,size_t>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=std::tuple<std::string,size_t,size_t> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=torch::jit::SourceRange ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=torch::jit::SourceRange ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=torch::jit::SourceRange ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<torch::j it::SourceRange>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch/csrc/jit/frontend/source_range.h(169): note: see reference to class template instantiation 'c10::op tional' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=torch::jit::SourceRange ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=torch::jit::InlinedCallStackPtr ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=torch::jit::InlinedCallStackPtr ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=torch::jit::InlinedCallStackPtr ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<torch::j it::InlinedCallStackPtr>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch/csrc/jit/ir/scope.h(121): note: see reference to class template instantiation 'c10::optional<torch: :jit::InlinedCallStackPtr>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=torch::jit::InlinedCallStackPtr ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=torch::jit::ModuleInstanceInfo ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=torch::jit::ModuleInstanceInfo ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=torch::jit::ModuleInstanceInfo ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<torch::j it::ModuleInstanceInfo>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch/csrc/jit/ir/scope.h(125): note: see reference to class template instantiation 'c10::optional<torch: :jit::ModuleInstanceInfo>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=torch::jit::ModuleInstanceInfo ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=torch::jit::ScopePtr ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=torch::jit::ScopePtr ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=torch::jit::ScopePtr ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<torch::j it::ScopePtr>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch/csrc/jit/ir/constants.h(29): note: see reference to class template instantiation 'c10::optional<tor ch::jit::ScopePtr>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=torch::jit::ScopePtr ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=c10::impl::AnnotatedSchema ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=c10::impl::AnnotatedSchema ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=c10::impl::AnnotatedSchema ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<c10::imp l::AnnotatedSchema>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\ATen/core/dispatch/OperatorEntry.h(182): note: see reference to class template instantiation 'c10::option al' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=c10::impl::AnnotatedSchema ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=c10::impl::OperatorEntry::CppSignatureWithDebug ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=c10::impl::OperatorEntry::CppSignatureWithDebug ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=c10::impl::OperatorEntry::CppSignatureWithDebug ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<c10::imp l::OperatorEntry::CppSignatureWithDebug>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\ATen/core/dispatch/OperatorEntry.h(234): note: see reference to class template instantiation 'c10::option al' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=c10::impl::OperatorEntry::CppSignatureWithDebug ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=c10::either<c10::OperatorName,c10::FunctionSchema> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=c10::either<c10::OperatorName,c10::FunctionSchema> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=c10::either<c10::OperatorName,c10::FunctionSchema> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<c10::eit her<c10::OperatorName,c10::FunctionSchema>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\ATen/core/op_registration/op_registration.h(434): note: see reference to class template instantiation 'c1 0::optional<c10::either<c10::OperatorName,c10::FunctionSchema>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=c10::either<c10::OperatorName,c10::FunctionSchema> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=at::ThreadLocalState ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=at::ThreadLocalState ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=at::ThreadLocalState ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<at::Thre adLocalState>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch/csrc/jit/runtime/interpreter.h(141): note: see reference to class template instantiation 'c10::opti onal' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=at::ThreadLocalState ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=std::shared_ptr ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=std::shared_ptr ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=std::shared_ptr ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<std::sha red_ptr>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch/csrc/jit/api/function_impl.h(124): note: see reference to class template instantiation 'c10::option al<std::shared_ptr>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=std::shared_ptr ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=torch::jit::Method ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=torch::jit::Method ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=torch::jit::Method ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<torch::j it::Method>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch/csrc/jit/api/object.h(41): note: see reference to class template instantiation 'c10::optional<torch ::jit::Method>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=torch::jit::Method ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=std::vector<std::string,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=std::vector<std::string,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=std::vector<std::string,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<std::vec tor<std::string,std::allocator>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch/csrc/jit/api/module.h(295): note: see reference to class template instantiation 'c10::optional<std: :vector<std::string,std::allocator>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=std::vector<std::string,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=torch::nn::TripletMarginWithDistanceLossOptions::distance_function_t ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=torch::nn::TripletMarginWithDistanceLossOptions::distance_function_t ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=torch::nn::TripletMarginWithDistanceLossOptions::distance_function_t ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<torch::n n::TripletMarginWithDistanceLossOptions::distance_function_t>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch\csrc\api\include\torch/nn/options/loss.h(404): note: see reference to class template instantiation 'c10::optional' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=torch::nn::TripletMarginWithDistanceLossOptions::distance_function_t ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=std::vector<double,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=std::vector<double,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=std::vector<double,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<std::vec tor<double,std::allocator>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch\csrc\api\include\torch/nn/options/upsampling.h(26): note: see reference to class template instantia tion 'c10::optional<std::vector<double,std::allocator>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=std::vector<double,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=std::tuple<at::Tensor,at::Tensor> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=std::tuple<at::Tensor,at::Tensor> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=std::tuple<at::Tensor,at::Tensor> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<std::tup le<at::Tensor,at::Tensor>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch\csrc\api\include\torch/nn/modules/rnn.h(157): note: see reference to class template instantiation ' c10::optional<std::tuple<at::Tensor,at::Tensor>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=std::tuple<at::Tensor,at::Tensor> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=std::function<void (const torch::autograd::profiler::thread_event_lists &)> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=std::function<void (const torch::autograd::profiler::thread_event_lists &)> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=std::function<void (const torch::autograd::profiler::thread_event_lists &)> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<std::fun ction<void (const torch::autograd::profiler::thread_event_lists &)>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch/csrc/autograd/profiler_legacy.h(514): note: see reference to class template instantiation 'c10::opt ional<std::function<void (const torch::autograd::profiler::thread_event_lists &)>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=std::function<void (const torch::autograd::profiler::thread_event_lists &)> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=std::vector<std::vector<torch::autograd::profiler::LegacyEvent,std::allocator>,std::allocator<std::vector<torch::autograd::p rofiler::LegacyEvent,std::allocator>>> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=std::vector<std::vector<torch::autograd::profiler::LegacyEvent,std::allocator>,std::allocator<std::vector<torch::autograd::p rofiler::LegacyEvent,std::allocator>>> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=std::vector<std::vector<torch::autograd::profiler::LegacyEvent,std::allocator>,std::allocator<std::vector<torch::autograd::p rofiler::LegacyEvent,std::allocator>>> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<std::vec tor<std::vector<torch::autograd::profiler::LegacyEvent,std::allocator>,std::allocator<std::vector<torch::autograd::profiler::LegacyEvent,s td::allocator>>>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch/csrc/autograd/profiler_legacy.h(588): note: see reference to class template instantiation 'c10::opt ional<std::vector<std::vector<torch::autograd::profiler::LegacyEvent,std::allocator>,std::allocator<std::vector<torch::autograd::profiler: :LegacyEvent,std::allocator>>>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=std::vector<std::vector<torch::autograd::profiler::LegacyEvent,std::allocator>,std::allocator<std::vector<torch::autograd::p rofiler::LegacyEvent,std::allocator>>> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=std::vector<at::Tensor,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=std::vector<at::Tensor,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=std::vector<at::Tensor,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<std::vec tor<at::Tensor,std::allocator>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch\csrc\api\include\torch/optim/lbfgs.h(47): note: see reference to class template instantiation 'c10: :optional<std::vector<at::Tensor,std::allocator>>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=std::vector<at::Tensor,std::allocator> ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=c10::impl::InlineDeviceGuard ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=c10::impl::InlineDeviceGuard ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=c10::impl::InlineDeviceGuard ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<c10::imp l::InlineDeviceGuard>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/core/impl/InlineDeviceGuard.h(427): note: see reference to class template instantiation 'c10::optiona l<c10::impl::InlineDeviceGuard>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/cuda/CUDAGuard.h(140): note: see reference to class template instantiation 'c10::impl::InlineOptional DeviceGuard' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=c10::impl::InlineDeviceGuard ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=c10::impl::InlineStreamGuard ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=c10::impl::InlineStreamGuard ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=c10::impl::InlineStreamGuard ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<c10::imp l::InlineStreamGuard>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/core/impl/InlineStreamGuard.h(196): note: see reference to class template instantiation 'c10::optiona l<c10::impl::InlineStreamGuard>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/cuda/CUDAGuard.h(273): note: see reference to class template instantiation 'c10::impl::InlineOptional StreamGuard' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=c10::impl::InlineStreamGuard ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(183): warning C4624: 'c10::constexpr_storage_t': destructor was implicitly defined as deleted with [ T=c10::cuda::impl::CUDAGuardImpl ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(367): note: see reference to class template instantiation 'c10::constexpr_storage_t<T ' being compiled with [ T=c10::cuda::impl::CUDAGuardImpl ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to class template instantiation 'c10::trivially_copyable_op timization_optional_base' being compiled with [ T=c10::cuda::impl::CUDAGuardImpl ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(427): note: see reference to alias template instantiation 'c10::OptionalBase<c10::cud a::impl::CUDAGuardImpl>' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/core/impl/InlineStreamGuard.h(231): note: see reference to class template instantiation 'c10::optiona l' being compiled with [ T=c10::cuda::impl::CUDAGuardImpl ] C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/cuda/CUDAGuard.h(292): note: see reference to class template instantiation 'c10::impl::InlineMultiStr eamGuard' being compiled C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\c10/util/Optional.h(395): warning C4624: 'c10::trivially_copyable_optimization_optional_base': destruc tor was implicitly defined as deleted with [ T=c10::cuda::impl::CUDAGuardImpl ] [2/2] "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64/link.exe" bias_act.o bias_act.cuda.o /nologo /DLL c10.lib c10_cuda. lib torch_cpu.lib torch_cuda_cu.lib -INCLUDE:?searchsorted_cuda@native@at@@YA?AVTensor@2@AEBV32@0_N1@Z torch_cuda_cpp.lib -INCLUDE:?warp_size@cuda@at@@YAHXZ torch.lib /LIBPATH:C: \Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\lib torch_python.lib /LIBPATH:C:\Users\fsn20\miniconda3\envs\stylegan3\libs "/LIBPATH:C:\Program Files\NVIDIA GPU C omputing Toolkit\CUDA\v11.4\lib/x64" cudart.lib /out:bias_act_plugin.pyd FAILED: bias_act_plugin.pyd "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64/link.exe" bias_act.o bias_act.cuda.o /nologo /DLL c10.lib c10_cuda.lib to rch_cpu.lib torch_cuda_cu.lib -INCLUDE:?searchsorted_cuda@native@at@@YA?AVTensor@2@AEBV32@0_N1@Z torch_cuda_cpp.lib -INCLUDE:?warp_size@cuda@at@@YAHXZ torch.lib /LIBPATH:C:\Users \fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\lib torch_python.lib /LIBPATH:C:\Users\fsn20\miniconda3\envs\stylegan3\libs "/LIBPATH:C:\Program Files\NVIDIA GPU Computi ng Toolkit\CUDA\v11.4\lib/x64" cudart.lib /out:bias_act_plugin.pyd LINK : fatal error LNK1104: cannot open file 'bias_act_plugin.pyd' ninja: build stopped: subcommand failed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\fsn20\miniconda3\envs\stylegan3\lib\multiprocessing\process.py", line 315, in _bootstrap self.run() File "C:\Users\fsn20\miniconda3\envs\stylegan3\lib\multiprocessing\process.py", line 108, in run self._target(*self._args, **self._kwargs) File "C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\multiprocessing\spawn.py", line 65, in _wrap error_queue.put(traceback.format_exc()) File "C:\Users\fsn20\miniconda3\envs\stylegan3\lib\multiprocessing\queues.py", line 375, in put self._writer.send_bytes(obj) File "C:\Users\fsn20\miniconda3\envs\stylegan3\lib\multiprocessing\connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "C:\Users\fsn20\miniconda3\envs\stylegan3\lib\multiprocessing\connection.py", line 288, in _send_bytes waitres = _winapi.WaitForMultipleObjects( KeyboardInterrupt

Desktop (please complete the following information):

Additional context Ninja log file is here:

ninja log v5

3 3802 6568154534670044 bias_act.cuda.o 65b65521cc23fcdf 0 7728 6568154573915312 bias_act.o bdbdc5d60e762133 7732 7808 6568154574795210 bias_act_plugin.pyd 32eb53e31286dd81 4 7767 6568155174014625 bias_act.o bdbdc5d60e762133 7768 7839 6568155174782573 bias_act_plugin.pyd 32eb53e31286dd81 4 7597 6568156027291701 bias_act.o bdbdc5d60e762133 5 3747 6568162533323240 bias_act.cuda.o c9333a9bf1b2b303 2 7172 6568162567513754 bias_act.o b32d4bab1a9ab492 3 6987 6568164823930409 bias_act.o b32d4bab1a9ab492 6990 7082 6568164824958035 bias_act_plugin.pyd df4028bbc0a8483d 3 6859 6568165680850791 bias_act.o b32d4bab1a9ab492 5 3880 6568177700294856 bias_act.cuda.o 65b65521cc23fcdf 2 7920 6568177740621033 bias_act.o bdbdc5d60e762133 7922 7997 6568177741508662 bias_act_plugin.pyd 32eb53e31286dd81 4 7592 6568178590573265 bias_act.o bdbdc5d60e762133

ninja build is here: ninja_required_version = 1.3 cxx = cl nvcc = C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\bin\nvcc

cflags = -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include -IC:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\TH -IC:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\include" -IC:\Users\fsn20\miniconda3\envs\stylegan3\Include -D_GLIBCXX_USE_CXX11_ABI=0 /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc post_cflags = cuda_cflags = -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include -IC:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\TH -IC:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\include" -IC:\Users\fsn20\miniconda3\envs\stylegan3\Include -D_GLIBCXX_USE_CXX11_ABI=0 -DCUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -DCUDA_NO_BFLOAT16_CONVERSIONS -DCUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --use_fast_math cuda_post_cflags = ldflags = /DLL c10.lib c10_cuda.lib torch_cpu.lib torch_cuda_cu.lib -INCLUDE:?searchsorted_cuda@native@at@@YA?AVTensor@2@AEBV32@0_N1@Z torch_cuda_cpp.lib -INCLUDE:?warp_size@cuda@at@@YAHXZ torch.lib /LIBPATH:C:\Users\fsn20\miniconda3\envs\stylegan3\lib\site-packages\torch\lib torch_python.lib /LIBPATH:C:\Users\fsn20\miniconda3\envs\stylegan3\libs "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\lib/x64" cudart.lib

rule compile command = cl /showIncludes $cflags -c $in /Fo$out $post_cflags deps = msvc

rule cuda_compile depfile = $out.d deps = gcc command = $nvcc --generate-dependencies-with-compile --dependency-output $out.d $cuda_cflags -c $in -o $out $cuda_post_cflags

rule link command = "C$:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64/link.exe" $in /nologo $ldflags /out:$out

build bias_act.o: compile C$:\Users\fsn20\AppData\Local\torch_extensions\torch_extensions\Cache\bias_act_plugin\3cb576a0039689487cfba59279dd6d46-nvidia-geforce-rtx-3090\bias_act.cpp build bias_act.cuda.o: cuda_compile C$:\Users\fsn20\AppData\Local\torch_extensions\torch_extensions\Cache\bias_act_plugin\3cb576a0039689487cfba59279dd6d46-nvidia-geforce-rtx-3090\bias_act.cu

build bias_act_plugin.pyd: link bias_act.o bias_act.cuda.o

default bias_act_plugin.pyd

zjgt commented 3 years ago

Just to add, training on single GPU works.

jannehellsten commented 3 years ago

Something seems to go wrong during extension compilation. Not quite sure what. We don't typically run multi-GPU on Windows so I don't believe we've encountered this one.

I'm assuming you nuked the torch_extensions dir and retried with this bug too?

zjgt commented 3 years ago

I did completely delete the torch_extensions with no help. Multiple GPU training work well with the stylegan2. Something strange. I may just stick with single GPU for now. Thanks.

jdumont0201 commented 3 years ago

I can report the same behaviour here.

Single GPU runs well. gpus>1 fail at training stage with ninja warnings & errors, even after torch_extension cache cleanup and ninja reinstall.

Python 3.8 PyTorch 1.10 Cuda 11.4 Ninja 1.10.2, MS Visual Studio Community 2019 Gpu 0 3090 24GB, Gpu 1 3070 8GB

Same log as OP:

Command

python train.py --outdir=training-runs --data=C:\datasets\1024.zip --gpus=2 --snap=1  --resume=C:\models\stylegan3-t-ffhqu-1024x1024.pkl --metrics=none --cfg=stylegan3-t --batch=8 --gamma=8.6 --tick=1 --kimg=80 --seed 450

Log

Setting up augmentation...
Distributing across 2 GPUs...
Setting up training phases...
Exporting sample images...
Initializing logs...
Training for 80 kimg...

Error 1

RuntimeError: Error building extension 'bias_act_plugin': 
[1/2] cl /showIncludes -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\Users\jd\anaconda3\envs\stylegan3py38c\lib\site-packages\torch\include -IC:\Users\jd\anaconda3\envs\stylegan3py38c\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\jd\anaconda3\envs\stylegan3py38c\lib\site-packages\torch\include\TH -IC:\Users\jd\anaconda3\envs\stylegan3py38c\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\include" -IC:\Users\jd\anaconda3\envs\stylegan3py38c\Include -D_GLIBCXX_USE_CXX11_ABI=0 /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc -c C:\Users\jd\AppData\Local\torch_extensions\torch_extensions\Cache\py38_cu113\bias_act_plugin\3cb576a0039689487cfba59279dd6d46-nvidia-geforce-rtx-3090\bias_act.cpp /Fobias_act.o
Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30136 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

Error 2

[2/2] 
"C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\bin\Hostx64\x64/link.exe" bias_act.o bias_act.cuda.o /nologo /DLL c10.lib c10_cuda.lib torch_cpu.lib torch_cuda_cu.lib -INCLUDE:?searchsorted_cuda@native@at@@YA?AVTensor@2@AEBV32@0_N1@Z torch_cuda_cpp.lib -INCLUDE:?warp_size@cuda@at@@YAHXZ torch.lib /LIBPATH:C:\Users\jd\anaconda3\envs\stylegan3py38c\lib\site-packages\torch\lib torch_python.lib /LIBPATH:C:\Users\jd\anaconda3\envs\stylegan3py38c\libs "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\lib/x64" cudart.lib /out:bias_act_plugin.pyd
FAILED: bias_act_plugin.pyd
"C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\bin\Hostx64\x64/link.exe" bias_act.o bias_act.cuda.o /nologo /DLL c10.lib c10_cuda.lib torch_cpu.lib torch_cuda_cu.lib -INCLUDE:?searchsorted_cuda@native@at@@YA?AVTensor@2@AEBV32@0_N1@Z torch_cuda_cpp.lib -INCLUDE:?warp_size@cuda@at@@YAHXZ torch.lib /LIBPATH:C:\Users\jd\anaconda3\envs\stylegan3py38c\lib\site-packages\torch\lib torch_python.lib /LIBPATH:C:\Users\jd\anaconda3\envs\stylegan3py38c\libs "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\lib/x64" cudart.lib /out:bias_act_plugin.pyd
LINK : fatal error LNK1104: cannot open file 'bias_act_plugin.pyd'
ninja: build stopped: subcommand failed.
MLTQ commented 2 years ago

I'm running into the same error, which occurs just after this note: Setting up PyTorch plugin "bias_act_plugin"... then it spits out many many copies of this error duplicated on multiple files: C:\Python39\lib\site-packages\torch\include\c10/cuda/CUDAGuard.h(292): note: see reference to class template instantiation 'c10::impl::InlineMultiStreamGuard<c10::cuda::impl::CUDAGuardImpl>' being compiled C:\Python39\lib\site-packages\torch\include\c10/util/Optional.h(418): warning C4624: 'c10::trivially_copyable_optimization_optional_base<T>': destructor was implicitly defined as deleted with [ T=c10::cuda::impl::CUDAGuardImpl ] ninja: build stopped: subcommand failed.

I've actually tried this on both win10 and ubuntu 21.10, with the same error occuring on both (and 20.04 via WSL). On windows I'm using the wsys2 trick mentioned in the quoted issue, and in ubuntu it is whatever gcc is baked in. Next time I boot into ubuntu I'll update this with what versions are there. All attempts are with a single GPU, and I have tried nuking the torch extensions folder. Command to replicate: python train.py --outdir ./training_out --cfg=stylegan3-r --data=./datasets/s512.zip --gpus=1 --batch=32 --gamma=6.6

nurpax commented 2 years ago

@MLTQ are you sure that’s the specific error that breaks the build? It looks like a note and a warning.

MLTQ commented 2 years ago

@nurpax yes unfortunately it stops the build

MLTQ commented 2 years ago

Hi, just wanted to update this for anyone that finds it in the future: Reinstalling nvidia drivers and CUDA 11.3 solved the issue!

domef commented 2 years ago

I'm training the model in colab and I'm having problems too. The training gets stucked after tick 0.

takeraparterer commented 1 year ago

I'm training the model in colab and I'm having problems too. The training gets stucked after tick 0.

happens to me too running on windows, it just stops