Only quick96 works is it normal?

JTzLinux commented 1 year ago

Expected behavior

Should Train the DeepFake

Actual behavior

When I try to start it (AMP, AMP SRC, SAEHD) and enter all values this is the end result

Steps to reproduce

I tried the DX12 Version same result

Other relevant information

Windows 10 Pro 10.0.19045 Build 19045
3.10.7 OUTPUT: Initializing models: 80%|##################################################4 | 4/5 [00:18<00:04, 4.54s/it] Error: OOM when allocating tensor of shape [5,5,256,512] and type float [[node src_dst_opt/ms_encoder/down4/conv1/weight_0/Initializer/Const (defined at C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py:37) ]] Caused by op 'src_dst_opt/ms_encoder/down4/conv1/weight_0/Initializer/Const', defined at: File "threading.py", line 884, in _bootstrap File "threading.py", line 916, in _bootstrap_inner File "threading.py", line 864, in run File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\mainscripts\Trainer.py", line 58, in trainerThread debug=debug) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\models\ModelBase.py", line 193, in init self.on_initialize() File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\models\Model_AMP\Model.py", line 304, in on_initialize self.src_dst_opt.initialize_variables (self.G_weights, vars_on_cpu=optimizer_vars_on_cpu) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py", line 37, in initialize_variables ms = { v.name : tf.getvariable ( f'ms{v.name}'.replace(':','_'), v.shape, dtype=v.dtype, initializer=tf.initializers.constant(0.0), trainable=False) for v in trainable_weights } File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py", line 37, in ms = { v.name : tf.getvariable ( f'ms{v.name}'.replace(':','_'), v.shape, dtype=v.dtype, initializer=tf.initializers.constant(0.0), trainable=False) for v in trainable_weights } File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1479, in get_variable aggregation=aggregation) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1220, in get_variable aggregation=aggregation) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 547, in get_variable aggregation=aggregation) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 499, in _true_getter aggregation=aggregation) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 911, in _get_single_variable aggregation=aggregation) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 213, in call return cls._variable_v1_call(*args, kwargs) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 176, in _variable_v1_call aggregation=aggregation) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 155, in previous_getter = lambda kwargs: default_variable_creator(None, kwargs) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 2495, in default_variable_creator expected_shape=expected_shape, import_scope=import_scope) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 217, in call return super(VariableMetaclass, cls).call(*args, *kwargs) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 1395, in init constraint=constraint) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 1503, in _init_from_args initial_value(), name="initial_value", dtype=dtype) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 883, in shape.as_list(), dtype=dtype, partition_info=partition_info) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\init_ops.py", line 230, in call self.value, dtype=dtype, shape=shape, verify_shape=verify_shape) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\constant_op.py", line 179, in constant_v1 allow_broadcast=False) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\constant_op.py", line 289, in _constant_impl name=name).outputs[0] File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func return func(args, kwargs) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 3300, in create_op op_def=op_def) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 1801, in init self._traceback = tf_stack.extract_stack() ResourceExhaustedError (see above for traceback): OOM when allocating tensor of shape [5,5,256,512] and type float [[node src_dst_opt/ms_encoder/down4/conv1/weight_0/Initializer/Const (defined at C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py:37) ]] Traceback (most recent call last): File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call return fn(*args) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [5,5,256,512] and type float [[{{node src_dst_opt/ms_encoder/down4/conv1/weight_0/Initializer/Const}}]] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\mainscripts\Trainer.py", line 58, in trainerThread debug=debug) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\models\ModelBase.py", line 193, in init self.on_initialize() File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\models\Model_AMP\Model.py", line 545, in on_initialize model.init_weights() File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\layers\Saveable.py", line 106, in init_weights nn.init_weights(self.get_weights()) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\ops__init.py", line 48, in init_weights nn.tf_sess.run (ops) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 929, in run run_metadata_ptr) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run run_metadata) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [5,5,256,512] and type float [[node src_dst_opt/ms_encoder/down4/conv1/weight_0/Initializer/Const (defined at C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py:37) ]] Caused by op 'src_dst_opt/ms_encoder/down4/conv1/weight_0/Initializer/Const', defined at: File "threading.py", line 884, in _bootstrap File "threading.py", line 916, in _bootstrap_inner File "threading.py", line 864, in run File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\mainscripts\Trainer.py", line 58, in trainerThread debug=debug) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\models\ModelBase.py", line 193, in init self.on_initialize() File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\models\Model_AMP\Model.py", line 304, in on_initialize self.src_dst_opt.initialize_variables (self.G_weights, vars_on_cpu=optimizer_vars_on_cpu) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py", line 37, in initialize_variables ms = { v.name : tf.getvariable ( f'ms{v.name}'.replace(':','_'), v.shape, dtype=v.dtype, initializer=tf.initializers.constant(0.0), trainable=False) for v in trainable_weights } File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py", line 37, in ms = { v.name : tf.getvariable ( f'ms{v.name}'.replace(':','_'), v.shape, dtype=v.dtype, initializer=tf.initializers.constant(0.0), trainable=False) for v in trainable_weights } File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1479, in get_variable aggregation=aggregation) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1220, in get_variable aggregation=aggregation) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 547, in get_variable aggregation=aggregation) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 499, in _true_getter aggregation=aggregation) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 911, in _get_single_variable aggregation=aggregation) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 213, in call return cls._variable_v1_call(*args, kwargs) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 176, in _variable_v1_call aggregation=aggregation) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 155, in previous_getter = lambda kwargs: default_variable_creator(None, **kwargs) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 2495, in default_variable_creator expected_shape=expected_shape, import_scope=import_scope) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 217, in call return super(VariableMetaclass, cls).call(*args, **kwargs) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 1395, in init constraint=constraint) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 1503, in _init_from_args initial_value(), name="initial_value", dtype=dtype) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 883, in shape.as_list(), dtype=dtype, partition_info=partition_info) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\init_ops.py", line 230, in call self.value, dtype=dtype, shape=shape, verify_shape=verify_shape) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\constant_op.py", line 179, in constant_v1 allow_broadcast=False) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\constant_op.py", line 289, in _constant_impl name=name).outputs[0] File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func return func(*args, **kwargs) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 3300, in create_op op_def=op_def) File "C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 1801, in init__ self._traceback = tf_stack.extract_stack() ResourceExhaustedError (see above for traceback): OOM when allocating tensor of shape [5,5,256,512] and type float [[node src_dst_opt/ms_encoder/down4/conv1/weight_0/Initializer/Const (defined at C:\Users\admin\Desktop\DFP\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py:37) ]]

WesleyNery commented 1 year ago

That is an OOM. You're using a model that is too much for your card to handle. Get a lower-resolution one and/or turn off some things like adabelief.

JTzLinux commented 1 year ago

Ok will try

JTzLinux commented 1 year ago

but nope didn't work I tried it and well the same problem occurs the GPU/CPU was only had 90% and maybe the ram 50% :/

JTzLinux commented 1 year ago

never mind I just didn't turn down enough thanks!

iperov / DeepFaceLab