error in traning model with metal

[xyfl0203]

Hi,guys with edit ".dfl/DeepFaceLab/core/lears/nn.py" line 201 "suggest_all_gpu=True" we can choose metal when traning model but it still report an error : Choose one or several GPU idxs (separated by comma).

[CPU] : CPU [0] : METAL

[0] Which GPU indexes to choose? : 0

Metal device set to: Apple M1

systemMemory: 16.00 GB maxCacheSize: 5.33 GB

Initializing models: 0%| | 0/7 [00:00<?, ?it/s] Error: Cannot assign a device for operation encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (defined at /Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py:61) placed on device Device assignments active during op 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' creation: with tf.device(None): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py:1793> with tf.device(/GPU:0): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py:277> was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0]. Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_nameindex=-1 requested_devicename='/device:GPU:0' assigned_devicename='' resource_devicename='/device:GPU:0' supported_devicetypes=[CPU] possibledevices=[] Identity: GPU CPU Mul: GPU CPU AddV2: GPU CPU Sub: GPU CPU RandomUniform: GPU CPU Assign: CPU VariableV2: GPU CPU Const: GPU CPU

Colocation members, user-requested devices, and framework assigned devices, if any: encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/min (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/max (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (RandomUniform) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/sub (Sub) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/mul (Mul) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform (AddV2) encoder/down1/downs_0/conv1/weight (VariableV2) /device:GPU:0 encoder/down1/downs_0/conv1/weight/Assign (Assign) /device:GPU:0 encoder/down1/downs_0/conv1/weight/read (Identity) /device:GPU:0 Assign_2 (Assign) /device:GPU:0 Assign_210 (Assign) /device:GPU:0

 [[node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform

(defined at /Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py:61) ]]

Errors may have originated from an input operation. Input Source operations connected to node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: In[0] encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape:

Operation defined at: (most recent call last)

File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 930, in _bootstrap self._bootstrap_inner()

File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run()

File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 910, in run self._target(*self._args, **self._kwargs)

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)(

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize()

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights()

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights self.build()

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name)

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build()

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name)

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub self._buildsub(sublayer, f"{name}{i}")

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build()

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name)

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub layer.build_weights()

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable ) Additional information about colocations:No node-device colocations were active during op 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' creation. Device assignments active during op 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' creation: with tf.device(None): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py:1793> with tf.device(/GPU:0): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py:277>

Original stack trace for 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform': File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 930, in _bootstrap self._bootstrap_inner() File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run() File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 910, in run self._target(*self._args, self._kwargs) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights self.build() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub self._buildsub(sublayer, f"{name}{i}") File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub layer.build_weights() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable ) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1579, in get_variable return get_variable_scope().get_variable( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1322, in get_variable return var_store.get_variable( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 578, in get_variable return _true_getter( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 531, in _true_getter return self._get_single_variable( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 952, in _get_single_variable v = variables.VariableV1( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler return fn(*args, *kwargs) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 268, in call return cls._variable_v1_call(args, kwargs) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 213, in _variable_v1_call return previous_getter( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 206, in previous_getter = lambda kwargs: default_variable_creator(None, kwargs) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 2628, in default_variable_creator return variables.RefVariable( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler return fn(*args, kwargs) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 272, in call return super(VariableMetaclass, cls).call(*args, *kwargs) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 1651, in init self._init_from_args( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 1794, in _init_from_args initial_value = initial_value() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/init_ops.py", line 832, in call return random_ops.random_uniform( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler return fn(args, kwargs) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/util/dispatch.py", line 1096, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/random_ops.py", line 311, in random_uniform result = gen_random_ops.random_uniform( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/gen_random_ops.py", line 737, in randomuniform , _, _op, _outputs = _op_def_library._apply_op_helper( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/framework/op_def_library.py", line 744, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 3697, in _create_op_internal ret = Operation( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 2101, in init self._traceback = tf_stack.extract_stack_for_node(self._c_op)

Traceback (most recent call last): File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1380, in _do_call return fn(*args) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1362, in _run_fn self._extend_graph() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1403, in _extend_graph tf_session.ExtendSession(self._session) tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node {{colocation_node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}} was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0]. Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_nameindex=-1 requested_devicename='/device:GPU:0' assigned_devicename='' resource_devicename='/device:GPU:0' supported_devicetypes=[CPU] possibledevices=[] Identity: GPU CPU Mul: GPU CPU AddV2: GPU CPU Sub: GPU CPU RandomUniform: GPU CPU Assign: CPU VariableV2: GPU CPU Const: GPU CPU

Colocation members, user-requested devices, and framework assigned devices, if any: encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/min (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/max (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (RandomUniform) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/sub (Sub) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/mul (Mul) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform (AddV2) encoder/down1/downs_0/conv1/weight (VariableV2) /device:GPU:0 encoder/down1/downs_0/conv1/weight/Assign (Assign) /device:GPU:0 encoder/down1/downs_0/conv1/weight/read (Identity) /device:GPU:0 Assign_2 (Assign) /device:GPU:0 Assign_210 (Assign) /device:GPU:0

 [[{{node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 657, in on_initialize model.init_weights() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Saveable.py", line 106, in init_weights nn.init_weights(self.get_weights()) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/ops/init.py", line 48, in init_weights nn.tf_sess.run (ops) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 970, in run result = self._run(None, fetches, feed_dict, options_ptr, File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1193, in _run results = self._do_run(handle, final_targets, final_fetches, File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1373, in _do_run return self._do_call(_run_fn, feeds, fetches, targets, options, File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1399, in _do_call raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (defined at /Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py:61) placed on device Device assignments active during op 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' creation: with tf.device(None): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py:1793> with tf.device(/GPU:0): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py:277> was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0]. Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_nameindex=-1 requested_devicename='/device:GPU:0' assigned_devicename='' resource_devicename='/device:GPU:0' supported_devicetypes=[CPU] possibledevices=[] Identity: GPU CPU Mul: GPU CPU AddV2: GPU CPU Sub: GPU CPU RandomUniform: GPU CPU Assign: CPU VariableV2: GPU CPU Const: GPU CPU

Colocation members, user-requested devices, and framework assigned devices, if any: encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/min (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/max (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (RandomUniform) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/sub (Sub) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/mul (Mul) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform (AddV2) encoder/down1/downs_0/conv1/weight (VariableV2) /device:GPU:0 encoder/down1/downs_0/conv1/weight/Assign (Assign) /device:GPU:0 encoder/down1/downs_0/conv1/weight/read (Identity) /device:GPU:0 Assign_2 (Assign) /device:GPU:0 Assign_210 (Assign) /device:GPU:0

 [[node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform

(defined at /Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py:61) ]]

Errors may have originated from an input operation. Input Source operations connected to node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: In[0] encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape:

Operation defined at: (most recent call last)

File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 930, in _bootstrap self._bootstrap_inner()

File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run()

File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 910, in run self._target(*self._args, **self._kwargs)

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)(

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize()

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights()

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights self.build()

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name)

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build()

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name)

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub self._buildsub(sublayer, f"{name}{i}")

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build()

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name)

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub layer.build_weights()

File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable ) Additional information about colocations:No node-device colocations were active during op 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' creation. Device assignments active during op 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' creation: with tf.device(None): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py:1793> with tf.device(/GPU:0): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py:277>

Original stack trace for 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform': File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 930, in _bootstrap self._bootstrap_inner() File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run() File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 910, in run self._target(*self._args, self._kwargs) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights self.build() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub self._buildsub(sublayer, f"{name}{i}") File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub layer.build_weights() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable ) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1579, in get_variable return get_variable_scope().get_variable( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1322, in get_variable return var_store.get_variable( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 578, in get_variable return _true_getter( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 531, in _true_getter return self._get_single_variable( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 952, in _get_single_variable v = variables.VariableV1( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler return fn(*args, *kwargs) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 268, in call return cls._variable_v1_call(args, kwargs) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 213, in _variable_v1_call return previous_getter( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 206, in previous_getter = lambda kwargs: default_variable_creator(None, kwargs) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 2628, in default_variable_creator return variables.RefVariable( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler return fn(*args, kwargs) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 272, in call return super(VariableMetaclass, cls).call(*args, *kwargs) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 1651, in init self._init_from_args( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 1794, in _init_from_args initial_value = initial_value() File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/init_ops.py", line 832, in call return random_ops.random_uniform( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler return fn(args, kwargs) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/util/dispatch.py", line 1096, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/random_ops.py", line 311, in random_uniform result = gen_random_ops.random_uniform( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/gen_random_ops.py", line 737, in randomuniform , _, _op, _outputs = _op_def_library._apply_op_helper( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/framework/op_def_library.py", line 744, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 3697, in _create_op_internal ret = Operation( File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 2101, in init self._traceback = tf_stack.extract_stack_for_node(self._c_op)

[AntaresDanielJune]

Same problem

[cantonalex]

getting this error

`Initializing models: 0%| | 0/5 [00:00<?, ?it/s] Error: Graph execution error:

Detected at node 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' defined at (most recent call last): File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 930, in _bootstrap self._bootstrap_inner() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 910, in run self._target(*self._args, *self._kwargs) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)( File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights self.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub self._buildsub(sublayer, f"{name}{i}") File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub layer.build_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable ) Node: 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' Detected at node 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' defined at (most recent call last): File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 930, in _bootstrap self._bootstrap_inner() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 910, in run self._target(self._args, **self._kwargs) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)( File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights self.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub self._buildsub(sublayer, f"{name}{i}") File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub layer.build_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable ) Node: 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' Cannot assign a device for operation encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node {{colocation_node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}} was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0]. Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_nameindex=-1 requested_devicename='/device:GPU:0' assigned_devicename='' resource_devicename='/device:GPU:0' supported_devicetypes=[CPU] possibledevices=[] Identity: GPU CPU Mul: GPU CPU AddV2: GPU CPU Sub: GPU CPU RandomUniform: GPU CPU Assign: CPU VariableV2: GPU CPU Const: GPU CPU

Colocation members, user-requested devices, and framework assigned devices, if any: encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/min (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/max (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (RandomUniform) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/sub (Sub) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/mul (Mul) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform (AddV2) encoder/down1/downs_0/conv1/weight (VariableV2) /device:GPU:0 encoder/down1/downs_0/conv1/weight/Assign (Assign) /device:GPU:0 encoder/down1/downs_0/conv1/weight/read (Identity) /device:GPU:0 Assign_2 (Assign) /device:GPU:0

 [[{{node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}}]]

Original stack trace for 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform': File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 930, in _bootstrap self._bootstrap_inner() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 910, in run self._target(*self._args, self._kwargs) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)( File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights self.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub self._buildsub(sublayer, f"{name}{i}") File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub layer.build_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable ) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1617, in get_variable return get_variable_scope().get_variable( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1333, in get_variable return var_store.get_variable( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 589, in get_variable return _true_getter( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 542, in _true_getter return self._get_single_variable( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 963, in _get_single_variable v = variables.VariableV1( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler return fn(*args, *kwargs) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 265, in call return cls._variable_v1_call(args, kwargs) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 210, in _variable_v1_call return previous_getter( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 203, in previous_getter = lambda kwargs: default_variable_creator(None, kwargs) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 2723, in default_variable_creator return variables.RefVariable( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler return fn(*args, kwargs) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 269, in call return super(VariableMetaclass, cls).call(*args, *kwargs) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 1648, in init self._init_from_args( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 1791, in _init_from_args initial_value = initial_value() File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/init_ops.py", line 881, in call return random_ops.random_uniform( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler return fn(args, kwargs) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/util/dispatch.py", line 1082, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/random_ops.py", line 309, in random_uniform result = gen_random_ops.random_uniform( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/gen_random_ops.py", line 737, in randomuniform , _, _op, _outputs = _op_def_library._apply_op_helper( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/framework/op_def_library.py", line 740, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 3776, in _create_op_internal ret = Operation( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 2175, in init self._traceback = tf_stack.extract_stack_for_node(self._c_op)

Traceback (most recent call last): File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1377, in _do_call return fn(*args) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1359, in _run_fn self._extend_graph() File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1400, in _extend_graph tf_session.ExtendSession(self._session) tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node {{colocation_node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}} was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0]. Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_nameindex=-1 requested_devicename='/device:GPU:0' assigned_devicename='' resource_devicename='/device:GPU:0' supported_devicetypes=[CPU] possibledevices=[] Identity: GPU CPU Mul: GPU CPU AddV2: GPU CPU Sub: GPU CPU RandomUniform: GPU CPU Assign: CPU VariableV2: GPU CPU Const: GPU CPU

Colocation members, user-requested devices, and framework assigned devices, if any: encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/min (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/max (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (RandomUniform) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/sub (Sub) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/mul (Mul) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform (AddV2) encoder/down1/downs_0/conv1/weight (VariableV2) /device:GPU:0 encoder/down1/downs_0/conv1/weight/Assign (Assign) /device:GPU:0 encoder/down1/downs_0/conv1/weight/read (Identity) /device:GPU:0 Assign_2 (Assign) /device:GPU:0

 [[{{node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)( File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 657, in on_initialize model.init_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Saveable.py", line 106, in init_weights nn.init_weights(self.get_weights()) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/ops/init.py", line 48, in init_weights nn.tf_sess.run (ops) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 967, in run result = self._run(None, fetches, feed_dict, options_ptr, File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1190, in _run results = self._do_run(handle, final_targets, final_fetches, File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1370, in _do_run return self._do_call(_run_fn, feeds, fetches, targets, options, File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1396, in _do_call raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:

Detected at node 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' defined at (most recent call last): File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 930, in _bootstrap self._bootstrap_inner() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 910, in run self._target(*self._args, *self._kwargs) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)( File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights self.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub self._buildsub(sublayer, f"{name}{i}") File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub layer.build_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable ) Node: 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' Detected at node 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' defined at (most recent call last): File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 930, in _bootstrap self._bootstrap_inner() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 910, in run self._target(self._args, **self._kwargs) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)( File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights self.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub self._buildsub(sublayer, f"{name}{i}") File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub layer.build_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable ) Node: 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' Cannot assign a device for operation encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node {{colocation_node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}} was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0]. Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_nameindex=-1 requested_devicename='/device:GPU:0' assigned_devicename='' resource_devicename='/device:GPU:0' supported_devicetypes=[CPU] possibledevices=[] Identity: GPU CPU Mul: GPU CPU AddV2: GPU CPU Sub: GPU CPU RandomUniform: GPU CPU Assign: CPU VariableV2: GPU CPU Const: GPU CPU

Colocation members, user-requested devices, and framework assigned devices, if any: encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/min (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/max (Const) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (RandomUniform) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/sub (Sub) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/mul (Mul) encoder/down1/downs_0/conv1/weight/Initializer/random_uniform (AddV2) encoder/down1/downs_0/conv1/weight (VariableV2) /device:GPU:0 encoder/down1/downs_0/conv1/weight/Assign (Assign) /device:GPU:0 encoder/down1/downs_0/conv1/weight/read (Identity) /device:GPU:0 Assign_2 (Assign) /device:GPU:0

 [[{{node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}}]]

Original stack trace for 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform': File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 930, in _bootstrap self._bootstrap_inner() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run() File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 910, in run self._target(*self._args, self._kwargs) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread model = models.import_model(model_class_name)( File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init self.on_initialize() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights self.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub self._buildsub(sublayer, f"{name}{i}") File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub layer.build() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build self._build_sub(v[name],name) File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub layer.build_weights() File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable ) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1617, in get_variable return get_variable_scope().get_variable( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1333, in get_variable return var_store.get_variable( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 589, in get_variable return _true_getter( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 542, in _true_getter return self._get_single_variable( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 963, in _get_single_variable v = variables.VariableV1( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler return fn(*args, *kwargs) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 265, in call return cls._variable_v1_call(args, kwargs) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 210, in _variable_v1_call return previous_getter( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 203, in previous_getter = lambda kwargs: default_variable_creator(None, kwargs) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 2723, in default_variable_creator return variables.RefVariable( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler return fn(*args, kwargs) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 269, in call return super(VariableMetaclass, cls).call(*args, *kwargs) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 1648, in init self._init_from_args( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 1791, in _init_from_args initial_value = initial_value() File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/init_ops.py", line 881, in call return random_ops.random_uniform( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler return fn(args, kwargs) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/util/dispatch.py", line 1082, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/random_ops.py", line 309, in random_uniform result = gen_random_ops.random_uniform( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/gen_random_ops.py", line 737, in randomuniform , _, _op, _outputs = _op_def_library._apply_op_helper( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/framework/op_def_library.py", line 740, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 3776, in _create_op_internal ret = Operation( File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 2175, in init self._traceback = tf_stack.extract_stack_for_node(self._c_op)

`

[cantonalex]

Wondering if this is our answer https://developer.apple.com/forums/thread/700169?answerId=704851022#704851022

[Smiril]

Well got some partial working copy DeepFaceLab_MacOS_M1

[prashantspandey]

@Smiril I am using your fork but cant train my model on GPU . It says tf.reduce_mean has some problem. Please help me run it on the GPU. Thanks

[Smiril]

@prashantspandey try Metal device

[prashantspandey]

@Smiril I am using M2 macbook pro. Are there any settings I need to change?

[Smiril]

@prashantspandey just test and report