search

deepfakes / faceswap-playground

User dedicated repo for the faceswap project
306 stars 194 forks source link

Tensorflow warning with dfaker plugin #271

Closed Kirin-kun closed 5 years ago

Kirin-kun commented 5 years ago

I'm trying the various plugins and with dfaker, on the training start, it gives a few warnings (errors?) , of the type:

2019-03-30 08:54:48.861438: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 0, topological sort failed with message: The graph couldn't be sorted in topological order.
2019-03-30 08:54:48.881440: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 1, topological sort failed with message: The graph couldn't be sorted in topological order.

The training still starts, so I don't know if there will be any consequences.

Can it be ignored or is it a concerning error?

torzdf commented 5 years ago

This is an annoying error which has crept in when using masks. We're trying to find what causes it, but it's proving tricky.

It doesn't seem to impact training

torzdf commented 5 years ago

I have traced this issue back to the Keras implementation of the Penalized Loss Function. I have now reverted that commit back to the pure Tensorflow implementation.

Unfortunately this issue seems to persist in TF even after reverting, so I will next look into how to get TF back to a state where this doesn't occur :/

torzdf commented 5 years ago

Ok. If you still have issues after updating to the latest version, go into your Nvidia ComputeCache folder: ~/.nv/ComputeCache (Linux) C:\users\<USERNAME>\AppData\NVIDIA\ComputeCache (windows)

and delete the contents.

Kirin-kun commented 5 years ago

Warning went away I think.