search

leichtrhino / ChimeraNet

Unofficial implementation of music separation model by Luo et.al.
MIT License
13 stars 2 forks source link

Topological sorting of the Graph #11

Closed prashant45 closed 5 years ago

prashant45 commented 5 years ago

When running the training, before the initial epoch I get a log like:

2019-08-02 08:32:55.251530: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 0, topological sort failed with message: The graph couldn't be sorted in topological order.
2019-08-02 08:32:55.305719: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 1, topological sort failed with message: The graph couldn't be sorted in topological order.
2019-08-02 08:32:56.161844: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 0, topological sort failed with message: The graph couldn't be sorted in topological order.
2019-08-02 08:32:56.213568: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 1, topological sort failed with message: The graph couldn't be sorted in topological order.

Although, I don't have any problem with the final model accuracy. I think, this is related to the fact that there are multiple heads in the Functional definition of the model. Do you also get the same when you run the script? Which Keras/tensorflow version do you use?

The versions, I am using are:

leichtrhino commented 5 years ago

I run a training script on following environment:

and OOM error occured but no error about topological sort.

The model's graph does not contain any loop so error about topological sort should not occur. Reducing batch size or disabling gpu might suppress the error.