2020-02-17 09:33:46.136926: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:561] remapper failed: Invalid argument: MutableGraphView::MutableGraphView error: node 'loss/time_distributed_1_loss/mean_squared_error/weighted_loss/concat' has self cycle fanin 'loss/time_distributed_1_loss/mean_squared_error/weighted_loss/concat'.

[shamoons]

My model is:

    model = Sequential()
    model.add(LSTM(32, input_shape=(
        SEQ_LENGTH, VECTOR_SIZE), return_sequences=True))
    model.add(TimeDistributed(Dense(VECTOR_SIZE, activation='relu')))

    adam_optimizer = optimizers.Adam(
        learning_rate=0.001, beta_1=0.9, beta_2=0.999, amsgrad=False)
    model.compile(loss='mean_squared_error', optimizer=adam_optimizer)

    print(model.summary())
    callbacks = [WandbCallback(), EarlyStopping(
        monitor='val_loss', patience=10)]

With:

TOTAL_SAMPLES = 2676
SEQ_LENGTH = 100
VECTOR_SIZE = 129
BATCH_SIZE = 1

And my DataGenerator is:

class DataGenerator(Sequence):
    def __init__(self, corrupted_path, seq_length=10, batch_size=20, train_set=False, test_set=False):
        corrupted_base_path = path.abspath(corrupted_path)
        corrupted_base_path_parts = corrupted_base_path.split('/')
        clean_base_path = corrupted_base_path_parts.copy()
        clean_base_path[-1] = 'dev-clean'
        clean_base_path = '/'.join(clean_base_path)

        corrupted_audio_file_paths = list(
            sorted(glob.iglob(corrupted_base_path + '/**/*.flac', recursive=True)))

        clean_audio_file_paths = list(
            sorted(glob.iglob(clean_base_path + '/**/*.flac', recursive=True)))

        cutoff_index = int(len(corrupted_audio_file_paths) * 0.9)

        if train_set == True:
            self.clean_file_paths = clean_audio_file_paths[0: cutoff_index]
            self.corrupted_file_paths = corrupted_audio_file_paths[0: cutoff_index]
        if test_set == True:
            self.clean_file_paths = clean_audio_file_paths[cutoff_index:]
            self.corrupted_file_paths = corrupted_audio_file_paths[cutoff_index:]

        self.seq_length = seq_length
        self.batch_size = batch_size
        return

    def __len__(self):
        return math.ceil(len(self.clean_file_paths) / self.batch_size)

    def __getitem__(self, index):
        batch_index = index * self.batch_size

        corrupted_samples, corrupted_sample_rate = sf.read(
            self.corrupted_file_paths[batch_index])

        clean_samples, clean_sample_rate = sf.read(
            self.clean_file_paths[batch_index])

        _, _, corrupted_spectrogram = signal.spectrogram(
            corrupted_samples, corrupted_sample_rate)

        _, _, clean_spectrogram = signal.spectrogram(
            clean_samples, clean_sample_rate)

        # By default, the first axis is frequencies and the second is time.
        # We swap them here.
        input_spectrogram = np.swapaxes(corrupted_spectrogram, 0, 1)
        output_spectrogram = np.swapaxes(clean_spectrogram, 0, 1) * 1234

        inputs = []
        outputs = []

        while len(inputs) < self.batch_size:
            input_sliced = self.__get_random_slice(
                input_spectrogram, self.seq_length)

            output_sliced = self.__get_random_slice(
                output_spectrogram, self.seq_length)

            inputs.append(input_sliced)
            outputs.append(output_sliced)

        return np.array(inputs), np.array(outputs)

    def __get_random_slice(self, data, slice_length):
        start_index = random.randint(0, len(data) - slice_length)
        end_index = start_index + slice_length

        return data[start_index:end_index]

In between epochs, I get the following error:

2674/2676 [============================>.] - ETA: 0s - loss: 2.1575e-042020-02-17 09:35:18.266485: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:561] remapper failed: Invalid argument: MutableGraphView::MutableGraphView error: node 'loss/time_distributed_1_loss/mean_squared_error/weighted_loss/concat' has self cycle fanin 'loss/time_distributed_1_loss/mean_squared_error/weighted_loss/concat'.
2020-02-17 09:35:18.267798: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:717] Iteration = 0, topological sort failed with message: The graph couldn't be sorted in topological order.
2020-02-17 09:35:18.268390: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:717] Iteration = 1, topological sort failed with message: The graph couldn't be sorted in topological order.
2020-02-17 09:35:18.273071: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:561] arithmetic_optimizer failed: Invalid argument: The graph couldn't be sorted in topological order.
2020-02-17 09:35:18.273493: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:561] remapper failed: Invalid argument: MutableGraphView::MutableGraphView error: node 'loss/time_distributed_1_loss/mean_squared_error/weighted_loss/concat' has self cycle fanin 'loss/time_distributed_1_loss/mean_squared_error/weighted_loss/concat'.
2020-02-17 09:35:18.273917: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:717] Iteration = 0, topological sort failed with message: The graph couldn't be sorted in topological order.
2020-02-17 09:35:18.274322: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:717] Iteration = 1, topological sort failed with message: The graph couldn't be sorted in topological order.
2676/2676 [==============================] - 96s 36ms/step - loss: 2.1577e-04 - val_loss: 1.7582e-04

[shamoons]

For now, yes, I am trying to learn a simple mapping since corrupted_spectrogram and clean_spectogram are the same. So I'm making the output just a multiplication of the input

[Ziad-Mohamedd]

i have the same issue, We need a solution

[sushreebarsa]

@shamoons Moving this issue to closed status as there has been no recent activity.In case you still face the error please create a new issue,we will get you the right help.Thanks!