Closed ajohnson49 closed 1 year ago
I might have figured out a really inefficient way of doing it using the modules called during training. If anyone has a better way, lemme know:
one_ex = {'audio': 'datasets/LJSpeech-1.1/wavs/LJ007-0224.wav', 'text': 'watever the text is'} two_ex = {'audio': 'datasets/LJSpeech-1.1/wavs/LJ029-0032.wav', 'text': 'watever that text is'} three_ex = {'audio': 'datasets/LJSpeech-1.1/wavs/LJ040-0007.wav', 'text': 'watever this text is'}
eval_data = [one_ex, two_ex, three_ex]
eval_ds = create_tf_dataset(eval_data, bs=3)
for ex in eval_ds: source =ex['source'] target = ex['target'] print(ex['target']) break
target_start_token_idx=2 target_end_token_idx=3
bs = tf.shape(source)[0] preds = model.generate(source, target_start_token_idx) preds = preds.numpy() for i in range(bs): target_text = "".join([idx_tochar[] for _ in target[i, :]]) prediction = "" for idx in preds[i, :]: prediction +=idx_to_char[idx] if idx == target_end_token_idx: break print(f"target: {target_text.replace('-','')}") print(f"prediction: {prediction}\n")
@ajohnson49 yes, generate works like that, let me ask you a question. Does the prediction works with new data? bc it seems to have awful predictions with data that is not from the training audios, how was your experience with that?
Yeah, it worked for me in the end. The dataset included in the example is tiny, so it doesn't generalize well. I pointed the training towards the 100hr set in LibriSpeech, so it does much better for outside data now.
@ajohnson49 thank you, so the key to get better results with outside data is to use other dataset, and other question. Why 12000 for training give small results? Shouldn't they be enough?
I haven't looked at this code in months, but if I remember correctly, it's using around 25 hours of audio data for training. A lot of the standard ASR systems use the 1000 hour LibriSpeech dataset. I think you need at least 100 hours of audio data to start approaching good ASR results for general applications, so 25 hours isn't even close. I think the dataset used in the example is just for show, and you need to use a much bigger dataset for good generalizable results.
@ajohnson49 hello again, I have another problem, I'm using the librispeech 500 hours for training, but the problem is that I'm getting loss: nan per epoch an predictions are --------- this happened to you too? I am using the same parameters as the keras docs
There's a lot of things that could cause that. There could be a nan in the input, the input could be improperly formatted, there could be a zero divide happening somewhere in the data preprocessing, there could be a vanishing or exploding gradient, there could could be an issue with the loss function or hyper parameters used, etc.. You'd have to do some debugging to find the issue. I would look at the output of each step in the code to see the first place the nans come up.
There's a lot of things that could cause that. There could be a nan in the input, the input could be improperly formatted, there could be a zero divide happening somewhere in the data preprocessing, there could be a vanishing or exploding gradient, there could could be an issue with the loss function or hyper parameters used, etc.. You'd have to do some debugging to find the issue. I would look at the output of each step in the code to see the first place the nans come up.
Look this is the notebook with the whole code I'm using https://www.kaggle.com/code/bernardoolisan/speechrecognition-dot If you run it, it will say loss: nan and the predictions after epoch 1 will be <------>
I don't see where you're using LibriSpeech. It looks like you're using the same LJ Speech dataset. Different datasets might be set up differently and need to be preprocessed differently, so there may be something incompatible about the way it's being read in. It's hard to tell without looking at the input files, though. You may just want to print out the output of each step to see where the nans occur.
@ajohnson49 look https://www.kaggle.com/code/bernardoolisan/speechrecognition-dot is not LJ Speech dataset, is librispeech.
@ajohnson49 maybe, can you share me your parameters of you code? or you don't have it anymore, and how can i see the output of steps?
I say that it looks like you're still using the LJ Speech dataset because that's the way the code is formatted. For Librispeech, you would need to read in the transcript from the trans.txt file for each .flac file in order to get the transcript which I didn't see in your code. I just did a ctrl+F search, so I might have missed something. But in general, you would need something like:
my_train_dataset = glob('../path/to/Librispeech/train-clean-100///*.flac')
my_train_list = [] for flac in my_train_dataset: trans_file_path=os.path.dirname(file) #get dir name trans_file=glob(trans_file_path+'*trans.txt')[0] #get transcript file name trans_file_open=open(trans_file) trans_file_lines=trans_file_open.readlines() flac_num = flac.split('/')[-1].replace('flac','') #get corresponding transcript line get_text = [line for line in trans_file_lines if flac_num in line] #get transcript dictionary ={'audio': flac , 'text': get_text[0]} #create dictionary my_train_list.append(dictionary)
ds_train = create_tf_dataset(my_train_list, bs=64) #create tf dataset object
and you can just print the output of any step in the code
I say that it looks like you're still using the LJ Speech dataset because that's the way the code is formatted. For Librispeech, you would need to read in the transcript from the trans.txt file for each .flac file in order to get the transcript which I didn't see in your code. I just did a ctrl+F search, so I might have missed something. But in general, you would need something like:
my_train_dataset = glob('../path/to/Librispeech/train-clean-100///*.flac')
my_train_list = []
for flac in my_train_dataset:
trans_file_path=os.path.dirname(file) #get dir name trans_file=glob(trans_file_path+'*trans.txt')[0] #get transcript file name trans_file_open=open(trans_file) trans_file_lines=trans_file_open.readlines() flac_num = flac.split('/')[-1].replace('flac','') #get corresponding transcript line get_text = [line for line in trans_file_lines if flac_num in line] #get transcript dictionary ={'audio': flac , 'text': get_text[0]} #create dictionary my_train_list.append(dictionary)
ds_train = create_tf_dataset(my_train_list, bs=64) #create tf dataset object
and you can just print the output of any step in the code
Yes, it is actually that, the only thing that change is that instead of .flac files they are .wav files but is the same trans.txt
@ajohnson49 what was you learning rate with the librispeech? did you change some params when training with librispeech?
I don't think that I changed anything else in the code. You might benefit from optimizing the hyper parameters, though.
@ajohnson49 I think the problem is normalizing my data, did you normalized your data "librispeech" or just doing it normal? s there a way you can share me your code or sth? it would be nice to see it
Not sure what you mean by normalize. You can make the range of the inputs -1 to 1 if you want. That can help reduce the exploding gradient problem in some cases. Not sure if it makes a difference here, though, since the code already has batch normalization. I don't have my original code, so I just implemented this example quickly:
import re import os import random from glob import glob import tensorflow as tf from tensorflow import keras from tensorflow.keras import layers os.environ["CUDA_VISIBLE_DEVICES"]="1"
class TokenEmbedding(layers.Layer): def init(self, num_vocab=1000, maxlen=100, num_hid=64): super().init() self.emb = tf.keras.layers.Embedding(num_vocab, num_hid) self.pos_emb = layers.Embedding(input_dim=maxlen, output_dim=num_hid)
def call(self, x):
maxlen = tf.shape(x)[-1]
x = self.emb(x)
positions = tf.range(start=0, limit=maxlen, delta=1)
positions = self.pos_emb(positions)
return x + positions
class SpeechFeatureEmbedding(layers.Layer): def init(self, num_hid=64, maxlen=100): super().init() self.conv1 = tf.keras.layers.Conv1D( num_hid, 11, strides=2, padding="same", activation="relu" ) self.conv2 = tf.keras.layers.Conv1D( num_hid, 11, strides=2, padding="same", activation="relu" ) self.conv3 = tf.keras.layers.Conv1D( num_hid, 11, strides=2, padding="same", activation="relu" ) self.pos_emb = layers.Embedding(input_dim=maxlen, output_dim=num_hid)
def call(self, x):
x = self.conv1(x)
x = self.conv2(x)
return self.conv3(x)
class TransformerEncoder(layers.Layer): def init(self, embed_dim, num_heads, feed_forward_dim, rate=0.1): super().init() self.att = layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim) self.ffn = keras.Sequential( [ layers.Dense(feed_forward_dim, activation="relu"), layers.Dense(embed_dim), ] ) self.layernorm1 = layers.LayerNormalization(epsilon=1e-6) self.layernorm2 = layers.LayerNormalization(epsilon=1e-6) self.dropout1 = layers.Dropout(rate) self.dropout2 = layers.Dropout(rate)
def call(self, inputs, training):
attn_output = self.att(inputs, inputs)
attn_output = self.dropout1(attn_output, training=training)
out1 = self.layernorm1(inputs + attn_output)
ffn_output = self.ffn(out1)
ffn_output = self.dropout2(ffn_output, training=training)
return self.layernorm2(out1 + ffn_output)
class TransformerDecoder(layers.Layer): def init(self, embed_dim, num_heads, feed_forward_dim, dropout_rate=0.1): super().init() self.layernorm1 = layers.LayerNormalization(epsilon=1e-6) self.layernorm2 = layers.LayerNormalization(epsilon=1e-6) self.layernorm3 = layers.LayerNormalization(epsilon=1e-6) self.self_att = layers.MultiHeadAttention( num_heads=num_heads, key_dim=embed_dim ) self.enc_att = layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim) self.self_dropout = layers.Dropout(0.5) self.enc_dropout = layers.Dropout(0.1) self.ffn_dropout = layers.Dropout(0.1) self.ffn = keras.Sequential( [ layers.Dense(feed_forward_dim, activation="relu"), layers.Dense(embed_dim), ] )
def causal_attention_mask(self, batch_size, n_dest, n_src, dtype):
"""Masks the upper half of the dot product matrix in self attention.
This prevents flow of information from future tokens to current token.
1's in the lower triangle, counting from the lower right corner.
"""
i = tf.range(n_dest)[:, None]
j = tf.range(n_src)
m = i >= j - n_src + n_dest
mask = tf.cast(m, dtype)
mask = tf.reshape(mask, [1, n_dest, n_src])
mult = tf.concat(
[tf.expand_dims(batch_size, -1), tf.constant([1, 1], dtype=tf.int32)], 0
)
return tf.tile(mask, mult)
def call(self, enc_out, target):
input_shape = tf.shape(target)
batch_size = input_shape[0]
seq_len = input_shape[1]
causal_mask = self.causal_attention_mask(batch_size, seq_len, seq_len, tf.bool)
target_att = self.self_att(target, target, attention_mask=causal_mask)
target_norm = self.layernorm1(target + self.self_dropout(target_att))
enc_out = self.enc_att(target_norm, enc_out)
enc_out_norm = self.layernorm2(self.enc_dropout(enc_out) + target_norm)
ffn_out = self.ffn(enc_out_norm)
ffn_out_norm = self.layernorm3(enc_out_norm + self.ffn_dropout(ffn_out))
return ffn_out_norm
class Transformer(keras.Model): def init( self, num_hid=64, num_head=2, num_feed_forward=128, source_maxlen=100, target_maxlen=100, num_layers_enc=4, num_layers_dec=1, num_classes=10, ): super().init() self.loss_metric = keras.metrics.Mean(name="loss") self.num_layers_enc = num_layers_enc self.num_layers_dec = num_layers_dec self.target_maxlen = target_maxlen self.num_classes = num_classes
self.enc_input = SpeechFeatureEmbedding(num_hid=num_hid, maxlen=source_maxlen)
self.dec_input = TokenEmbedding(
num_vocab=num_classes, maxlen=target_maxlen, num_hid=num_hid
)
self.encoder = keras.Sequential(
[self.enc_input]
+ [
TransformerEncoder(num_hid, num_head, num_feed_forward)
for _ in range(num_layers_enc)
]
)
for i in range(num_layers_dec):
setattr(
self,
f"dec_layer_{i}",
TransformerDecoder(num_hid, num_head, num_feed_forward),
)
self.classifier = layers.Dense(num_classes)
def decode(self, enc_out, target):
y = self.dec_input(target)
for i in range(self.num_layers_dec):
y = getattr(self, f"dec_layer_{i}")(enc_out, y)
return y
def call(self, inputs):
source = inputs[0]
target = inputs[1]
x = self.encoder(source)
y = self.decode(x, target)
return self.classifier(y)
@property
def metrics(self):
return [self.loss_metric]
def train_step(self, batch):
"""Processes one batch inside model.fit()."""
source = batch["source"]
target = batch["target"]
dec_input = target[:, :-1]
dec_target = target[:, 1:]
with tf.GradientTape() as tape:
preds = self([source, dec_input])
one_hot = tf.one_hot(dec_target, depth=self.num_classes)
mask = tf.math.logical_not(tf.math.equal(dec_target, 0))
loss = self.compiled_loss(one_hot, preds, sample_weight=mask)
trainable_vars = self.trainable_variables
gradients = tape.gradient(loss, trainable_vars)
self.optimizer.apply_gradients(zip(gradients, trainable_vars))
self.loss_metric.update_state(loss)
return {"loss": self.loss_metric.result()}
def test_step(self, batch):
source = batch["source"]
target = batch["target"]
dec_input = target[:, :-1]
dec_target = target[:, 1:]
preds = self([source, dec_input])
one_hot = tf.one_hot(dec_target, depth=self.num_classes)
mask = tf.math.logical_not(tf.math.equal(dec_target, 0))
loss = self.compiled_loss(one_hot, preds, sample_weight=mask)
self.loss_metric.update_state(loss)
return {"loss": self.loss_metric.result()}
def generate(self, source, target_start_token_idx):
"""Performs inference over one batch of inputs using greedy decoding."""
bs = tf.shape(source)[0]
enc = self.encoder(source)
dec_input = tf.ones((bs, 1), dtype=tf.int32) * target_start_token_idx
dec_logits = []
for i in range(self.target_maxlen - 1):
dec_out = self.decode(enc, dec_input)
logits = self.classifier(dec_out)
logits = tf.argmax(logits, axis=-1, output_type=tf.int32)
last_logit = tf.expand_dims(logits[:, -1], axis=-1)
dec_logits.append(last_logit)
dec_input = tf.concat([dec_input, last_logit], axis=-1)
return dec_input
saveto = "../home/datasets/LJSpeech-1.1" wavs = glob("{}/*/.wav".format(saveto), recursive=True)
id_to_text = {} with open(os.path.join(saveto, "metadata.csv"), encoding="utf-8") as f: for line in f: id = line.strip().split("|")[0] text = line.strip().split("|")[2] id_to_text[id] = text
def get_data(wavs, id_to_text, maxlen=50): """ returns mapping of audio paths and transcription texts """ data = [] for w in wavs: id = w.split("/")[-1].split(".")[0] if len(id_to_text[id]) < maxlen: data.append({"audio": w, "text": id_to_text[id]}) return data
import numpy as np import tensorflow_io as tfio
class VectorizeChar: def init(self, max_len=50): self.vocab = ( ["-", "#", "<", ">"]
[" ", ".", ",", "?"] ) self.max_len = max_len self.char_to_idx = {} for i, ch in enumerate(self.vocab): self.char_to_idx[ch] = i
def call(self, text): text = text.lower() text = text[: self.max_len - 2] text = "<" + text + ">" pad_len = self.max_len - len(text) return [self.char_to_idx.get(ch, 1) for ch in text] + [0] * pad_len
def get_vocabulary(self): return self.vocab
max_target_len = 200 # all transcripts in out data are < 200 characters data = get_data(wavs, id_to_text, max_target_len) vectorizer = VectorizeChar(max_target_len) print("vocab size", len(vectorizer.get_vocabulary()))
def create_textds(data): texts = [["text"] for _ in data] text_ds = [vectorizer(t) for t in texts] text_ds = tf.data.Dataset.from_tensor_slices(text_ds) return text_ds
def path_to_audio(path):
audio = tf.io.read_file(path)
audio = tfio.audio.decode_flac(audio, dtype=tf.int16) #audio, _ = tf.audio.decode_wav(audio, 1)
audio = tf.cast(audio, tf.float32)
audio = tf.squeeze(audio, axis=-1)
stfts = tf.signal.stft(audio, frame_length=200, frame_step=80, fft_length=256)
x = tf.math.pow(tf.abs(stfts), 0.5)
# normalisation
means = tf.math.reduce_mean(x, 1, keepdims=True)
stddevs = tf.math.reduce_std(x, 1, keepdims=True)
x = (x - means) / stddevs
value_not_nan = tf.dtypes.cast(tf.math.logical_not(tf.math.is_nan(x)), dtype=tf.float32)
tf.math.multiply_no_nan(x, value_not_nan)
audio_len = tf.shape(x)[0]
# padding to 10 seconds
pad_len = 2754
paddings = tf.constant([[0, pad_len], [0, 0]])
x = tf.pad(x, paddings, "CONSTANT")[:pad_len, :]
return x
def create_audiods(data): flist = [["audio"] for _ in data] audio_ds = tf.data.Dataset.from_tensor_slices(flist) audio_ds = audio_ds.map( path_to_audio, num_parallel_calls=tf.data.experimental.AUTOTUNE ) return audio_ds
def create_tf_dataset(data, bs=4): audio_ds = create_audio_ds(data) text_ds = create_text_ds(data) ds = tf.data.Dataset.zip((audio_ds, text_ds)) ds = ds.map(lambda x, y: {"source": x, "target": y}) ds = ds.batch(bs) ds = ds.prefetch(tf.data.experimental.AUTOTUNE) return ds
my_train_dataset = glob('../../data/Databases/LibriSpeech/Librispeech/train-clean-100/103//.flac') my_val_dataset = glob('../../data/Databases/LibriSpeech/Librispeech/train-clean-100/8630//.flac') my_test_dataset=glob('../../data/Databases/LibriSpeech/Librispeech/train-clean-100/1502//.flac')
my_train_list = [] for flac in my_train_dataset: trans_file_path=os.path.dirname(flac) trans_file=glob(trans_file_path+'/*trans.txt')[0] trans_file_open=open(trans_file) trans_file_lines=trans_file_open.readlines() flac_num = flac.split('/')[-1].replace('.flac','') get_text = [line for line in trans_file_lines if flac_num in line] dictionary ={'audio': flac , 'text': get_text[0].replace(flac_num,'')} my_train_list.append(dictionary)
ds_train = create_tf_dataset(my_train_list, bs=64)
my_val_list = []
for flac in my_val_dataset: trans_file_path=os.path.dirname(flac) trans_file=glob(trans_file_path+'/*trans.txt')[0] trans_file_open=open(trans_file) trans_file_lines=trans_file_open.readlines() flac_num = flac.split('/')[-1].replace('.flac','') get_text = [line for line in trans_file_lines if flac_num in line] dictionary ={'audio': flac , 'text': get_text[0].replace(flac_num,'')} my_val_list.append(dictionary)
ds_val = create_tf_dataset(my_val_list, bs=5)
my_test_list = []
for flac in my_test_dataset: trans_file_path=os.path.dirname(flac) trans_file=glob(trans_file_path+'/*trans.txt')[0] trans_file_open=open(trans_file) trans_file_lines=trans_file_open.readlines() flac_num = flac.split('/')[-1].replace('.flac','') get_text = [line for line in trans_file_lines if flac_num in line] dictionary ={'audio': flac , 'text': get_text[0].replace(flac_num,'')} my_test_list.append(dictionary)
ds_test = create_tf_dataset(my_test_list, bs=32)
class DisplayOutputs(keras.callbacks.Callback): def init( self, batch, idx_to_token, target_start_token_idx=27, target_end_token_idx=28 ): """Displays a batch of outputs after every epoch
Args:
batch: A test batch containing the keys "source" and "target"
idx_to_token: A List containing the vocabulary tokens corresponding to their indices
target_start_token_idx: A start token index in the target vocabulary
target_end_token_idx: An end token index in the target vocabulary
"""
self.batch = batch
self.target_start_token_idx = target_start_token_idx
self.target_end_token_idx = target_end_token_idx
self.idx_to_char = idx_to_token
def on_epoch_end(self, epoch, logs=None):
if epoch % 5 != 0:
return
source = self.batch["source"]
target = self.batch["target"].numpy()
bs = tf.shape(source)[0]
preds = self.model.generate(source, self.target_start_token_idx)
preds = preds.numpy()
for i in range(bs):
target_text = "".join([self.idx_to_char[_] for _ in target[i, :]])
prediction = ""
for idx in preds[i, :]:
prediction += self.idx_to_char[idx]
if idx == self.target_end_token_idx:
break
print(f"target: {target_text.replace('-','')}")
print(f"prediction: {prediction}\n")
class CustomSchedule(keras.optimizers.schedules.LearningRateSchedule): def init( self, init_lr=0.00001, lr_after_warmup=0.001, final_lr=0.00001, warmup_epochs=15, decay_epochs=85, steps_per_epoch=203, ): super().init() self.init_lr = init_lr self.lr_after_warmup = lr_after_warmup self.final_lr = final_lr self.warmup_epochs = warmup_epochs self.decay_epochs = decay_epochs self.steps_per_epoch = steps_per_epoch
def calculate_lr(self, epoch):
""" linear warm up - linear decay """
warmup_lr = (
self.init_lr
+ ((self.lr_after_warmup - self.init_lr) / (self.warmup_epochs - 1)) * epoch
)
decay_lr = tf.math.maximum(
self.final_lr,
self.lr_after_warmup
- (epoch - self.warmup_epochs)
* (self.lr_after_warmup - self.final_lr)
/ (self.decay_epochs),
)
return tf.math.minimum(warmup_lr, decay_lr)
def __call__(self, step):
epoch = step // self.steps_per_epoch
return self.calculate_lr(epoch)
batch = next(iter(ds_val))
idx_to_char = vectorizer.get_vocabulary() display_cb = DisplayOutputs( batch, idx_to_char, target_start_token_idx=2, target_end_token_idx=3 ) # set the arguments as per vocabulary index for '<' and '>'
model = Transformer( num_hid=200, num_head=2, num_feed_forward=400, target_maxlen=max_target_len, num_layers_enc=4, num_layers_dec=1, num_classes=34, ) loss_fn = tf.keras.losses.CategoricalCrossentropy( from_logits=True, label_smoothing=0.1, )
learning_rate = CustomSchedule( init_lr=0.00001, lr_after_warmup=0.001, final_lr=0.00001, warmup_epochs=15, decay_epochs=85, steps_per_epoch=len(ds_train), ) optimizer = keras.optimizers.SGD(learning_rate=learning_rate) model.compile(optimizer=optimizer, loss=loss_fn)
history = model.fit(ds_train, validation_data=ds_val, callbacks=[display_cb], epochs=100)
model.summary()
from glob import glob import soundfile as sf import numpy as np import pandas as pd
def get_eval(eval_ds): for ex in eval_ds: source =ex['source'] target = ex['target'] break
target_start_token_idx=2
target_end_token_idx=3
bs = tf.shape(source)[0]
preds = model.generate(source, target_start_token_idx)
preds = preds.numpy()
for i in range(bs):
target_text = "".join([idx_to_char[_] for _ in target[i, :]])
prediction = ""
for idx in preds[i, :]:
prediction +=idx_to_char[idx]
if idx == target_end_token_idx:
break
print(f"target: {target_text.replace('-','')}")
print(f"prediction: {prediction}\n")
shift=100 left=0 right=shift
while left<len(my_test_list): try_list=my_test_list[left:right]
ds_t = create_tf_dataset(try_list, bs=shift)
get_eval(ds_t)
left+=shift
right+=shift
@ajohnson49 I found out why you were looking at LJSpeech, I updated, here it is https://www.kaggle.com/code/bernardoolisan/speechrecognition-dot maybe you can check it out
Hi @ajohnson49 ,
In the tutorial there is no code provided for inference after training, because as you can see the model is trained for only epochs=1 and hence the inference results might not looks good. In real time we may need to train the model for many more epochs till we get desired accuracy which will take much time. Hence the Author left it to users choice if I am not wrong.
Maybe you can use a simple code like model.predict(ds.take(1)))
to check the results for a single batch .
But I can see now that at the end author provided some data of prediction and target around 30 epochs to visualise the results.
I hope limitation of epochs made the author to not add model inference results in the example.
Thanks!
This issue is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.
This issue was closed because it has been inactive for 28 days. Please reopen if you'd like to work on this further.
The example in transformer_asr.py is really interesting, but it only shows how to train the model. Can someone give an example of inference is done with the model to get TTS transcripts? Just calling model.predict() doesn't seem to be supported for this model.