deeplearning - 5/w3

[getElementsByName]

5 Sequence Models - 3 Sequence models & Attention mechanism

• Encoder-decoder model / Sequence to sequence model
• Beam Search
• Bleu Score
• Attention Model
• spectrogram

Various sequence to sequence architectures

Basic Models

• Encoder-decoder model
• Sequence to sequence model 예
  • French -> English translation
• image-to-sequence or image captioning model 예
  • Image Captioning
• you don't want a randomly chosen translation, you maybe want the most likely translation

Picking the most likely sentence

• conditional language model

Beam Search

Refinements to Beam Search

• Sometimes it's called a normalized log likelihood objective.
• underflow -> production -> log (sum)
• the more terms you have together, the more negative this thing becomes. -> normalize this by the number of words in your translation
  • And this significantly reduces the penalty for outputting longer translations.
• beam width가 1~10에서 효과가 클 것이다

Error analysis in beam search

• your model has two main components.
  • neural network model (seq2seq, RNN)
  • Beam search algorithm

Bleu Score (optional)

• BLEU(Bilingual Evaluation Understudy)
  • the BLEU score is an understudy, could be a substitute for having humans evaluate every output of a machine translation system.

• BP, Stands for brevity penalty
  • machine translation system이 짧은 문장의 결과를 내었을때, 의도치 않게 정확도가 높아지는 문제를 보정

Attention Model Intuition

기존 구조의 한계

• [X] for very long sentences, maybe longer than 30 or 40 words, the performance comes down
  • encoder의 결과를 모두 기억했다가 decoder로 한번에 전달
• [ ] short sentences are just hard to translate, hard to get all the words

Attention Model 아이디어

• Encoder의 내용들에 각각 가중치를 적용하여 decoder에 반영

Attention Model

• c: context vector
• e: energies

Speech recognition - Audio data

Speech recognition

• pre-processing
  • spectrograms
    • 주파수와 진폭(세기)정보를 시간의 흐름에 따라 표현
    • Short-time Fourier Transform 의 결과로 표현
  • 달팽이관 구조 https://brunch.co.kr/@cogsciin/45

• 음성데이터에 대해 생각할 수 있는 한가지 neural network model

• CTC (Connectionist temporal classification)

  • input과 output의 time step 수를 동일하게 하는 방안

Trigger Word Detection

• trigger word detection (sometimes also called keyword detection, or wake word detection)

  • Trigger word detection is the technology that allows devices like Amazon Alexa, Google Home, Apple Siri, and Baidu DuerOS to wake up upon hearing a certain word.

• y_hat when that that clip has finished saying "activate".

• Rather than updating the label of a single time step, we will update several(50) steps of the output to have target label 1.

  • That updating several consecutive time steps can make the training data more balanced.

[getElementsByName]

Do we need to use beam search in training process?

https://stackoverflow.com/questions/44228271/do-we-need-to-use-beam-search-in-training-process

transformer

• https://www.tensorflow.org/tutorials/text/transformer
• https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html