deeplearning - 3/w2

[getElementsByName]

Structuring Machine Learning Projects - Week 2. ML strategy (2)

주요 개념

• Understand what multi-task learning and transfer learning are
• Recognize bias, variance and data-mismatch by looking at the performances of your algorithm on train/dev/test sets

Error Analysis

Carrying out error analysis

• a set of mislabeled examples(false positive + false negative), either in your dev set, or in your development set.
• hand engineering things
  • I know that in machine learning, sometimes we speak disparagingly of hand engineering things, or using too much value insight. But if you're building applied systems, then this simple counting procedure, error analysis, can save you a lot of time.
• the ceiling on performance
  • Upper bound on how much you could improve performance by working on a specific problem
• This, in just 5 to 10 minutes, gives you an estimate of how worthwhile this direction is.

예시


Cleaning up incorrectly labeled data

• mislabeled, incorrectly labeled
  • So I've used the term, mislabeled examples, to refer to if your learning algorithm outputs the wrong value of Y.
  • But I'm going to say, incorrectly labeled examples, to refer to if in the data set you have in the training set or the dev set or the test set, the label for Y, whatever a human label assigned to this piece of data, is actually incorrect.

training set

• robust to (near) random errors
  • probably okay to just leave the errors
  • <-> systemic errors (consistently wrong)

development set

test set

• make the dev/test sets come from same distribution
• examining examples your algorithm got right as well as ones it get wrong
• training / dev,test sets may come from slightly different distributions.
  • (later chpater) data mismatch -> training-dev set

error analysis에 'incorrectly labeled' 고려하는 경우


Build your first system quickly, then iterate

• if you are tackling a new problem for the first time
  • to figure out all the different directions you can go in, which ones are actually the most worthwhile
• This advice applies less strongly if you're working on an application area in which you have significant prior experience.
  • ex) if there's a significant body of academic literature

Guideline: Build your first system quickly, then iterate

'

Mismatched training and dev/test set

Training and testing on different distributions

• (previous chapter)
  • dev/test sets (+ validation metric): defining what target you want to aim at.
  • The way you choose your training set will affect how well you can actually hit that target
• a huge hunger for training data
• training on data that comes from a different distribution than your dev and test sets

Speech recognition example


Bias and Variance with mismatched data distributions

• train-dev set, train-dev set error

Addressing data mismatch

• artificial data synthesis
  • And artificial data synthesis does work. In speech recognition, I've seen artificial data synthesis significantly boost the performance of what were already very good speech recognition system. So, it can work very well.
  • overfit to a small set: But, if you're using artificial data synthesis, just be cautious and bear in mind whether or not you might be accidentally simulating data only from a tiny subset of the space of all possible examples.

quiz

10) If the synthesized images look realistic, then the model will just see them as if you had added useful data to identify road signs and traffic signals in a foggy weather. I will very likely help.


Learning from multiple tasks

Transfer learning

• pre-training
  • pre-initialize
• fine tuning

참고자료

• https://towardsdatascience.com/transfer-learning-from-pre-trained-models-f2393f124751

Feature Extraction -> Fine Tuning

• pretrained model에 부분 layer만 trainable 설정 + trainable 토글
  • https://keras.io/guides/transfer_learning/
  • https://www.tensorflow.org/tutorials/images/transfer_learning
• pretrained model trainable = True 설정 + 연속적 learning rate 튜닝
  • https://colab.research.google.com/notebooks/tpu.ipynb

Multi-task learning

• In multi-task learning, you start off simultaneously, trying to have one neural network do several things at the same time.
• And then each of these task helps hopefully all of the other task.
  • If some of the earlier features in neural network can be shared between these different types of objects, then you find that training one neural network to do four things results in better performance than training four completely separate neural networks to do the four tasks separately.
• you just omit unlabel term from summation but just sum over only the values where there is a label.
• the main difference between this and softmax regression, is that unlike softmax regression, which assigned a single label to single example. This one image can have multiple labels.

End-to-end deep learning

What is end-to-end deep learning?

• multi-step approach <-> end-to-end
  • multi-step approach
    • each of the two problems you're solving is actually much simpler.
    • a lot of data for each of the two sub-tasks.
• machine learning <-> hand-designed

Whether to use end-to-end deep learning

• I think of a learning algorithm as having two main sources of knowledge. One is the data and the other is whatever you hand design, be components, or features, or other things.
• hand-designed system can actually allow humans to inject a lot of knowledge about the problem into an algorithm that should be very helpful.
  • hand-designed components tend to help more when you're training on a small training set
  • it's kind of a double edged sword that could hurt or help but it does tend to help more

[getElementsByName]

Transfer Learning

• https://towardsdatascience.com/transfer-learning-from-pre-trained-models-f2393f124751

Feature Extraction -> Fine Tuning

• pretrained model 전체 trainable 토글
  • https://keras.io/guides/transfer_learning/
• pretrained model에 부분 layer만 trainable 설정 + trainable 토글
  • https://www.tensorflow.org/tutorials/images/transfer_learning
• pretrained model trainable = True 설정 + 연속적 learning rate 튜닝
  • https://colab.research.google.com/notebooks/tpu.ipynb

[getElementsByName]

quiz

10) If the synthesized images look realistic, then the model will just see them as if you had added useful data to identify road signs and traffic signals in a foggy weather. I will very likely help.


[getElementsByName]

• previous chapter
  • dev/test sets (+ validation metric): defining what target you want to aim at.
  • The way you choose your training set will affect how well you can actually hit that target