Structuring Machine Learning Projects - Week 2. ML strategy (2)
주요 개념
Understand what multi-task learning and transfer learning are
Recognize bias, variance and data-mismatch by looking at the performances of your algorithm on train/dev/test sets
Error Analysis
Carrying out error analysis
a set of mislabeled examples(false positive + false negative), either in your dev set, or in your development set.
hand engineering things
I know that in machine learning, sometimes we speak disparagingly of hand engineering things, or using too much value insight. But if you're building applied systems, then this simple counting procedure, error analysis, can save you a lot of time.
the ceiling on performance
Upper bound on how much you could improve performance by working on a specific problem
This, in just 5 to 10 minutes, gives you an estimate of how worthwhile this direction is.
So I've used the term, mislabeled examples, to refer to if your learning algorithm outputs the wrong value of Y.
But I'm going to say, incorrectly labeled examples, to refer to if in the data set you have in the training set or the dev set or the test set, the label for Y, whatever a human label assigned to this piece of data, is actually incorrect.
training set
robust to (near) random errors
probably okay to just leave the errors
<-> systemic errors (consistently wrong)
development set
test set
make the dev/test sets come from same distribution
examining examples your algorithm got right as well as ones it get wrong
training / dev,test sets may come from slightly different distributions.
(later chpater) data mismatch -> training-dev set
error analysis에 'incorrectly labeled' 고려하는 경우
![image](https://user-images.githubusercontent.com/21363169/86311903-62462a00-bc5c-11ea-943c-aaa1fbe570dc.png)
Build your first system quickly, then iterate
if you are tackling a new problem for the first time
to figure out all the different directions you can go in, which ones are actually the most worthwhile
This advice applies less strongly if you're working on an application area in which you have significant prior experience.
ex) if there's a significant body of academic literature
Guideline: Build your first system quickly, then iterate
![image](https://user-images.githubusercontent.com/21363169/86312161-0f20a700-bc5d-11ea-8b63-391663095081.png)'
Mismatched training and dev/test set
Training and testing on different distributions
(previous chapter)
dev/test sets (+ validation metric): defining what target you want to aim at.
The way you choose your training set will affect how well you can actually hit that target
a huge hunger for training data
training on data that comes from a different distribution than your dev and test sets
Speech recognition example
![image](https://user-images.githubusercontent.com/21363169/86312388-8bb38580-bc5d-11ea-8eb6-8013da1c0b84.png)
Bias and Variance with mismatched data distributions
train-dev set, train-dev set error
Addressing data mismatch
artificial data synthesis
And artificial data synthesis does work. In speech recognition, I've seen artificial data synthesis significantly boost the performance of what were already very good speech recognition system. So, it can work very well.
overfit to a small set: But, if you're using artificial data synthesis, just be cautious and bear in mind whether or not you might be accidentally simulating data only from a tiny subset of the space of all possible examples.
quiz
10) If the synthesized images look realistic, then the model will just see them as if you had added useful data to identify road signs and traffic signals in a foggy weather. I will very likely help.
![Screenshot from 2020-06-30 21-28-31](https://user-images.githubusercontent.com/21363169/86126079-adefbb00-bb18-11ea-8807-d0680a81f934.png)
In multi-task learning, you start off simultaneously, trying to have one neural network do several things at the same time.
And then each of these task helps hopefully all of the other task.
If some of the earlier features in neural network can be shared between these different types of objects, then you find that training one neural network to do four things results in better performance than training four completely separate neural networks to do the four tasks separately.
you just omit unlabel term from summation but just sum over only the values where there is a label.
the main difference between this and softmax regression, is that unlike softmax regression, which assigned a single label to single example. This one image can have multiple labels.
End-to-end deep learning
What is end-to-end deep learning?
multi-step approach <-> end-to-end
multi-step approach
each of the two problems you're solving is actually much simpler.
a lot of data for each of the two sub-tasks.
machine learning <-> hand-designed
Whether to use end-to-end deep learning
I think of a learning algorithm as having two main sources of knowledge. One is the data and the other is whatever you hand design, be components, or features, or other things.
hand-designed system can actually allow humans to inject a lot of knowledge about the problem into an algorithm that should be very helpful.
hand-designed components tend to help more when you're training on a small training set
it's kind of a double edged sword that could hurt or help but it does tend to help more
10) If the synthesized images look realistic, then the model will just see them as if you had added useful data to identify road signs and traffic signals in a foggy weather. I will very likely help.
![Screenshot from 2020-06-30 21-28-31](https://user-images.githubusercontent.com/21363169/86126079-adefbb00-bb18-11ea-8807-d0680a81f934.png)
Structuring Machine Learning Projects - Week 2. ML strategy (2)
주요 개념
multi-task learning
andtransfer learning
aredata-mismatch
by looking at the performances of your algorithm on train/dev/test setsError Analysis
Carrying out
error analysis
mislabeled examples
(false
positive +false
negative), either in yourdev
set, or in your development set.hand engineering things
the ceiling on performance
Upper bound
on how much you couldimprove performance
by working on a specific problemworthwhile
this direction is.예시
![image](https://user-images.githubusercontent.com/21363169/86311831-3b87f380-bc5c-11ea-86f3-e43c3807ca83.png)Cleaning up incorrectly labeled data
mislabeled
,incorrectly labeled
mislabeled
examples, to refer to if yourlearning algorithm outputs
the wrong value of Y.incorrectly labeled
examples, to refer to if in the data set you have in the training set or the dev set or the test set, the label for Y, whatever ahuman label assigned
to this piece of data, is actually incorrect.training set
random
errorsdevelopment set
test set
same distribution
right
as well as ones it get wrongerror analysis에 'incorrectly labeled' 고려하는 경우
![image](https://user-images.githubusercontent.com/21363169/86311903-62462a00-bc5c-11ea-943c-aaa1fbe570dc.png)Build your first system quickly, then iterate
the most worthwhile
Guideline: Build your first system quickly, then iterate
![image](https://user-images.githubusercontent.com/21363169/86312161-0f20a700-bc5d-11ea-8b63-391663095081.png)'Mismatched training and dev/test set
Training and testing on different distributions
what target
you want to aim at.how well
you can actually hit that targettraining
on data that comes from adifferent distribution
than yourdev and test sets
Speech recognition example
![image](https://user-images.githubusercontent.com/21363169/86312388-8bb38580-bc5d-11ea-8eb6-8013da1c0b84.png)Bias and Variance with mismatched data distributions
Addressing data mismatch
quiz
10) If the synthesized images look realistic, then the model will just see them as if you had added useful data to identify road signs and traffic signals in a foggy weather. I will very likely help.
![Screenshot from 2020-06-30 21-28-31](https://user-images.githubusercontent.com/21363169/86126079-adefbb00-bb18-11ea-8807-d0680a81f934.png)Learning from multiple tasks
Transfer learning
참고자료
Feature Extraction -> Fine Tuning
trainable = True
설정 + 연속적 learning rate 튜닝Multi-task learning
earlier features
in neural network canbe shared
between these different types of objects, then you find that training one neural network to do four things results in better performance than training four completely separate neural networks to do the four tasks separately.omit unlabel
term from summation but just sum over only the values where there is a label.End-to-end deep learning
What is end-to-end deep learning?
Whether to use end-to-end deep learning
two main sources
of knowledge. One is thedata
and the other is whatever youhand design
, be components, or features, or other things.hand-designed
system can actually allow humans toinject a lot of knowledge
about the problem into an algorithm that should be very helpful.small training set
a double edged sword
that could hurt or help but it does tend to help more