Missing Karpathy tips

[trebacz626]

here is a list of karpathy tips that we can implement as steps

• [ ] init well. Initialize the final layer weights correctly. E.g. if you are regressing some values that have a mean of 50 then initialize the final bias to 50. If you have an imbalanced dataset of a ratio 1:10 of positives:negatives, set the bias on your logits such that your network predicts probability of 0.1 at initialization. Setting these correctly will speed up convergence and eliminate “hockey stick” loss curves where in the first few iteration your network is basically just learning the bias.
• [ ] human baseline. Monitor metrics other than loss that are human interpretable and checkable (e.g. accuracy). Whenever possible evaluate your own (human) accuracy and compare to it. Alternatively, annotate the test data twice and for each example treat one annotation as prediction and the second as ground truth.
• [ ] use backprop to chart dependencies. Your deep learning code will often contain complicated, vectorized, and broadcasted operations. A relatively common bug I’ve come across a few times is that people get this wrong (e.g. they use view instead of transpose/permute somewhere) and inadvertently mix information across the batch dimension. It is a depressing fact that your network will typically still train okay because it will learn to ignore data from the other examples. One way to debug this (and other related problems) is to set the loss to be something trivial like the sum of all outputs of example i, run the backward pass all the way to the input, and ensure that you get a non-zero gradient only on the i-th input. The same strategy can be used to e.g. ensure that your autoregressive model at time t only depends on 1..t-1. More generally, gradients give you information about what depends on what in your network, which can be useful for debugging.
• [ ]

[SebChw]

TBD in v3

[kordc]

I'd also love the visualize just before the net - but it's so task-agnostic that I don't have any idea how to implement it well

[kordc]

I think it should be converted to 4 different issues, where we decide whether we have time to do it or not

[SebChw]

Generally I think most of these shouldn't be steps themselves.

1. Init well - this should be model modifier - or even some necessary function to implemetn
2. Human baseline - For this we can only warn user to use metrics different than just loss.
3. This is very nice Idk if this should be a step or again a network modifier
4. Concerning visualize just before the net for me this is a perfect usecase of ArtDecorator

[kordc]

1. Maybe not necessary, but it could be a function to implement.
2. Sure, just information that one can write its own "metric"
3. to be discussed
4. Sure, maybe it could be a use case shown in some tutorial. Whole tutorial of ArtDecorator

[SebChw]

4. It is actually used in the very first tutorial

[kordc]

Can I close this issue and create:

• Implement init well (for 0.3)
• Add an information about human metrics
• Create a step use backprop to chart dependencies ?