Reorganize beginner section

[epogrebnyak]

Updates from:

• https://github.com/jacobhilton/deep_learning_curriculum (focus on transformers)
• Raschka book

1. Math prerequisites

Taking a derivative to find a point of minimum or maximum, integrating to find an area under the curve and multiplying matrices are basic operations you need calculus and linear algebra for. Probability and mathematic statistics are the next math courses you take usually second or third year in college. After that you are prepared to jump into econometrics or machine learning.

Each of these subjects were taught for long time in universities, which means there are good textbooks and pathways developped to guide learning. However introductory math is hard to learn on your own without instruction and tutoring. You are required to fill gaps on each level of understanding before going to a next level and keep up the motivation to study further.

The MML book makes a great job of reviewing topics in needed for machine learning, yet you can also investigate specific resources for calulus, linear algebra, probability and mathematic statistics.

2. Reader profiles:

• beginner, first-time learner
• reviewer, tries to keep up
• switchers to machine learning engineering (from econometrics, from software engineering)

[epogrebnyak]

3. A perfect book shelf: top pick and why + same shelf

4. Information theory https://www.reddit.com/r/compsci/comments/1zubm2/what_are_good_books_on_information_theory/?onetap_auto=true&one_tap=true, Cover and Thomas + original Shannon paper.

[epogrebnyak]

5. Doing research: http://joschu.net/blog/opinionated-guide-ml-research.html

6. Michael Nielsen

7. Case by case:

Language models: Easier: Train a small GPT-2 model from scratch using existing implementations of the architecture (e.g. from Hugging Face). Maybe try grokking. Harder: Implement the transformer yourself. You could also fine-tune with human preferences, as in Ziegler et al.) (2019) or Stiennon et al. (2020). Reinforcement learning (I probably wouldn’t start with these): Easier: Try to replicate one of the common RL algorithms, like DQN/PPO/A3C. There are papers that talk about ways these methods don’t replicate. See Spinning up in deep RL as a way to ease into these exercises. Harder: Deep RL from human preferences (see this also) or GAIL. Computer vision: Very easy: train an MLP on MNIST. Easy: train a ResNet or another close-to-state-of-the-art model on ImageNet.

8. AMA section - two examples

9. NYU + fast.ai fast.ai is a practical course in deep learning (DL) that approaches DL from a coding (not math/statistics) perspective. If you already have some knowledge of how DL works, it is probably better to learn from the PyTorch tutorials. Or learn from those tutorials after doing fast.ai. PyTorch is a good framework to start with, but if you’re already good with TensorFlow or JAX you probably don’t need to pick PyTorch up until a project/job requires it. Deep Learning Specialization (Ng), your standard DL class (CS 230 at Stanford). Deep Learning by NYU (LeCun).

[epogrebnyak]

10. https://goodresearch.dev/_static/book.pdf