Exercises Day1

[constantinpape]

Summary of issues with exercises of day 1:

• 1e-3 as learning rate is too high for the LogReg and the MLP. A good learning rate is 5.e-4 or 1.e-4.
• In exercise 2: trying just some new filters is too open ended; many skimage filters are slow or only work with 2d images so they need to be wrapped in another function. (Especially because the results are not so great with high LR).
• In exercise 2: applying both skimage and torch conv filters is inconvenient, because it requires a lot of switching between the axis conventions.

Ideas for updating the exercises: Exercise 2: Try different learning rates for the LogReg model here (and start from a better learning rate). Don't do filters. Exercise 3: Start with a bit smaller MLP. Check if best LR for LogReg also works well for MLP. Otherwise make tuning the learning rate a task again. First task: larger architecture. Second task: use skimage filters (laplacian + gaussian) as input instead of image. Advanced task: Express the filters as torch.nn.conv2d.

Also, use SGD instead of Adam? Check how much this changes the learning rate issues.

[constantinpape]

Logistic regression is just not very powerful (interestingly 1e-3 is best learning rate). I think it would be best to keep that exercise minimal, maybe just have a couple of questions, and have all the tasks in the MLP exercise.