search

GokuMohandas / Made-With-ML

Learn how to design, develop, deploy and iterate on production-grade ML applications.
https://madewithml.com
MIT License
37.1k stars 5.88k forks source link

Foundations --> Logistic regression feedback/errors #199

Closed gitgithan closed 2 years ago

gitgithan commented 2 years ago

  1. Webpage says W dimension is Dx1 but notebook says DxC. Prefer webpage to also show DxC to expose people to the more general multi-class W

  2. Two errors causing notebook to not run top-down a. Extra single quote behind k: plt.scatter(X[:, 0], X[:, 1], c=[colors[_y] for _y in y], s=25, edgecolors="k"') b. SyntaxError: Double quotes to index dictionary early closing double quotes for f-string (happens in 2 cells) print (f"m:b = {class_counts["malignant"]/class_counts["benign"]:.2f}")

  3. Hope the matrix calculus section had more explanation, feels to me like for people who understand it, they won't need the formulas, but for people who don't understand, it doesn't help much. Some questions I had going through that section.

    1. What's the physical meaning of y and j indexes in loss formula? Why does W have y subscript in numerator and sum across j in denominator? Why does denominator's W have no subscript. Seems to me like both y, j refer to one of the classes in a set of unique classes.
    2. In gradients formula, what is the physical meaning of Wy Wj, and why are we differentiating wrt to them?
    3. Why did i disappear from subscript of X in gradients section
    4. Why do some W have subscripts y/j while some W don't have any subscript
    5. Linking to some derivations like these (https://towardsdatascience.com/derivative-of-the-softmax-function-and-the-categorical-cross-entropy-loss-ffceefc081d1) would be very helpful

  4. How did db = np.sum(dscores, axis=0, keepdims=True) implementation come about? Was expecting a formula version describing gradient wrt bias but previous it's mentioned We'll leave the bias weights out for now to avoid complicating the backpropagation calculation

  5. W_{unscaled} includes sum in formula which it shouldn't?

GokuMohandas commented 2 years ago
  1. good idea. By the way, the notebooks are no longer up to date and I'm going to edit all of them this winter. For now, let's use the webpage as the up-to-date source and I'll just port over all the code from the webpage to the notebooks later this year.
  2. Good catch, must've happened when I tried to use double quotes everywhere and did an erroneous replace.
  3. I'll definitely clean these up a lot more when I sit down to polish all of these. I haven't touched these specific lessons in almost 6 years now (just moved them to here) so I'll go over them this winter.
  4. I'll include this in the update too. But I left out bias in the math (to avoid clutter) but not in the code.
  5. Correct, for same reasoning we discussed last week. I'll just make the change to the webpage now but will update notebooks later when I polish them.

I really appreciate these @gitgithan!