How does nn.conv2D implement the feature map table of LeNet-5?

I have been going through the pytorch documentation of conv2D. Reading the docs from pytorch I see

torch.nn.Conv2d(in_channels: int, out_channels: int, kernel_size: Union[T, Tuple[T, T]], stride: Union[T, Tuple[T, T]] = 1, padding: Union[T, Tuple[T, T]] = 0, dilation: Union[T, Tuple[T, T]] = 1, groups: int = 1, bias: bool = True, padding_mode: str = 'zeros')

From the LeNet paper of November 1998 I see that the third convolution layer is implemented with 6 input layers and 16 output layers. The 16 output layers are made from a combination of the 6 input layers according to a table, also in the paper:

LenetFeatureMap

From the chapter 5 tutorial, C3 is implemented with

lenet.add_module('C3', nn.Conv2d(in_channels=6, out_channels=16, kernel_size=5))

What I do not see is how pytorch implements this feature map table. It seems a little like a "loaves and fishes" issue :-).

P.S. If I might kill two birds with one stone, can i ask which plots are being implemented with IPython? I have observed that usually the plots that you generate are implemented in the plots directories from each chapter, using matplotlib. I was looking for HTML plots that were generated with IPython, because you call the IPython.core.display module and import display, HTML. But, I could not seem to spot how it was used.

Thank You Tom

Hi Tom,

The PyTorch implementation of C3 layer is a "reinterpretation" if you will of the original paper. It does not reflect the fancy customized column assignment of the depicted Table 1. I guess LeCun did it like that back then to try to squeeze some performance out of it - not something we need to be concerned anymore.

Regarding the IPython imports, I've only used them in the very first cell of the notebook to make it wider - if you're running it locally, it will make visualization a bit nicer, but that's all there is to it.

Best, Daniel

On Sun, Dec 27, 2020 at 5:45 AM Daniel Voigt Godoy notifications@github.com wrote:

Hi Tom,

The PyTorch implementation of C3 layer is a "reinterpretation" if you will of the original paper. It does not reflect the fancy customized column assignment of the depicted Table 1. I guess LeCun did it like that back then to try to squeeze some performance out of it - not something we need to be concerned anymore.

Regarding the IPython imports, I've only used them in the very first cell of the notebook to make it wider - if you're running it locally, it will make visualization a bit nicer, but that's all there is to it.

Best, Daniel

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/dvgodoy/PyTorchStepByStep/issues/16#issuecomment-751469716, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHGGHBTN6AALEHB3PE7WLDSW426NANCNFSM4VITCUVQ .

Got it!

Thank You Tom

Hi Tom,

Just published a new chapter :-) Not sure if you're receiving notifications from Leanpub, so I thought of letting you know here. Have fun!

Best, Daniel

Daniel,

No, I did not know about the release of chapter 16 until you told me. Of course, I could have been checking now and then. My bad.

In the meantime, I started another tutorial with "Hands on machine learning with Scikit learn, Keras and Tensorflow". Let me tell you that I wish YOU were the one that had written it. Your writing style is much clearer.

Thank You Tom

On Thu, Jan 14, 2021 at 1:10 PM Daniel Voigt Godoy notifications@github.com wrote:

Hi Tom,

Just published a new chapter :-) Not sure if you're receiving notifications from Leanpub, so I thought of letting you know here. Have fun!

Best, Daniel

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/dvgodoy/PyTorchStepByStep/issues/16#issuecomment-760478080, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHGGHFTRSRD4S4V5FZO3UDSZ5MVLANCNFSM4VITCUVQ .

Daniel,

I am sorry about the confusion. I got mixed up between your pytorch step by step course and your Machine learning course fundamentals course. I think that I did get the new chapter.

One question that I have is that I was going though parts of the machine learning course and I could not find a way to load the linear algebra module

intuitiveml.algebra.LinearAlgebra. Any idea of where I can find it?

Thank You Tom

On Thu, Jan 14, 2021 at 7:30 PM Tom Cipollone minertom11551@gmail.com wrote:

Daniel,

No, I did not know about the release of chapter 16 until you told me. Of course, I could have been checking now and then. My bad.

In the meantime, I started another tutorial with "Hands on machine learning with Scikit learn, Keras and Tensorflow". Let me tell you that I wish YOU were the one that had written it. Your writing style is much clearer.

Thank You Tom

On Thu, Jan 14, 2021 at 1:10 PM Daniel Voigt Godoy < notifications@github.com> wrote:

Hi Tom,

Just published a new chapter :-) Not sure if you're receiving notifications from Leanpub, so I thought of letting you know here. Have fun!

Best, Daniel

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/dvgodoy/PyTorchStepByStep/issues/16#issuecomment-760478080, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHGGHFTRSRD4S4V5FZO3UDSZ5MVLANCNFSM4VITCUVQ .

Hi,

Thanks for your comment on my writing style, that means a lot to me :-)

Regarding the LinearAlgebra module: I've developed this package as base for the ML Fundamentals course, but it was kinda "hacky" so I've uploaded it to PyPI only (but not GitHub).

If you install it using pip install intuitiveml, you should be able to import the LinearAlgebra module and use it as you like.

Also, the code for "intuitiveml" is open, you can download its source from PyPI here: https://pypi.org/project/intuitiveml/#files

Hope it helps.

Best, Daniel

Hi Danial,

I know that you are based in Germany, or the Netherlands, so, the chances of my going to anything but an online class taught by you are probably low. But, do you have a video course online or one planned for the future?

i would really like to see your "take" on support vector machines. As I mentioned earlier, I am going through "hands on machine learning" by Geron. Definitely not a good introduction to some of the topics. Finding that I had to go far afield from the actual book in order to find good material for making the topic "real".

Regards Tom Cipollone

On Fri, Jan 15, 2021 at 4:23 AM Daniel Voigt Godoy notifications@github.com wrote:

Hi,

Thanks for your comment on my writing style, that means a lot to me :-)

Regarding the LinearAlgebra module: I've developed this package as base for the ML Fundamentals course, but it was kinda "hacky" so I've uploaded it to PyPI only (but not GitHub).

If you install it using pip install intuitiveml, you should be able to import the LinearAlgebra module and use it as you like.

Also, the code for "intuitiveml" is open, you can download its source from PyPI here: https://pypi.org/project/intuitiveml/#files

Hope it helps.

Best, Daniel

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/dvgodoy/PyTorchStepByStep/issues/16#issuecomment-760911678, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHGGHBTJUZTVMK76CMVSG3S2AXTNANCNFSM4VITCUVQ .

Hi Tom,

I don't have a video course yet, but it is definitely in my plans for 2021 after I finish the book.

Regarding SVMs, I guess you've already gone through the corresponding notebook in ML Fundamentals (https://github.com/dvgodoy/ML_Fundamentals/blob/master/13.%20Support%20Vector%20Machine%20(SVM).ipynb)?

There isn't much to it actually: it is a glorified linear regression. The trick is, you want to have all points in the positive class to have the "predicted y" being 1 or more, while all points in the negative class must have their "predicted y" being -1 or less. The +1/-1 is completely arbitrary. Moreover, you want two points, the extremes of each class, to be exactly 1 and -1. There is one set of weights for the "y = w1x1 + w2x2 +..." that makes that happen (IF the points are separable by a straight line).

Since most real problems won't be separable like that, you need to find a compromise between this linear regression and the points that are trespassing. That's a soft margin, and you control how soft/hard it is by using the C multiplier on the penalty of the trespasser points.

Then there is the kernel trick, which is just a fancy way of applying a function to all the features (usually polynomials) to increase dimensionality and make the points more easily separable. The kernel trick itself is just a mathematical shortcut to avoid actually computing these transformations. Roughly speaking, it is like: the function of the dot product yields the same result as the dot product of the function.

Why dot product, right? Again, fancy mathematical thing... minimizing the weights of that linear regression is equivalent to a different optimization problem that involves computing dot products for all pairs of data points.

SVMs are a clever way of addressing nonlinearities in datasets, especially before NNs and activation functions became the norm. For some reason, it is still common to see them used in medicine/healthcare, but not so much anywhere else (that I know of).

I am not sure if that's what you were looking for... is there any other specifics you were interested in?

Best, Daniel

Daniel,

Thank you so much for your kind input. Now that I know a little more about SVM I don't believe that it is worth much in the line of research that I am pursuing. I had expected that I would find a good mathematical explanation of the SVM as compared to the Gilbert Strang from MIT Linear algebra lectures #15 and #16 which described Linear Regression and the Normal equations very clearly. No such luck with SVM. In any case, CNNs seem to be "the thing" best suited for object detection.

My interest is in object detection. You may remember that I already used YOLOV3 in a project of object detection of aircraft using the darknet framework. I'll never use the darknet framework again because it is not very flexible and the community that uses it is very small.

There is a version of YOLOV5 using pytorch that was put on github by ultralytics https://github.com/ultralytics/yolov5. https://github.com/ultralytics/yolov5

There is also a version of YOLOV3 using tensorflow 2 from https://github.com/zzh8829/yolov3-tf2.

At this point, I feel somewhat just a little competent about the Pytorch framework, thanks to your book, but I want to be able to compare it to the tensorflow framework.

Please let me know when you are working on the course.

Thank You Tom

On Wed, Jan 20, 2021 at 11:36 AM Daniel Voigt Godoy < notifications@github.com> wrote:

Hi Tom,

I don't have a video course yet, but it is definitely in my plans for 2021 after I finish the book.

Regarding SVMs, I guess you've already gone through the corresponding notebook in ML Fundamentals ( https://github.com/dvgodoy/ML_Fundamentals/blob/master/13.%20Support%20Vector%20Machine%20(SVM).ipynb )?

There isn't much to it actually: it is a glorified linear regression. The trick is, you want to have all points in the positive class to have the "predicted y" being 1 or more, while all points in the negative class must have their "predicted y" being -1 or less. The +1/-1 is completely arbitrary. Moreover, you want two points, the extremes of each class, to be exactly 1 and -1. There is one set of weights for the "y = w1x1 + w2x2 +..." that makes that happen (IF the points are separable by a straight line).

Since most real problems won't be separable like that, you need to find a compromise between this linear regression and the points that are trespassing. That's a soft margin, and you control how soft/hard it is by using the C multiplier on the penalty of the trespasser points.

Then there is the kernel trick, which is just a fancy way of applying a function to all the features (usually polynomials) to increase dimensionality and make the points more easily separable. The kernel trick itself is just a mathematical shortcut to avoid actually computing these transformations. Roughly speaking, it is like: the function of the dot product yields the same result as the dot product of the function.

Why dot product, right? Again, fancy mathematical thing... minimizing the weights of that linear regression is equivalent to a different optimization problem that involves computing dot products for all pairs of data points.

SVMs are a clever way of addressing nonlinearities in datasets, especially before NNs and activation functions became the norm. For some reason, it is still common to see them used in medicine/healthcare, but not so much anywhere else (that I know of).

I am not sure if that's what you were looking for... is there any other specifics you were interested in?

Best, Daniel

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/dvgodoy/PyTorchStepByStep/issues/16#issuecomment-763883729, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHGGHHUAICRYT4FYB2WXULS24WDHANCNFSM4VITCUVQ .

Hi Daniel,

After having worked on "Hands on machine Learning with Scikit Learn, Keras and Tensorflow" for a couple of days, I have decided to put it aside. I find it far to obscure for the equivalent functionality of pytorch.

Therefore, I will be reviewing your course/book again and concentrating on object detection with pytorch. For my use I am finding Keras/Tensorflow not worth the effort.

Regards Tom

On Wed, Jan 20, 2021 at 4:45 PM Tom Cipollone minertom11551@gmail.com wrote:

Daniel,

Thank you so much for your kind input. Now that I know a little more about SVM I don't believe that it is worth much in the line of research that I am pursuing. I had expected that I would find a good mathematical explanation of the SVM as compared to the Gilbert Strang from MIT Linear algebra lectures #15 and #16 which described Linear Regression and the Normal equations very clearly. No such luck with SVM. In any case, CNNs seem to be "the thing" best suited for object detection.

My interest is in object detection. You may remember that I already used YOLOV3 in a project of object detection of aircraft using the darknet framework. I'll never use the darknet framework again because it is not very flexible and the community that uses it is very small.

There is a version of YOLOV5 using pytorch that was put on github by ultralytics https://github.com/ultralytics/yolov5. https://github.com/ultralytics/yolov5

There is also a version of YOLOV3 using tensorflow 2 from https://github.com/zzh8829/yolov3-tf2.

At this point, I feel somewhat just a little competent about the Pytorch framework, thanks to your book, but I want to be able to compare it to the tensorflow framework.

Please let me know when you are working on the course.

Thank You Tom

On Wed, Jan 20, 2021 at 11:36 AM Daniel Voigt Godoy < notifications@github.com> wrote:

Hi Tom,

I don't have a video course yet, but it is definitely in my plans for 2021 after I finish the book.

Regarding SVMs, I guess you've already gone through the corresponding notebook in ML Fundamentals ( https://github.com/dvgodoy/ML_Fundamentals/blob/master/13.%20Support%20Vector%20Machine%20(SVM).ipynb )?

There isn't much to it actually: it is a glorified linear regression. The trick is, you want to have all points in the positive class to have the "predicted y" being 1 or more, while all points in the negative class must have their "predicted y" being -1 or less. The +1/-1 is completely arbitrary. Moreover, you want two points, the extremes of each class, to be exactly 1 and -1. There is one set of weights for the "y = w1x1 + w2x2 +..." that makes that happen (IF the points are separable by a straight line).

Since most real problems won't be separable like that, you need to find a compromise between this linear regression and the points that are trespassing. That's a soft margin, and you control how soft/hard it is by using the C multiplier on the penalty of the trespasser points.

Then there is the kernel trick, which is just a fancy way of applying a function to all the features (usually polynomials) to increase dimensionality and make the points more easily separable. The kernel trick itself is just a mathematical shortcut to avoid actually computing these transformations. Roughly speaking, it is like: the function of the dot product yields the same result as the dot product of the function.

Why dot product, right? Again, fancy mathematical thing... minimizing the weights of that linear regression is equivalent to a different optimization problem that involves computing dot products for all pairs of data points.

SVMs are a clever way of addressing nonlinearities in datasets, especially before NNs and activation functions became the norm. For some reason, it is still common to see them used in medicine/healthcare, but not so much anywhere else (that I know of).

I am not sure if that's what you were looking for... is there any other specifics you were interested in?

Best, Daniel

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/dvgodoy/PyTorchStepByStep/issues/16#issuecomment-763883729, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHGGHHUAICRYT4FYB2WXULS24WDHANCNFSM4VITCUVQ .

dvgodoy / PyTorchStepByStep

How does nn.conv2D implement the feature map table of LeNet-5? #16