No Issue. I purchased the book. Can you hint towards future chapters?

[minertom]

HI, I just purchased the book as I am in the process of learning Pytorch. I was wondering if you could give a hint towards future chapters.

Also, How would I be notified when future chapters are available?

Thank You Tom

[dvgodoy]

Hi Tom,

Thank you for supporting my work :-)

I've added the planned chapters below - it should be on the TOC of Leanpub's webpage, but there is an issue in their website at the moment and I cannot update it... but here it is!

If you enabled notifications in Leanpub, you'll receive an email every time I publish a new chapter - I've been publishing at least one chapter per month. The dates below are estimates (they may be moved earlier, but not later). Part III does not have any details because I am currently outlining it :-)

Best, Daniel

• Bonus Chapter: Feature Space (Nov/2020)
    ◦ Two-Dimensional Feature Space
    ◦ Transformations
    ◦ A Two-Dimensional Model
    ◦ Decision Boundary, Activation Style!
    ◦ More Functions, More Boundaries
    ◦ More Layers, More Boundaries
    ◦ More Dimensions, More Boundaries
    ◦ Recap
• Chapter 5: Convolutions (Dec/2020)
    ◦ Spoilers
    ◦ Jupyter Notebook
    ◦ Convolutions
        ▪ Filter/Kernel
        ▪ Convolving
        ▪ Moving Around
        ▪ Shape
        ▪ Convolving in PyTorch
        ▪ Striding
        ▪ Padding
        ▪ A REAL Filter
    ◦ Pooling
    ◦ Flattening
    ◦ Dimensions
    ◦ Typical Architecture
    ◦ A Multiclass Classification Problem
        ▪ Data Generation
        ▪ Data Preparation
        ▪ Loss
        ▪ Classification Losses Showdown!
        ▪ Model Configuration
        ▪ Model Training
    ◦ Visualizing Filters and More!
        ▪ Static Method
        ▪ Visualizing Filters
        ▪ Hooks
        ▪ Visualizing Feature Maps
        ▪ Visualizing Classifier Layers
        ▪ Accuracy
        ▪ Loader Apply
    ◦ Putting It All Together
    ◦ Recap
• Chapter 6: Rock, Paper, Scissors (Jan/2021)
    ◦ Spoilers
    ◦ Jupyter Notebook
    ◦ Rock, Paper, Scissors...
    ◦ Data Preparation
        ▪ ImageFolder
        ▪ Standardization
        ▪ The Real Datasets
    ◦ Three-Channel Convolutions
    ◦ Fancier Model
    ◦ Dropout
    ◦ Model Configuration
    ◦ Model Training
    ◦ Learning Rates
        ▪ Finding LR
        ▪ Adaptive Learning Rate
        ▪ Stochastic Gradient Descent (SGD)
              Momentum
              Nesterov
              Flavors of SGD
        ▪ Learning Rate Schedulers
        ▪ Adaptive vs Cycling
    ◦ Putting It All Together
    ◦ Recap
• Chapter 7: Transfer Learning (Feb/2021)
    ◦ Spoilers
    ◦ Jupyter Notebook
    ◦ Transfer Learning
    ◦ ImageNet
    ◦ ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
    ◦ Transfer Learning in Practice
        ▪ Pre-Trained Model
        ▪ Model Configuration
        ▪ Data Preparation
        ▪ Model Training
        ▪ Generating a Dataset of Features
        ▪ Top Model
    ◦ Auxiliary Classifiers (Side-Heads)
    ◦ 1x1 Convolutions
    ◦ Inception Modules
    ◦ Batch Normalization
        ▪ Running Statistics
        ▪ Evaluation Phase
        ▪ Momentum
        ▪ BatchNorm2d
        ▪ Other Normalizations
        ▪ Small Summary
    ◦ Vanishing Gradients & Initialization Schemes
    ◦ Residual Connections
          Learning the Identity
          The Power of Shortcuts
          Residual Blocks
    ◦ Putting It All Together
          Fine-Tuning
          Feature Extraction
    ◦ Recap
• Part III: Sequences and NLP (working titles - from Mar/2021)
    ◦ Chapter 8: Recurrent Neural Networks
    ◦ Chapter 9: Language Models and Embeddings
    ◦ Chapter 10: Down the Rabbit Hole

[minertom]

Daniel,

Thank you so much for this reply. I am working on three or four tutorials for pytorch at the moment and yours looks fascinating.

Just to let you know where this is going, I am a "retired" electronics design engineer. When I started designing hardware I worked in the early image processing industry. When I tell people that at the time there were no special purpose microprocessors and that we built our own multiplier/accumulator boards from discrete components and that we designed and built our own graphics display cards, they look at me as if I come from the dark ages (perhaps it is true... stone knives and bearskins were still a thing ) . Back then, in the early 80's, we could not have dreamed of using general purpose computers for image processing (I remember mainframes and their card readers......ugly, hurtful things). Although I was not involved with the algorithm design at the time, we had to use mathematically efficient algorithms in order to reduce the computational load. I have some ideas as to how to use some of those algorithms for convolutions in the new frameworks, because it seems to me, and I could be wrong, that the majority of users these days are just throwing computer power at the problem by purchasing faster CPU's, more expensive GPU's or using more AWS cloud services. My eventual aim is to use FPGA's for acceleration.

I use the term "retired" in quotes because I really don't want to be. But I am older now and a lot of what I used to do for a living has been outsourced. I should have moved into software decades ago and we appear to be in what I would call a new golden age of opportunity WRT software utilization.

Regards Tom

On Thu, Nov 12, 2020 at 7:37 AM Daniel Voigt Godoy notifications@github.com wrote:

Hi Tom,

Thank you for supporting my work :-)

I've added the planned chapters below - it should be on the TOC of Leanpub's webpage, but there is an issue in their website at the moment and I cannot update it... but here it is!

If you enabled notifications in Leanpub, you'll receive an email every time I publish a new chapter - I've been publishing at least one chapter per month. The dates below are estimates (they may be moved earlier, but not later). Part III does not have any details because I am currently outlining it :-)

Best, Daniel

• Bonus Chapter: Feature Space (Nov/2020)

◦ Two-Dimensional Feature Space

◦ Transformations

◦ A Two-Dimensional Model

◦ Decision Boundary, Activation Style!

◦ More Functions, More Boundaries

◦ More Layers, More Boundaries

◦ More Dimensions, More Boundaries

◦ Recap

• Chapter 5: Convolutions (Dec/2020)

◦ Spoilers

◦ Jupyter Notebook

◦ Convolutions

    ▪ Filter/Kernel

    ▪ Convolving

    ▪ Moving Around

    ▪ Shape

    ▪ Convolving in PyTorch

    ▪ Striding

    ▪ Padding

    ▪ A REAL Filter

◦ Pooling

◦ Flattening

◦ Dimensions

◦ Typical Architecture

◦ A Multiclass Classification Problem

    ▪ Data Generation

    ▪ Data Preparation

    ▪ Loss

    ▪ Classification Losses Showdown!

    ▪ Model Configuration

    ▪ Model Training

◦ Visualizing Filters and More!

    ▪ Static Method

    ▪ Visualizing Filters

    ▪ Hooks

    ▪ Visualizing Feature Maps

    ▪ Visualizing Classifier Layers

    ▪ Accuracy

    ▪ Loader Apply

◦ Putting It All Together

◦ Recap

• Chapter 6: Rock, Paper, Scissors (Jan/2021)

◦ Spoilers

◦ Jupyter Notebook

◦ Rock, Paper, Scissors...

◦ Data Preparation

    ▪ ImageFolder

    ▪ Standardization

    ▪ The Real Datasets

◦ Three-Channel Convolutions

◦ Fancier Model

◦ Dropout

◦ Model Configuration

◦ Model Training

◦ Learning Rates

    ▪ Finding LR

    ▪ Adaptive Learning Rate

    ▪ Stochastic Gradient Descent (SGD)

          Momentum

          Nesterov

          Flavors of SGD

    ▪ Learning Rate Schedulers

    ▪ Adaptive vs Cycling

◦ Putting It All Together

◦ Recap

• Chapter 7: Transfer Learning (Feb/2021)

◦ Spoilers

◦ Jupyter Notebook

◦ Transfer Learning

◦ ImageNet

◦ ImageNet Large Scale Visual Recognition Challenge (ILSVRC)

◦ Transfer Learning in Practice

    ▪ Pre-Trained Model

    ▪ Model Configuration

    ▪ Data Preparation

    ▪ Model Training

    ▪ Generating a Dataset of Features

    ▪ Top Model

◦ Auxiliary Classifiers (Side-Heads)

◦ 1x1 Convolutions

◦ Inception Modules

◦ Batch Normalization

    ▪ Running Statistics

    ▪ Evaluation Phase

    ▪ Momentum

    ▪ BatchNorm2d

    ▪ Other Normalizations

    ▪ Small Summary

◦ Vanishing Gradients & Initialization Schemes

◦ Residual Connections

      Learning the Identity

      The Power of Shortcuts

      Residual Blocks

◦ Putting It All Together

      Fine-Tuning

      Feature Extraction

◦ Recap

• Part III: Sequences and NLP (working titles - from Mar/2021)

◦ Chapter 8: Recurrent Neural Networks

◦ Chapter 9: Language Models and Embeddings

◦ Chapter 10: Down the Rabbit Hole

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/dvgodoy/PyTorchStepByStep/issues/6#issuecomment-726154948, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHGGHH37OUM5QLWCREWWNDSPP6LVANCNFSM4TS7K3FQ .

[dvgodoy]

Hi Tom,

I am glad you're enjoying the tutorials, please let me know if you find that some given concept or topic could benefit from additional explanation. I can relate to you - I learned programming using a Sinclair ZX-81 and a TRS-80 III in the 80s when I was a kid. At first, there were not even floppy disks, the programs were saved to a cassette. I've only seen the punched cards in college, though.

You're right about people throwing computational power at problems... the current bottleneck for users trying to learn and/or apply deep learning on their own is budget: GPUs or cloud time are somewhat expensive, especially in low-income countries. I've never heard of people using FPGA's for this purpose - but I think they were used in an early stage of the cryptocurrency mining - I wonder how performance/cost would compare to using GPUs.

Anyway, welcome to the world of deep learning! There are so many interesting things to pursue in this field that the real problem is to choose what to do next :-)

Best, Daniel

[minertom]

Hi Daniel,

Thank you for the reply. I believe that a properly configured Systolic Array architecture could be parallelized in an FPGA in order to do much of the heavy convolutional lifting. So, in effect, you have a serialized and parallelized architecture that seems tailor made for CNN's. Anything at all would use less power than a graphics card GPU. Perhaps a SBC like the Nvidia Jetson capabilities could be improved with better utilization of resources. When I can prove this, I will write the paper describing the algorithm.

BTW, yes, I learned in college on card readers. A nightmare. A room full of punch card machines, of which, 50% were out of service at any one time. Waiting on line for one to free up, perhaps to even revise one punch card. Then you are at the tender mercies of the card reader, which tended to "munch" cards which then had to be re-punched. And then, to the other side of the building, awaiting the output on large computer paper, that is, if the paper supply did not run out. Those were the days. NOT.

BTW-V2. I remember the Z80. If I am not mistaken, it had two accumulators. That made assembly language programming more fun.

Regards Tom

On Fri, Nov 13, 2020 at 10:58 AM Daniel Voigt Godoy < notifications@github.com> wrote:

Hi Tom,

I am glad you're enjoying the tutorials, please let me know if you find that some given concept or topic could benefit from additional explanation. I can relate to you - I learned programming using a Sinclair ZX-81 and a TRS-80 III in the 80s when I was a kid. At first, there were not even floppy disks, the programs were saved to a cassette. I've only seen the punched cards in college, though.

You're right about people throwing computational power at problems... the current bottleneck for users trying to learn and/or apply deep learning on their own is budget: GPUs or cloud time are somewhat expensive, especially in low-income countries. I've never heard of people using FPGA's for this purpose - but I think they were used in an early stage of the cryptocurrency mining - I wonder how performance/cost would compare to using GPUs.

Anyway, welcome to the world of deep learning! There are so many interesting things to pursue in this field that the real problem is to choose what to do next :-)

Best, Daniel

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/dvgodoy/PyTorchStepByStep/issues/6#issuecomment-726973050, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHGGHG5VL5YGBDQGYTMAFDSPV6TTANCNFSM4TS7K3FQ .