search

the-full-stack / fsdl-text-recognizer-2022

Source of the FSDL 2022 labs, which are at https://github.com/full-stack-deep-learning/fsdl-text-recognizer-2022-labs
https://fullstackdeeplearning.com/course
MIT License
81 stars 26 forks source link

Log input/output tensor stats #33

Open charlesfrye opened 2 years ago

charlesfrye commented 2 years ago

As noted in discussion here, there can be a difference between the visualization the content of a Tensor as human-interpretable media and the actual contents of that Tensor.

So it's useful to have the raw values logged. But the raw tensor values are fat, unstructured blobs -- for an image, taking up space like a high-resolution bitmap rather than like a png.

Following the principle that 20% of the information can catch 80% of the bugs, we should instead log descriptive statistics of the input, output, and target tensors. Considerations for this kind of logging are discussed in the (private) 2021 repo here and here.

charlesfrye commented 2 years ago

Prototyped this by having students implement it in an exercise and they found that training slowed down by a factor of 5. It should be possible to do this without slowing training and the torchmetrics code seems to be doing it the right way (applying running reduce operations), so this was surprising.