search

phlippe / uvadlc_notebooks

Repository of Jupyter notebook tutorials for teaching the Deep Learning Course at the University of Amsterdam (MSc AI), Fall 2023
https://uvadlc-notebooks.readthedocs.io/en/latest/
MIT License
2.59k stars 590 forks source link

[Not a Bug, Clarification Required] Validation Step for EBMs (Tutorial 8) #62

Closed jainraj closed 1 year ago

jainraj commented 2 years ago

Tutorial: 8

Describe the bug In the validation part of the code, it is mentioned that the validation/test step depends on what we want to do with the EBM. Could you elaborate a bit more on that? And is the current implementation done keeping in mind generation as an objective?

To Reproduce (if any steps necessary) Steps to reproduce the behavior:

  1. Go to 'https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial8/Deep_Energy_Models.html#Training-algorithm'
  2. Scroll down to 'validation_step()'
  3. See comments in the first 2 lines

Expected behavior More clarity on the statement.

phlippe commented 2 years ago

Hi, thanks for your question! You are correct that the current validation step is a 'cheap' implementation for checking the generation performance of an energy-based model. However, keep in mind that even though the model may be able to distinguish real- and fake-images, the generation performance can greatly vary from the perspective of a user. Better metrics would be to evaluate the energy-based model similar to what is used for GANs, e.g. FID or Inception Score. Alternatively, an application of energy-based models is to denoise images. Hence, you could also evaluate the model on how well it can denoise unseen test images. I've tried to clarify this in the new commit b1636c2a1931a42a80cd445f7ebe7748a6fdc1c6