search

sash-a / CoDeepNEAT

An implementation of CoDeepNEAT using pytorch with extensions
MIT License
29 stars 10 forks source link

Undercounting Batches in Accuracy Calculation #2

Open ztosi opened 2 years ago

ztosi commented 2 years ago

Hi there, nice project! I've actually been using it as a base for some of my own work on scaling CDN on HPC systems. In doing that I noticed a bug, though:

https://github.com/sash-a/CoDeepNEAT/blob/3476078a48986107ea1fc1a7ab1b42a55ac4f62f/src/phenotype/neural_network/evaluator/evaluator.py#L117

In test_nn where you calculate accuracy, you set the counter used as the denominator equal to the batch_idx, which undercounts the number of batches by one. In the limit, one can imagine a case where the entire dataset is a single batch and in that case "count" would be 0.

I was using a fairly large batch size and getting really inflated accuracy numbers.

sash-a commented 2 years ago

Hey sorry for getting back to you so late, I don't think I got a notification.

Good catch, I think the cleanest fix would probably be:

for batch_idx, (inputs, targets) in enumerate(test_loader, 1):

You can open a PR if you want :smile: