Testing issues - Githubissues

neuroailab / tfutils

Utilities for working with tensorflow

MIT License

25 stars 8 forks source link

Testing issues #37

Closed qbilius closed 7 years ago

qbilius commented 7 years ago

This line https://github.com/neuroailab/tfutils/blob/master/tfutils/base.py#L414 is setting validation results to always be saved. Bug or is there some reasoning behind this?
When a filters file is not in cache and needs to be downloaded from DB, there is an untarring issue tarfile.ReadError: unexpected end of data.
"loading pointlessly" exception always happens because save_params is not passed in test. I think – not sure about the actual issue,

yamins81 commented 7 years ago

Yes, the reasoning behind this is: if you're doing validation, you always are wanting to save a record. (Namely, the metrics of the validation itself). The need_to_save variable doesn't make sense to determine based on the type of logic you use in the case of training. However, in the case of validation_only == True, filters are never being saved -- just metric results.

qbilius commented 7 years ago

Just like in training, there are times one will not want to save anything. Two cases in my mind: (i) debugging (since validation targets are computed using custom functions, debugging will often happen); (ii) general playing around just to see how a particular validation would work (e.g., have top1 and top5 computed but just wanted to get an idea what top10 was like).

Also, what would happen in the feature extraction case? It's running test_from_params, so I guess stuff will be saved. Very often when features are accessed, there is no need to save a record.

yamins81 commented 7 years ago

Hmm. I guess I always want a record to be be saved during feature extraction, since I'll want to save the features themselves. But we could have need_to_save modulated by do_save.

yamins81 commented 7 years ago

I'll pull the do_save gating outside that if clause

qbilius commented 7 years ago

I mean, there is a certain beauty in saving records always, for every interaction with tfutils. Maybe that should be the core philosophy for tfutils. In that case, the only valid case in my view when we could allow no saving is debug mode. This could be regulated via a command-line --debug flag. Thoughts?

yamins81 commented 7 years ago

Well, we do have do_save flag already, so we might as well just continue the logic to the testing case as well. I just actually pulled the do_save gating outside the if clause, so do_save will now be active for testing as well. Ok to close this issue for now?

qbilius commented 7 years ago

It's okay for now as a quick fix. I'll open a new issue to discuss the "save everything" option.