Add tests and error checking to custom encoders PR

[pablo-gar]

Add unit tests and potentially error checking for the case when the user-provided encoder is incompatible/incorrect to #1069