Remove some unnecessary tf.casts

[uri-granta]

Related issue(s)/PRs:

Summary

Remove some unnecessary casting, in particular in the various samplers.

Fully backwards compatible: yes

PR checklist

• [ ] The quality checks are all passing
• [ ] The bug case / new feature is covered by tests
• [ ] Any new features are well-documented (in docstrings or notebooks)