Add ability to remove listeners

[andrewda]

This would be super helpful for when we no longer want to receive updates from a listener.