Fix: avoid panic by never closing channel twice

[yunginnanet]

we need to never close that channel twice, that's an easy way to catch a stacktrace (as I have been while testing with this library)

use same logic we do for init (sync.Once) in Server with our closure of the c.closer channel in Conn