Remove remaining unsafe code

[joshlf]

This removes the remaining unsafe code (note that the README says that the crate is written in "safe Rust"). It does so at the cost of adding a new dependency, so feel free to just close this if it's determined to be not worth the tradeoff.

[Manishearth]

Yeah unfortunately this is at the root of a number of deptrees so we're somewhat careful about deps. Fine to update the readme to say "except for one function" and link to it.