Fix up a panic if a fmt::Display itself failed during parsing

[KodrAus]

If an implementation of fmt::Display itself fails then we'll panic attempting to unwrap our error context. It might be time to look at some better basis for parsing than std::fmt, which makes tracking errors difficult.

[KodrAus]

I've also removed the few unsafe blocks too. They don't seem to make much of a difference, and since there's also generics involved if we ever decide to expose things like the text module they'll need to be rethought anyways.