Removes forced inlining and adds other optimizations

[aarondandy]

This also implements a few other bits of refactoring. It didn't seem to impact performance any but did simplify the code.