Fix logic of enabling in-place byte swap

[wkliao]

This affects only when running on a Big Endian machine.

When byte swapping is not necessary, this commit avoids extra internal buffer allocation.