Having looked at the file, I see three worrisome factors:
The pages have a CropBox that is smaller than the MediaBox. This box is what the device uses to set the window, I expect. But our code uses the MediaBox to figure out the scale and landscape/portrait logic.
The CropBox doesn't have a corner at (0,0). I'm not sure yet, but I suspect this may be leading to an offset of the annotations.
The pages all have a rotation of 270, just to confuse everything further.
I don't think being landscape actually plays into this at all.
As reported in rschroll/rmfuse#3.