Keyboard weirdly shrink to fingertip position when sliding on the emoji keyboard

Describe the bug Keyboard weirdly shrink to fingertip position when sliding on the emoji keyboard. Additionally, after the InputBarAccessoryView is shown once, when focusing on other normal textField etc, sliding on the emoji keyboard would cause the same problem to occur (it works normally if InputBarAccessoryView is never shown)

To Reproduce Steps/code to reproduce the behavior:

    lazy var messageInputBar = InputBarAccessoryView()
    override var canBecomeFirstResponder: Bool {
        return true
    }

    override open var inputAccessoryView: UIView? {
        return messageInputBar
    }

Just focus on the input bar, change to the system emoji keyboard, and slide to the left on the emojis

Expected behavior emojis slide and you can select emojis in the next pages

Environment

What version of InputBarAccessoryView are you using? 5.3.1
What version of iOS are you running on? 14.7.1
What version of Swift are you running on? 5.4
What device(s) are you testing on? Are these simulators? Different devices & simulators all having the same issue
Is the issue you're experiencing reproducable in the example app? Can't run example app

nathantannar4 / InputBarAccessoryView

Keyboard weirdly shrink to fingertip position when sliding on the emoji keyboard #216