I've found it really irritating mining something by switching to a tool with the toolbelt, and then a zombie walks up to me and i have to wait a solid second to be able to use my sword again even when i've swapped from my empty second hotbar slot to my first one which has the sword.
I think using the number keys or mouse wheel to change the selected hotbar slot should immediately swap off the current tool from the toolbelt.
I've found it really irritating mining something by switching to a tool with the toolbelt, and then a zombie walks up to me and i have to wait a solid second to be able to use my sword again even when i've swapped from my empty second hotbar slot to my first one which has the sword. I think using the number keys or mouse wheel to change the selected hotbar slot should immediately swap off the current tool from the toolbelt.