floor collisions can probably be solved by updating the ray casting approach against voxels to also ray cast into the BVH's for the triangle meshes you have imported
wall collisions can be solved by performing forward raycasting in blast mode, and disabling that in build mode to make it so you can avoid getting yourself trapped