1.27: Backport destroy slot on worker thread2

[solo-changelog-bot[bot]]

Issues linked to changelog: https://github.com/solo-io/solo-projects/issues/6713 https://github.com/solo-io/envoy-gloo/pull/292

[ashishb-solo]

/skip-ci

i was gonna say, let's just let it run through. then i realized that ci takes 3 hours in this repository 👎

[nfuden]

/skip-changelog