Handle security patches that don't have a pull request

[directionless]

When we merge through a security release, there is no pull request. This adds some simple, if imperfect, workarounds for this