Diff server closures on pull_request

[e1mo]

Also updated the triggers, only building on pushes to main since the rest will eventually be a PR to main, so that the we can ditch the avoid duplicates action.