Project lock failure can update status of finished landing zone with rapid request spamming

[mikkonie]

I've witnessed the following on our production server:

• A project has multiple active landing zones
• A user rapidly spams ZoneSubmitMoveAPIView for all zones via the REST API
• Most requests end up setting the zone status to FAILED due to the project being locked, as only one moving taskflow at a time can be running per project
• One of the spam requests lands on zone X, there is no lock active and the move taskflow succeeds
• ..but in the end, the now moved zone still ends up as FAILED with a project lock error

Looking at the timeline for that zone, it seems we have received multiple requests for moving the zone after it has already finished.

It seems that when setting the failed status due to project lock, we don't look into the zone status?

No taskflow should be attempted on a finished zone to begin with and the UI/API view should catch that before the lock is even checked. But maybe there just have been so many near-simultaneous requests, that at the point where we fail the lock check, the zone status has not yet been updated anyway. This may be a tricky one. Maybe a queue solution like proposed in #1910 could help...

Current plan to tackle this:

• Ensure we don't allow submitting operations to zones with a busy status
• Add a check for current landing zone status when reporting lock failure
  • If status not in ACTIVE or FAILED, do not update
• ???

[mikkonie]

This should be, at least mostly, fixed. It's still possible that if there is an ongoing non-finished taskflow and near-simultaneous spam is done, we temporarily get a lock failure as the zone status. But even then, the ongoing task will eventually submit the correct status.

That would seem good enough, but if we identify cases where this still remains a problem, I'll reopen the ticket.