[📍] Handle concurrency

[dave-2]

Right now it's not safe for two requests to modify job at the same time. There are a number of cases where we want this:

• User interaction on running jobs. E.g. stopping a running job. (#4341)
• Data migration for running jobs.
• To support Cloud Pub/Sub instead of a taskqueue polling loop, for greater efficiency and reliability (#3900)

As far as I know, App Engine doesn't provide any synchronization primitives other than datastore transactions. So we'll need to figure out how to work with just those?

Task queues also have a max_concurrent_requests option which we can set, but that may limit our scalability.

@anniesullie @perezju @simonhatch

[simonhatch]

Datastore supports strong consistency and has some wrappers around atomic operations like transactions.

You could also just defer operations until the start of a job run.

[dave-2]

Deferring operations may work for user interactions but not cloud pub/sub.