[BUG] lost update while concurrent synthetics private location API calls

[biscout42]

Kibana version: 8.14.3 , 8.15.0

Elasticsearch version: 8.14.3 , 8.15.0

Original install method (e.g. download page, yum, from source, etc.): Docker compose

Describe the bug: Two concurrent private location create requests leads to first of them is lost.

In case of concurrent tests which create private locations, one of the locations is lost. Here is an example of kibana debug log:

[2024-08-20T11:17:06.931+00:00][DEBUG][elasticsearch.query.data] 201 - 220.0B
PUT /.kibana_8.15.0/_doc/synthetics-privates-locations%3Asynthetics-privates-locations-singleton?refresh=wait_for&require_alias=true
{"synthetics-privates-locations":{"locations":[{"label":"monitor-pll-kuooipy79h3a3gnw2wydly","id":"fbb70664-2851-473f-880a-42ca4f8269a1","agentPolicyId":"fbb70664-2851-473f-880a-42ca4f8269a1","isServiceManaged":false}]},"type":"synthetics-privates-locations","references":[],"managed":false,"coreMigrationVersion":"8.8.0","typeMigrationVersion":"10.1.0","updated_at":"2024-08-20T11:17:06.635Z","created_at":"2024-08-20T11:17:06.635Z"}
[2024-08-20T11:17:06.934+00:00][DEBUG][http.server.response] POST /api/synthetics/private_locations 200 319ms - 210.0B
[2024-08-20T11:17:06.934+00:00][DEBUG][elasticsearch.query.data] 200 - 220.0B
PUT /.kibana_8.15.0/_doc/synthetics-privates-locations%3Asynthetics-privates-locations-singleton?refresh=wait_for&require_alias=true
{"synthetics-privates-locations":{"locations":[{"label":"pl-test-label","id":"0e6c28d6-148a-47cb-b6fd-def77d3a795c","agentPolicyId":"0e6c28d6-148a-47cb-b6fd-def77d3a795c","tags":["a","b"],"isServiceManaged":false,"geo":{"lat":42.42,"lon":-42.42}}]},"type":"synthetics-privates-locations","references":[],"managed":false,"coreMigrationVersion":"8.8.0","typeMigrationVersion":"10.1.0","updated_at":"2024-08-20T11:17:06.634Z","created_at":"2024-08-20T11:17:06.634Z"}
[2024-08-20T11:17:06.935+00:00][DEBUG][http.server.response] POST /api/synthetics/private_locations 200 321ms - 239.0B

After that follow-up request for private location monitor-pll-kuooipy79h3a3gnw2wydly fails. In this example it happened with an attempt to create a monitor for this private location (see error below), and I have another tests, where follow-up request GET /api/synthetics/private_locations/locationId fails with 404. Error example:

{
 "statusCode": 400,
 "error": "Bad Request",
 "message": "Invalid locations specified. Private Location(s) 'monitor-pll-kuooipy79h3a3gnw2wydly' not found. Available private locations are 'pl-test-label'"
}

API client sees correct 200 reply after creating monitor-pll-kuooipy79h3a3gnw2wydly private location in the example above (via POST, with all its details in the body)

Steps to reproduce:

1. Run several POST /api/synthetics/private_locations requests concurrently.
2. Run GET /api/synthetics/private_locations/location_id for each of created locations
3. One of GET requests fails.

Expected behavior: All private locations exist in Kibana in case of concurrent POST requests.

Provide logs and/or server output (if relevant): see above

Any additional context: The root cause could be in our private location API implementation, which uses a single saved object to keep track of private locations.

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[smith]

@jughosta Synthetics is owned by @elastic/obs-ux-management-team. I'll add the team label.