[Security Solution] There was an error uploading the value list / Data stream does not exist (400) error for a user with correct permissions

[andrew-goldstein]

A user reported they received an unexpected error (detailed below) while uploading a value list when both of the following are true:

• The user has a role that appears to have the correct permissions
• A (different) superuser has NOT previously imported a value list in the same space (before the user with fewer permissions attempts to upload a value list to the same space)

When this issue occurs, the following 400 (bad request) error is displayed:

There was an error uploading the value list.

To import a list item, the data steam must exist first. Data stream ".lists-try_again" does not exist (400)

{
  "name": "Error",
  "body": {
    "message": "To import a list item, the data steam must exist first. Data stream \".lists-try_again\" does not exist",
    "status_code": 400
  },
  "message": "Bad Request",
  "stack": "Error: Bad Request\n    at Fetch.fetchResponse (http://localhost:5601/9007199254740991/bundles/core/core.entry.js:15494:13)\n    at async interceptResponse (http://localhost:5601/9007199254740991/bundles/core/core.entry.js:15802:10)\n    at async http://localhost:5601/9007199254740991/bundles/core/core.entry.js:15391:39"
}

The error above appears in the screenshot below:

Workaround

To workaround for this error / issue:

1. Login to the Kibana space as a superuser (e.g. the elastic user)

2. Import a value list to the space while logged in as a superuser

3. Logout the superuser

4. Login to Kibana as a user with fewer permissions (to the same space)

5. Import a value list (to the same space) as a user with fewer permissions

Kibana/Elasticsearch Stack version:

• 8.8.2
• main

A user reported this issue in Kibana version 8.8.2. The error may also be reproduced in main via a local Kibana development environment, per the 50 reproduction steps in this issue.

Steps to reproduce:

1. Login to the Default space as the elastic user

2. Navigate to Stack Management

3. Click Spaces

4. Click Create space

5. Enter the following space name: try_again

6. Click Create space to create the try_again space

7. In Stack Management, click Roles

8. Click Create role

9. Enter the following role name: testing_value_list_role

10. In the Index privileges / Indices section of Create role, enter the following Indices:

.lists-try_again
.alerts-security.alerts-try_again
.alerts-security.alerts-try_again

11. In the Index privileges / Privileges section of Create role, enter the following Privileges:

read
write
write
view_index_metadata
maintenance
create
create_index
delete_index
index

12. In the Kibana section of Create role, click Add Kibana privilege

13. In the Kibana privileges flyout, select the try_again space

14. In the Kibana privileges flyout, click Bulk actions > All, to grant Custom access to a features

Expected result:

• The Kibana privileges flyout looks like the following screenshot:

15. Click Add Kibana privilege

Expected result:

• After the Add Kibana privilege flyout is closed, the Create role page now looks like the following screenshot:

16. Click the Create role button

Expected result:

• The new testing_value_list_role role is saved

17. In Stack Management, click Users

18. In Users, click Create user

19. Enter the following username: testing_value_list

20. Enter and confirm a password

21. Select the testing_value_list_role from the Roles dropdown

22. Click Create user

Expected result:

• The new testing_value_list user with the testing_value_list_role is created

23. Log out the elastic user from Kibana

24. Login to Kibana as the new testing_value_list user

25. Navigate to Security > Alerts

26. Click Manage rules

27. Click Import value lists

Expected result:

• The new Import value lists flyout is displayed

28. Click the Select or drag and drop a file button

29. On your local file system, create a file named joe_list.csv, with the following contents:

127.0.0.1

30. Select the joe_list.csv file in the web browser's Open dialog, then click the Open button

31. In the Import value lists flyout, select IP addresses from the Type of value list dropdown

Expected result:

• The new Import value lists flyout is looks like the following screenshot:

32. Open the browser's developer tools

33. Navigate to the browser developer tools Network tab

34. Clear the Network tab's history of previous requests

35. Click the Import value list button

Expected result:

• A new value list for joe_list.csv is created for the testing_value_list user in the try_again space

Actual results:

• An error toaster with the title There was an error uploading the value list. appears, as shown in the following screenshot:

• The error toast includes the following message: To import a list item, the data steam must exist first. Data stream ".lists-try_again" does not exist (400) appears, per the previous screenshot

• In the The Network tab, the POST request to http://localhost:5601/s/try_again/api/lists/items/_import?type=ip returned a 400 (bad request) status code

36. Click See the full error

Expected result:

• A detailed error similar to the one below is displayed:

{
  "name": "Error",
  "body": {
    "message": "To import a list item, the data steam must exist first. Data stream \".lists-try_again\" does not exist",
    "status_code": 400
  },
  "message": "Bad Request",
  "stack": "Error: Bad Request\n    at Fetch.fetchResponse (http://localhost:5601/9007199254740991/bundles/core/core.entry.js:15494:13)\n    at async interceptResponse (http://localhost:5601/9007199254740991/bundles/core/core.entry.js:15802:10)\n    at async http://localhost:5601/9007199254740991/bundles/core/core.entry.js:15391:39"
}

per the following screenshot:

37. On the local file system, make a copy of the file joe_list.csv. Name the new file tuesdays.csv.

38. Log out the testing_value_list user from Kibana

39. Once again, login to Kibana as the elastic (superuser)

40. Select the try_again space

41. Navigate to Security > Alerts

42. Click Manage rules

43. Click Import value lists

44. Click the Select or drag and drop a file button

45. Select the tuesdays.csv file, then click Open

46. In the Import value lists flyout, select IP addresses from the Type of value list dropdown

47. Click the Import value list button

Expected results:

• The POST request to http://localhost:5601/s/try_again/api/lists/items/_import?type=ip returned a 200 (success) status code
• The tuesdays.csv file was created for the elastic user, per the screenshot below:

48. Log out the elastic user from Kibana

49. Once again, Login to Kibana as the new testing_value_list user

50. One more time, attempt to upload joe_list.csv as the testing_value_list user

Expected result:

• joe_list.csv is created for the testing_value_list user
• Both joe_list.csv, created by the testing_value_list user, and tuesdays.csv, created by the elastic user, exist

Note that the issue may only be reproduced ONCE per newly created space. The issue may NOT be reproduced again for the same space, even that space is deleted, and another space with the same name is (re)created.

Server side error thrown from import_list_item_route.ts

The error appears to be thrown by the following code in x-pack/plugins/lists/server/routes/list/import_list_item_route:

            const listIndexExists = await lists.getListIndexExists();
            if (!listIndexExists) {
              return siemResponse.error({
                body: `To import a list item, the data steam must exist first. Data stream "${lists.getListName()}" does not exist`,
                statusCode: 400,
              });
            }

The above was determined by adding the following console.trace to the code above:

            if (!listIndexExists) {
              console.trace('--> A: listIndexExists', listIndexExists);

              return siemResponse.error({
                body: `To import a list item, the data steam must exist first. Data stream "${lists.getListName()}" does not exist`,
                statusCode: 400,
              });
            }
            // otherwise migration is needed
            await lists.migrateListIndexToDataStream();
          }

The following console.trace was logged (on the server):

Trace: --> A: listIndexExists false
    at Object.fn (import_list_item_route.ts:58:23)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at core_versioned_route.ts:179:22
    at Router.handle (router.ts:228:30)
    at handler (router.ts:162:13)
    at exports.Manager.execute (/Users/andrew.goldstein/Projects/forks/andrew-goldstein/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/Users/andrew.goldstein/Projects/forks/andrew-goldstein/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/Users/andrew.goldstein/Projects/forks/andrew-goldstein/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/Users/andrew.goldstein/Projects/forks/andrew-goldstein/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
    at Request._execute (/Users/andrew.goldstein/Projects/forks/andrew-goldstein/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[vitaliidm]

The role setup does not have all required privileges

According to https://www.elastic.co/guide/en/security/current/detections-permissions-section.html, it needs cluster privilege manage and index privilege manage, which were not added in 11 step of reproduction. When I added manage to both index and cluster, it started to work as expected.

The component, responsible to display warning, is out of date: https://github.com/elastic/kibana/blob/8.11/x-pack/plugins/security_solution/public/detections/components/callouts/missing_privileges_callout/use_missing_privileges.ts#L14

It checks for wrong privileges on index and seems does do any checks on cluster privilege. So, as solution I see, we need to update that component with the right index privileges and also start to show missing cluster privilege as well

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)