Bug: User with ~100 classification groups cannot view submissions

[simonbjorzen-ts]

Describe the bug We are controlling access to information in AssemblyLine using the classfication engine. When logged on as an user with about 100 classifcation groups, an exception is thrown.

Error message:

File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/flask/app.py", line 2190, in wsgi_app response = self.full_dispatch_request() File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/flask/app.py", line 1486, in full_dispatch_request rv = self.handle_user_exception(e) File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/flask/app.py", line 1484, in full_dispatch_request rv = self.dispatch_request() File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/flask/app.py", line 1469, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(view_args) File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/assemblyline_ui/api/base.py", line 181, in base return func(*args, *kwargs) File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/assemblyline_ui/api/v4/submission.py", line 261, in get_file_tree return make_api_response(STORAGE.get_or_create_file_tree(data, config.submission.max_extraction_depth, File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/elasticapm/traces.py", line 1065, in decorated return func(args, kwds) File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/assemblyline/datastore/helper.py", line 716, in get_or_create_file_tree self.submission_tree.save(cache_key, cached_tree) File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/assemblyline/datastore/collection.py", line 1015, in save self.with_retries( File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/assemblyline/datastore/collection.py", line 306, in with_retries return self.datastore.with_retries(func, *args, kwargs) File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/assemblyline/datastore/store.py", line 309, in with_retries ret_val = func(*args, *kwargs) File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped return api(args, kwargs) File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/elasticsearch/_sync/client/init.py", line 2264, in index return self.perform_request( # type: ignore[return-value] File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( BadRequestError: BadRequestError(400, 'action_request_validation_exception', 'Validation Failed: 1: id [REDACTED_TLPA+SREL -- REDACTED, A lot of groups --] is too long, must be no longer than 512 bytes but was: 1211;')

To Reproduce Steps to reproduce the behavior:

1. Log on as an user with ~100 classification groups and view a submission

Expected behavior Access to the submission

Screenshots N/A

Environment (please complete the following information if pertinent):

• Assemblyline Version: 4.4.0.87
• Browser: chromium/edge

Additional context N/A

[cccs-sgaron]

Do you really need that many groups? This must break so much of the interface. I'm just trying to understand why you'd want data to be seperated is this many groups and why one user would be member of them all.

[simonbjorzen-ts]

Unfortunately, yes. Submissions only have one or two classifications at a time, so no issues with the UI. If this error is what it looks like; a limitation with ElasticSearch, I assume that it will be hard to fix.

The user that is a member of all groups is usually only accessed via API, individual users usually have ~10 groups.

[cccs-sgaron]

Instead of making a user part of all groups can you not just add a required group and make the API user part of that group? Required groups are automatically added to the submission as soon as a group is selected so this way it would ensure your API user always as access to the data.

[simonbjorzen-ts]

I don't think that's possible since the groups are dynamic groups, and not defined in the classifications config?

[cccs-sgaron]

Yeah so what I'd do is define that extra group in the classification.yml and set it to be a required group. Then make sure the API user is not part of any of the other groups. There is no issue setting up groups in the classification.yml and mixing this with the dynamic groups.

[cccs-rs]

Should be resolved in the latest release. If the issue still persists, feel free to reopen the issue 😀

[simonbjorzen-ts]

Hey, this does not appear to be resolved. Perhaps what I'm really after is a role/permission that allows global reading of all submissions, ignoring classification rules.

It also seems like I cannot re-open an issue.