[Bug]: How to add read and write permissions for a partition to a user？

[become-nice]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version: 2.2.10
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):kafka    
- SDK version(e.g. pymilvus v2.0.0rc2):pymilvus 2.2.12
- OS(Ubuntu or CentOS): Ubuntu
- CPU/Memory: 2c8g
- GPU: 
- Others:

Current Behavior

  In the milvus document, milvus can support partition-level user read and write permission settings, but in fact I can’t set it.

Expected Behavior

No response

Steps To Reproduce

May I ask whether it currently does not support partition-level read and write permission settings or the method I used is wrong.The method I use is as follows.

from pymilvus import (
    connections,
    utility, Role,
    FieldSchema, CollectionSchema, DataType,
    Collection,
)

if __name__ == '__main__':
    connections.connect(host="xx.xx.xx.xx", port="19530",user="root",password="yyyyyy")

    # utility.create_user("user2", "testforpartition")
    role = Role("guset")
    # role.add_user("user2")
    role.grant("Partition", "MultiPartitionTest_partition1", "Query")

  The error message is as follows.

RPC error: [grant_privilege], <MilvusException: (code=41, message=not found the object type[name: Partition], supported the object types: [User Collection Global])>, <Time:{'RPC start': '2023-08-18 11:13:39.491542', 'RPC error': '2023-08-18 11:13:39.493285'}>
Traceback (most recent call last):
  File "/home/liupeng13/milvus/create_role.py", line 14, in <module>
    role.grant("Partition", "MultiPartitionTest_partition1", "Query")
  File "/home/liupeng13/.local/lib/python3.11/site-packages/pymilvus/orm/role.py", line 150, in grant
    return self._get_connection().grant_privilege(self._name, object, object_name, privilege)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/liupeng13/.local/lib/python3.11/site-packages/pymilvus/decorators.py", line 109, in handler
    raise e
  File "/home/liupeng13/.local/lib/python3.11/site-packages/pymilvus/decorators.py", line 105, in handler
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/liupeng13/.local/lib/python3.11/site-packages/pymilvus/decorators.py", line 136, in handler
    ret = func(self, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/liupeng13/.local/lib/python3.11/site-packages/pymilvus/decorators.py", line 85, in handler
    raise e
  File "/home/liupeng13/.local/lib/python3.11/site-packages/pymilvus/decorators.py", line 50, in handler
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/liupeng13/.local/lib/python3.11/site-packages/pymilvus/client/grpc_handler.py", line 1181, in grant_privilege
    raise MilvusException(resp.error_code, resp.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=41, message=not found the object type[name: Partition], supported the object types: [User Collection Global])>

  I also tried the Collection parameter, also can't get it to set.

Milvus Log

No response

Anything else?

No response

[xiaofan-luan]

we don't support partition level RBAC yet. but it's a good issue, mark it as good first issue

[wayblink]

@xiaofan-luan Considering large groups‘ advanced need like partition/row/column level auth control, they always develop their own auth service, self-developed or based on open source project like Apache Sentry/Apache Ranger. How about we provide auth interface hook? Users can implement their own hook. And we can provide enterprise auth service on cloud.

[xiaofan-luan]

@xiaofan-luan Considering large groups‘ advanced need like partition/row/column level auth control, they always develop their own auth service, self-developed or based on open source project like Apache Sentry/Apache Ranger. How about we provide auth interface hook? Users can implement their own hook. And we can provide enterprise auth service on cloud.

This is actually a good suggestion, we can support third party plugin for auth

[xiaofan-luan]

but this service has to be very fast, authed in less than 1-2 milliseconds

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[airwindk]

Hi - wanted to re-open this - is this being looked into? This would help support production applications for sensitive documents where each document has its own entitlement group

[xiaofan-luan]

we don't have partition level rbac. Why not use multiple collections?

[airwindk]

There a few reasons that come to mind

1) Each document has its own user permissions and collection level access control will not be sufficient

2) we tend to favor storing documents of similar schema in the same Milvus collection for enhanced  query performance  / retrieval

3) Milvus restricts the number of collections to ~65k and we’ve heard that performance degrades as more collections are added

[xiaofan-luan]

That's true. The problem here is even if you use partition there is still some limitation here. May i know how many partitions you want to create in total? Why not use partition key features

[airwindk]

Thanks for the reply. As of now we are looking at about 10k-15k distinct collections, each of which require their own access control but this could increase as users upload more collections / documents. Milvus currently doesn’t support partition level authentication, does it?

[xiaofan-luan]

no, even we support partition level auth, having 15K level partitions is not a choice.

[xiaofan-luan]

We are try to optimize milvus to achive 10K collections, but there is still a lot challenge engineering side

[xiaofan-luan]

for each collection, we support no more than 1024 partitions.

[airwindk]

So the 10k number referred to the total number of distinct partitions (documents) spread across combinations of collection / partitions. I think it would be more like 700-1000 collections (for each distinct user) and anywhere between 10-100 partitions per collection for each document a user has access to. But thank you so much for the response. Understood capacity constraints if it’s an engineering challenge.

[xiaofan-luan]

you can try so to do 700 -100 collections with 10-100 partitions. There seems to be risk and with latest 2.4 this could be improved already