Is there a way to set the owner and owning group of an azure storage file and directory

[Mark-Knutson]

Is your feature request related to a problem? Please describe. The sdk documentation is not complete, and I can't find a python based function to set the owner and owning group of a file/directory in an azure storage account.

Describe the solution you'd like Documentation of an api call to set the owner and owning group of an azure storage directory and file using python. This would include how to reference $superuser in an such an api call.

Describe alternatives you've considered I am able to do this in powershell, but there is enormous convenience in doing it using python. I was wondering if this could be passed as part of the acl to set_access_control, but I cannot find documentation that indicates all of the options of this method.

Additional context Here is how I do this in powershell: Update-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystem -path $file.Name -Owner "`$superuser"

[github-actions[bot]]

Thank you for your feedback. This has been routed to the support team for assistance.

[navba-MSFT]

@Mark-Knutson Apologies for the late reply. Thanks for reaching out to us and reporting this issue. Could you please check if the below sample helps ?

from azure.storage.file import DataLakeServiceClient
from azure.storage.file import ContentSettings

# Retrieve the account connection string from the environment variable
# AZURE_STORAGE_CONNECTION_STRING.
conn_str = os.getenv('AZURE_STORAGE_CONNECTION_STRING')

# Create the service client with the connection string
service = DataLakeServiceClient.from_connection_string(conn_str=conn_str)

# Create a file system client
file_system_client = service.get_file_system_client(file_system="filesystem1")

# Create a directory client
dir_client = file_system_client.get_directory_client(directory_path="dir1")

# Create a file client
file_client = dir_client.get_file_client(file_path="file1")

# Update the file with the new content settings
file_client.set_content_settings(
    content_settings=ContentSettings(
        content_type="image/jpeg",
        content_encoding="UDF8",
        content_language="EN-US",
        cache_control="READ",
        content_disposition="True",
        content_md5="i727sP7HigloQDsqadNLHw=="
    )
)

# Update the file with the new permissions
file_client.set_permissions(
    permissions="rw-rw-rwx",
    owner="$superuser",
    group="$superuser"
)

# Update the file with the new metadata
file_client.set_metadata(
    metadata={
        "tag1": "value1",
        "tag2": "value2"
    }
)

# Get the file
file = file_client.get_file_properties()

# Print the file properties
print("File properties:")
print(file)

# Get the file acl
acl = file_client.get_access_control()

# Print the file acl
print("File ACL:")
print(acl)

# Get the file permissions
permissions = file_client.get_permissions()

# Print the file permissions
print("File permissions:")
print(permissions)

# Get the file metadata
metadata = file_client.get_metadata()

# Print the file metadata
print("File metadata:")
print(metadata)

[github-actions[bot]]

Hi @Mark-Knutson. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[Mark-Knutson]

The script did not work for me. I note the following things I had to change to get it to run:

Imports changed to this: from azure.storage.filedatalake import DataLakeServiceClient from azure.storage.filedatalake._models import ContentSettings

got error messages on named parameters which were cured with positional parameters:

Create a directory client

dir_client = file_system_client.get_directory_client(directory_path="/")

dir_client = file_system_client.get_directory_client("/")

Create a file client

file_client = dir_client.get_file_client(file_path="folder_1")

file_client = dir_client.get_file_client("folder_1")

For the operative code, I got this error: file_client.set_permissions( owner="$superuser", group="$superuser" )

AttributeError: 'DataLakeFileClient' object has no attribute 'set_permissions'

I am running this in vscode. In this environment, I am able to set acl for files and directories. I noted that the group and owner showed up in the acl from get_access_control. The setting of properties and metadata did not appear to be relevant.

I also tried to include superuser in the acl string, and got a different error: 'user:\$superuser:rwx,group:\$superuser:r-x,other::---,mask::rwx' Message: The named user or named group in the access control list is not valid.

This option was just a guess on my part, not in the example sent.

Thanks.

From: navba-MSFT @.> Sent: Wednesday, April 19, 2023 1:12 AM To: Azure/azure-sdk-for-python @.> Cc: Mark C Knutson @.>; Mention @.> Subject: [External] Re: [Azure/azure-sdk-for-python] Is there a way to set the owner and owning group of an azure storage file and directory (Issue #29861)

CAUTION: This email was sent from outside of Hennepin County. Unless you recognize the sender and know the content, do not click links or open attachments.

@Mark-Knutsonhttps://github.com/Mark-Knutson Apologies for the late reply. Thanks for reaching out to us and reporting this issue. Could you please check if the below sample helps ?

from azure.storage.file import DataLakeServiceClient

from azure.storage.file import ContentSettings

Retrieve the account connection string from the environment variable

AZURE_STORAGE_CONNECTION_STRING.

conn_str = os.getenv('AZURE_STORAGE_CONNECTION_STRING')

Create the service client with the connection string

service = DataLakeServiceClient.from_connection_string(conn_str=conn_str)

Create a file system client

file_system_client = service.get_file_system_client(file_system="filesystem1")

Create a directory client

dir_client = file_system_client.get_directory_client(directory_path="dir1")

Create a file client

file_client = dir_client.get_file_client(file_path="file1")

Update the file with the new content settings

file_client.set_content_settings(

content_settings=ContentSettings(

    content_type="image/jpeg",

    content_encoding="UDF8",

    content_language="EN-US",

    cache_control="READ",

    content_disposition="True",

    content_md5="i727sP7HigloQDsqadNLHw=="

)

)

Update the file with the new permissions

file_client.set_permissions(

permissions="rw-rw-rwx",

owner="$superuser",

group="$superuser"

)

Update the file with the new metadata

file_client.set_metadata(

metadata={

    "tag1": "value1",

    "tag2": "value2"

}

)

Get the file

file = file_client.get_file_properties()

Print the file properties

print("File properties:")

print(file)

Get the file acl

acl = file_client.get_access_control()

Print the file acl

print("File ACL:")

print(acl)

Get the file permissions

permissions = file_client.get_permissions()

Print the file permissions

print("File permissions:")

print(permissions)

Get the file metadata

metadata = file_client.get_metadata()

Print the file metadata

print("File metadata:")

print(metadata)

- Reply to this email directly, view it on GitHubhttps://github.com/Azure/azure-sdk-for-python/issues/29861#issuecomment-1514183993, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2UYQMBHTIOKUYXLVASYOJ3XB564LANCNFSM6AAAAAAW4CZ2BY. You are receiving this because you were mentioned.Message ID: @.**@.>>

Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system.

[navba-MSFT]

@Mark-Knutson Thanks for your reply. Apologies for the confusion. Here is the same code to set the ACL on directory and file in Azure datalake storage account.

import os, uuid, sys
from azure.storage.filedatalake import DataLakeServiceClient
from azure.core._match_conditions import MatchConditions
from azure.storage.filedatalake._models import ContentSettings

# Set the Storage account connection string from the environment variable
conn_str = "XXXXXXXXXXXXX"

# Create the service client with the connection string
service_client = DataLakeServiceClient.from_connection_string(conn_str=conn_str)

# Create a file system client
file_system_client = service_client.get_file_system_client(file_system="temp")

#try:
# Create a directory client
dir_client = file_system_client.create_directory("test1")

# Create a file client
file_client = dir_client.get_file_client("data.xlsx")

# set the permissions of the parent directory
new_dir_permissions = 'rwx------'
dir_client.set_access_control(permissions=new_dir_permissions)

# Set and display the Owner and Owner Group
dir_client.set_access_control(owner="OwnerOID", group="MyGroupName")
acl_props = dir_client.get_access_control()
print("New permissions of directory '{}' are {}.".format("test1", acl_props['permissions']))

file_client.set_access_control(permissions=new_dir_permissions)

print("Set the permissions of file '{}' to {}.".format("data.xlsx", new_dir_permissions))

If you want to add a complex acl setting, you can see the below syntax and pass this as a parameter to set_access_control():

acl = 'user::rwx,group::r-x,other::r--,user:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:r--'

set_access_control(acl=acl)

More Info here.

You need to tweak the above code using the set_access_control to add the owner and group to your directory / file:

set_access_control(owner=None, group=None, permissions=None, acl=None, kwargs)**

owner (str) – Optional. The owner of the file or directory.

group (str) – Optional. The owning group of the file or directory.

permissions (str) – Optional and only valid if Hierarchical Namespace is enabled for the account. Sets POSIX access permissions for the file owner, the file owning group, and others. Each class may be granted read, write, or execute permission. The sticky bit is also supported. Both symbolic (rwxrw-rw-) and 4-digit octal notation (e.g. 0766) are supported. permissions and acl are mutually exclusive.

acl (str) – Sets POSIX access control rights on files and directories. The value is a comma-separated list of access control entries. Each access control entry (ACE) consists of a scope, a type, a user or group identifier, and permissions in the format “[scope:][type]:[id]:[permissions]”. permissions and acl are mutually exclusive.

More Info here.

[github-actions[bot]]

Hi @Mark-Knutson. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[Mark-Knutson]

Excellent, just one more request..

Good progress here, thanks. I am able to use the set_access_control to set the owner, group, and acl. I am able to use set_access_control_recursive to set the acl-as I did before, but it does not appear to take the owner and group parameters.

One last request: Adding the owner and group capability to the r set_access_control_recursive api call would be a huge process improvement for us.

I also note that set_access_recursive does not appear to be documented in

azure.storage.filedatalake package - Azure SDK for Python 2.0.0 documentation (windows.net)https://azuresdkdocs.blob.core.windows.net/$web/python/azure-storage-file-datalake/12.0.0/azure.storage.filedatalake.html#azure.storage.filedatalake.DataLakeFileClient.set_access_control

This api documentation is great, and I have not seen it before. It never came up when I did internet searches on keywords such as 'set_access_control python'. Is there a more determinate way I can find this sort of api documentation?

The python azure storage apis, as constituted, are elegant and allow simple code to exercise them. Being able to implement these functions in python has allowed me to put them in databricks which represents a significant process improvement for team use of the functions. I have written scripts which permission an entire storage account container with specified ad groups in the acl. The databricks notebook metaphor is perfect for that.

I don't know how the regular and recursive functions (methods) look under the hood, but as a coder I might have investigated the possibility of a recursive function where the depth could be specified. Given such a function, set_access_control would be a special case of the recursive one with a depth set to 0 or 1. Not requesting that, just making an observation.

From: navba-MSFT @.> Sent: Thursday, April 20, 2023 9:07 AM To: Azure/azure-sdk-for-python @.> Cc: Mark C Knutson @.>; Mention @.> Subject: [External] Re: [Azure/azure-sdk-for-python] Is there a way to set the owner and owning group of an azure storage file and directory (Issue #29861)

CAUTION: This email was sent from outside of Hennepin County. Unless you recognize the sender and know the content, do not click links or open attachments.

@Mark-Knutsonhttps://github.com/Mark-Knutson Thanks for your reply. Apologies for the confusion. Here is the same code to set the ACL on directory and file in Azure datalake storage account.

import os, uuid, sys

from azure.storage.filedatalake import DataLakeServiceClient

from azure.core._match_conditions import MatchConditions

from azure.storage.filedatalake._models import ContentSettings

Set the Storage account connection string from the environment variable

conn_str = "XXXXXXXXXXXXX"

Create the service client with the connection string

service_client = DataLakeServiceClient.from_connection_string(conn_str=conn_str)

Create a file system client

file_system_client = service_client.get_file_system_client(file_system="temp")

try:

Create a directory client

dir_client = file_system_client.create_directory("test1")

Create a file client

file_client = dir_client.get_file_client("data.xlsx")

set the permissions of the parent directory

new_dir_permissions = 'rwx------'

dir_client.set_access_control(permissions=new_dir_permissions)

Set and display the Owner and Owner Group

dir_client.set_access_control(owner="OwnerOID", group="MyGroupName")

acl_props = dir_client.get_access_control()

print("New permissions of directory '{}' are {}.".format("test1", acl_props['permissions']))

file_client.set_access_control(permissions=new_dir_permissions)

print("Set the permissions of file '{}' to {}.".format("data.xlsx", new_dir_permissions))

If you want to add a complex acl setting, you can see the below syntax and pass this as a parameter to set_access_control():

acl = 'user::rwx,group::r-x,other::r--,user:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:r--'

set_access_control(acl=acl)

More Info herehttps://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-acl-python#set-the-acl-of-a-file.

You need to tweak the above code using the set_access_control to add the owner and group to your directory / file:

**set_access_control(owner=None, group=None, permissions=None, acl=None, kwargs)

owner (strhttps://docs.python.org/3.8/library/stdtypes.html#str) - Optional. The owner of the file or directory.

group (strhttps://docs.python.org/3.8/library/stdtypes.html#str) - Optional. The owning group of the file or directory.

permissions (strhttps://docs.python.org/3.8/library/stdtypes.html#str) - Optional and only valid if Hierarchical Namespace is enabled for the account. Sets POSIX access permissions for the file owner, the file owning group, and others. Each class may be granted read, write, or execute permission. The sticky bit is also supported. Both symbolic (rwxrw-rw-) and 4-digit octal notation (e.g. 0766) are supported. permissions and acl are mutually exclusive.

acl (strhttps://docs.python.org/3.8/library/stdtypes.html#str) - Sets POSIX access control rights on files and directories. The value is a comma-separated list of access control entries. Each access control entry (ACE) consists of a scope, a type, a user or group identifier, and permissions in the format "[scope:][type]:[id]:[permissions]". permissions and acl are mutually exclusive.

More Info herehttps://azuresdkdocs.blob.core.windows.net/$web/python/azure-storage-file-datalake/12.0.0/azure.storage.filedatalake.html#azure.storage.filedatalake.DataLakeFileClient.set_access_control.

- Reply to this email directly, view it on GitHubhttps://github.com/Azure/azure-sdk-for-python/issues/29861#issuecomment-1516395055, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2UYQMDENP4BEHE5WUCAAJTXCE7HFANCNFSM6AAAAAAW4CZ2BY. You are receiving this because you were mentioned.Message ID: @.**@.>>

Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system.

[navba-MSFT]

@Mark-Knutson Thanks for the update. Regarding your below pending asks:

1. Set recursive ACL via Python SDK: Refer the below sample code which calls set_access_control_recursive() function and update the right ACL permissions:

def set_permission_recursively(is_default_scope):

    try:
        file_system_client = service_client.get_file_system_client(file_system="my-container")

        directory_client = file_system_client.get_directory_client("my-parent-directory")

        acl = 'user::rwx,group::r-x,other::r--,user:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:r--'   

        if is_default_scope:
           acl = 'default:user::rwx,default:group::r-x,default:other::r--,default:user:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:r--'

        directory_client.set_access_control_recursive(acl=acl)

        acl_props = directory_client.get_access_control()

        print(acl_props['permissions'])

    except Exception as e:
     print(e)

More Info here.

2. Regarding the documentation improvement, I have created a workitem and shared the feedback internally with the Product owners. They will look into the docs enhancement request and track it separately.

[Mark-Knutson]

Thank-you for your attention to this. From my previous email, I was inquiring about setting the owner in a recursive manner. I already have code implemented that sets ACL recursively (but not owner and group) and received the sdk documentation for that earlier-and thanks for that assistance.

I can see that part of my previous email could be interpreted as a request for documentation on the recursive acl setting, but what I meant to say is that setting the owner and group was not documented in the set_access_control_recursive documentation-and maybe it has not been implemented yet. What would be great here is the ability to owning user and group to $superuser in a recursive api call. I am able to set it now with the non-recursive api call.

"Excellent, just one more request..

Good progress here, thanks. I am able to use the set_access_control to set the owner, group, and acl. I am able to use set_access_control_recursive to set the acl-as I did before, but it does not appear to take the owner and group parameters.

One last request: Adding the owner and group capability to the r set_access_control_recursive api call would be a huge process improvement for us.

I also note that set_access_recursive does not appear to be documented in

azure.storage.filedatalake package - Azure SDK for Python 2.0.0 documentation (windows.net)https://azuresdkdocs.blob.core.windows.net/$web/python/azure-storage-file-datalake/12.0.0/azure.storage.filedatalake.html#azure.storage.filedatalake.DataLakeFileClient.set_access_control "

From: navba-MSFT @.> Sent: Sunday, April 23, 2023 11:26 PM To: Azure/azure-sdk-for-python @.> Cc: Mark C Knutson @.>; Mention @.> Subject: [External] Re: [Azure/azure-sdk-for-python] Is there a way to set the owner and owning group of an azure storage file and directory (Issue #29861)

CAUTION: This email was sent from outside of Hennepin County. Unless you recognize the sender and know the content, do not click links or open attachments.

@Mark-Knutsonhttps://github.com/Mark-Knutson Thanks for the update. Regarding your below pending asks:

1. Set recursive ACL via Python SDK: Refer the below sample code which calls set_access_control_recursive() function and update the right ACL permissions:

def set_permission_recursively(is_default_scope):

try:

    file_system_client = service_client.get_file_system_client(file_system="my-container")

    directory_client = file_system_client.get_directory_client("my-parent-directory")

    acl = 'user::rwx,group::r-x,other::r--,user:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:r--'

    if is_default_scope:

       acl = 'default:user::rwx,default:group::r-x,default:other::r--,default:user:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:r--'

    directory_client.set_access_control_recursive(acl=acl)

    acl_props = directory_client.get_access_control()

    print(acl_props['permissions'])

except Exception as e:

 print(e)

More Info herehttps://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-acl-python#set-acls-recursively.

1. Regarding the documentation improvement, I have created a workitem and shared the feedback internally with the Product owners. They will look into the docs enhancement request and track it separately.

- Reply to this email directly, view it on GitHubhttps://github.com/Azure/azure-sdk-for-python/issues/29861#issuecomment-1519367641, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2UYQMHWSPGG5EPEYAZD7GDXCX6FLANCNFSM6AAAAAAW4CZ2BY. You are receiving this because you were mentioned.Message ID: @.**@.>>

Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system.