Azure / azure-cli

Azure Command-Line Interface
MIT License
3.97k stars 2.95k forks source link

ADLS Gen2 ACL set recursive #16792

Open sshah90 opened 3 years ago

sshah90 commented 3 years ago

Describe the bug

I am trying to give acl permission at root level directory and all child members by following this doc.

Command Name

az storage fs access set-recursive

Errors:

CLIInternalError: The command failed with an unexpected error. Here is the traceback:
(StorageErrorException('(SetAclMissingAces) Invalid ACL: the user, group and other entries are required.\nRequestId:4aabfb54-201f-0013-5ca3-fa8fe0000000\nTime:2021-02-04T03:11:32.7925451Z'), '(SetAclMissingAces) Invalid ACL: the user, group and other entries are required.\nRequestId:4aabfb54-201f-0013-5ca3-fa8fe0000000\nTime:2021-02-04T03:11:32.7925451Z', None)
Traceback (most recent call last):
  File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/multiapi/storagev2/filedatalake/v2020_02_10/_path_client.py", line 572, in _set_access_control_internal
    headers, resp = self._client.path.set_access_control_recursive(**options)
  File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/multiapi/storagev2/filedatalake/v2020_02_10/_generated/operations/_path_operations.py", line 1321, in set_access_control_recursive
    raise models.StorageErrorException(response, self._deserialize)
azure.multiapi.storagev2.filedatalake.v2020_02_10._generated.models._models_py3.StorageErrorException: (SetAclMissingAces) Invalid ACL: the user, group and other entries are required.
RequestId:4aabfb54-201f-0013-5ca3-fa8fe0000000
Time:2021-02-04T03:11:32.7925451Z

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 659, in execute
    raise ex
  File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 722, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 714, in _run_job
    return cmd_copy.exception_handler(ex)
  File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/cli/command_modules/storage/__init__.py", line 338, in new_handler
    raise ex
  File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 693, in _run_job
    result = cmd_copy(params)
  File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 330, in __call__
    return self.handler(*args, **kwargs)
  File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/cli/core/__init__.py", line 786, in default_command_handler
    return op(**command_args)
  File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/cli/command_modules/storage/operations/fs_directory.py", line 63, in set_access_control_recursive
    result = client.set_access_control_recursive(acl=acl, progress_hook=progress_callback, **kwargs)
  File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/multiapi/storagev2/filedatalake/v2020_02_10/_path_client.py", line 461, in set_access_control_recursive
    return self._set_access_control_internal(options=options, progress_hook=progress_hook,
  File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/multiapi/storagev2/filedatalake/v2020_02_10/_path_client.py", line 617, in _set_access_control_internal
    raise DataLakeAclChangeFailedError(error, error.message, last_continuation_token)
azure.multiapi.storagev2.filedatalake.v2020_02_10._models.DataLakeAclChangeFailedError: (StorageErrorException('(SetAclMissingAces) Invalid ACL: the user, group and other entries are required.\nRequestId:4aabfb54-201f-0013-5ca3-fa8fe0000000\nTime:2021-02-04T03:11:32.7925451Z'), '(SetAclMissingAces) Invalid ACL: the user, group and other entries are required.\nRequestId:4aabfb54-201f-0013-5ca3-fa8fe0000000\nTime:2021-02-04T03:11:32.7925451Z', None)
...

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

I am running the following command

az storage fs access set-recursive --acl "group:xxxxxx:rwx,group:xxxxx:r-x" -p / -f test --account-name xxxxx --account-key xxxx

Expected Behavior

Command Should grant access to the given AD group to the directory and all child members.

Environment Summary

macOS-10.15.7-x86_64-i386-64bit
Python 3.8.7
Installer: HOMEBREW

azure-cli 2.18.0

Extensions:
costmanagement 0.1.0

Additional Context

NA

yonzhan commented 3 years ago

storage

clementbosc commented 3 years ago

Same here !!

Juliehzl commented 3 years ago

@sshah90 you only need to specify one group in acl.

sshah90 commented 3 years ago

@Juliehzl still getting the same error

Juliehzl commented 3 years ago

@Juliehzl Zunli Hu FTE still getting the same error

May I know the acl string you are using?

sshah90 commented 3 years ago

az storage fs access set-recursive --acl "group:xxxx-b54c-xxxx-acd1-xxxx:rwx" -p /test -f test --account-name xxxx --account-key xxx

@Juliehzl I am using the above command

Juliehzl commented 3 years ago

Have you ever set acl in the path? You could check current acl status with az storage fs access show. If you want to change the permission for existing group, please use az storage fs access update-recursive.

sshah90 commented 3 years ago

In past I have used az storage fs access set on the root directory but never used set-recursive. (I ran az storage fs access show but didn't see my groups)

Though after running update-recursive I can see groups at directory permission.

I am still suspect some issue with set-recursive or maybe the error message needs to be a little more verbose.

ghost commented 3 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

Issue Details
## Describe the bug I am trying to give acl permission at root level directory and all child members by following [this](https://docs.microsoft.com/en-us/azure/storage/blobs/recursive-access-control-lists?tabs=azure-cli) doc. **Command Name** `az storage fs access set-recursive` **Errors:** ``` CLIInternalError: The command failed with an unexpected error. Here is the traceback: (StorageErrorException('(SetAclMissingAces) Invalid ACL: the user, group and other entries are required.\nRequestId:4aabfb54-201f-0013-5ca3-fa8fe0000000\nTime:2021-02-04T03:11:32.7925451Z'), '(SetAclMissingAces) Invalid ACL: the user, group and other entries are required.\nRequestId:4aabfb54-201f-0013-5ca3-fa8fe0000000\nTime:2021-02-04T03:11:32.7925451Z', None) Traceback (most recent call last): File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/multiapi/storagev2/filedatalake/v2020_02_10/_path_client.py", line 572, in _set_access_control_internal headers, resp = self._client.path.set_access_control_recursive(**options) File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/multiapi/storagev2/filedatalake/v2020_02_10/_generated/operations/_path_operations.py", line 1321, in set_access_control_recursive raise models.StorageErrorException(response, self._deserialize) azure.multiapi.storagev2.filedatalake.v2020_02_10._generated.models._models_py3.StorageErrorException: (SetAclMissingAces) Invalid ACL: the user, group and other entries are required. RequestId:4aabfb54-201f-0013-5ca3-fa8fe0000000 Time:2021-02-04T03:11:32.7925451Z During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/knack/cli.py", line 233, in invoke cmd_result = self.invocation.execute(args) File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 659, in execute raise ex File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 722, in _run_jobs_serially results.append(self._run_job(expanded_arg, cmd_copy)) File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 714, in _run_job return cmd_copy.exception_handler(ex) File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/cli/command_modules/storage/__init__.py", line 338, in new_handler raise ex File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 693, in _run_job result = cmd_copy(params) File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 330, in __call__ return self.handler(*args, **kwargs) File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/cli/core/__init__.py", line 786, in default_command_handler return op(**command_args) File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/cli/command_modules/storage/operations/fs_directory.py", line 63, in set_access_control_recursive result = client.set_access_control_recursive(acl=acl, progress_hook=progress_callback, **kwargs) File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/multiapi/storagev2/filedatalake/v2020_02_10/_path_client.py", line 461, in set_access_control_recursive return self._set_access_control_internal(options=options, progress_hook=progress_hook, File "/usr/local/Cellar/azure-cli/2.18.0/libexec/lib/python3.8/site-packages/azure/multiapi/storagev2/filedatalake/v2020_02_10/_path_client.py", line 617, in _set_access_control_internal raise DataLakeAclChangeFailedError(error, error.message, last_continuation_token) azure.multiapi.storagev2.filedatalake.v2020_02_10._models.DataLakeAclChangeFailedError: (StorageErrorException('(SetAclMissingAces) Invalid ACL: the user, group and other entries are required.\nRequestId:4aabfb54-201f-0013-5ca3-fa8fe0000000\nTime:2021-02-04T03:11:32.7925451Z'), '(SetAclMissingAces) Invalid ACL: the user, group and other entries are required.\nRequestId:4aabfb54-201f-0013-5ca3-fa8fe0000000\nTime:2021-02-04T03:11:32.7925451Z', None) ... ``` ## To Reproduce: Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information. - `az storage fs access set-recursive --acl {} -p {} -f {} --account-name {} --account-key {}` I am running the following command `az storage fs access set-recursive --acl "group:xxxxxx:rwx,group:xxxxx:r-x" -p / -f test --account-name xxxxx --account-key xxxx` ## Expected Behavior Command Should grant access to the given AD group to the directory and all child members. ## Environment Summary ``` macOS-10.15.7-x86_64-i386-64bit Python 3.8.7 Installer: HOMEBREW azure-cli 2.18.0 Extensions: costmanagement 0.1.0 ``` ## Additional Context NA
Author: sshah90
Assignees: Juliehzl
Labels: `OKR3.2 Candidate`, `Service Attention`, `Storage`, `feature-request`
Milestone: S184
vivuu1989 commented 2 years ago

Facing same kind of issue and below is the part of pipeline where I am trying to update the ACL as per the user input paramater values. for the tasks setting default and access policies steps , i have the issue.

Here my requirement is that, we have many existing big filesystems which are already assigned with some default and access policies. Now we need to timely update the exising permission of the acl for the securityprinciple or need to add new security principle, which wont affect the existing acl.

So what I read from docs, if we have to set a permission for subdirectory called s3 and its contents, we have to give execute permission for each of the users for the path itself. eg: storageaccount/fs/s1/s2/s3.

when I tried update-recursive command, its working for me , but most of the times its getting timed out as the directories are large and the access is granting recursively.

So I tried for anothr ways of updating the permissions only for the path mentioned, for example, for each users i am inputting and i require rwx permission inside s3 subdirectory,

1) first it should set access default "execute permission for those users tranceive to the location storageaccount/fs/s1/s2/s3 2) set the input permission given (rwx) for the user for the s3 subdirectory.

When I tried set --permission command , i couldnt make the task successfull for updating the permissions.

- stage: Create_ACL
  displayName: 'Create the Given ACL' 
  variables:
    - name: directory
      value: ${{ parameters.subdirectory }}
  jobs:    
  - deployment: Create_ACL      
    environment: myenv
    displayName: "Creating ACL"
    strategy:
      runOnce:
        deploy:
          steps:                            
          - ${{ each user in parameters.userslist }}:               
            - task: AzureCLI@2         
              displayName: 'setting Default ACL for Execution permission for ${{ user }}'
              inputs:
                azureSubscription: 'mysubs'
                scriptType: 'bash'
                scriptLocation: 'inlineScript'             
                inlineScript: |
                  acl=default:user:${{ user }}:--x
                  az storage fs access set --permission=$acl -p / -f  ${{parameters.fsname}} --account-name ${{parameters.storagename}} --auth-mode login
            - task: AzureCLI@2     
              displayName: 'setting Access ACL for Execution permission for ${{ user }}'
              inputs:
                azureSubscription: 'mysubs'
                scriptType: 'bash'
                scriptLocation: 'inlineScript'             
                inlineScript: |
                  acl=user:${{ user }}:--x
                  az storage fs access set --permission=$acl -p / -f  ${{parameters.fsname}} --account-name ${{parameters.storagename}} --auth-mode login             
            - task: AzureCLI@2   
              displayName: 'setting Default permission for the given fs or Subdirectory for user ${{ user }}'
              inputs:
                azureSubscription: 'mysubs'
                scriptType: 'bash'
                scriptLocation: 'inlineScript'
                ${{ if ne(parameters.subdirectory, ' ') }}:                  
                  inlineScript: |
                    acl=default:user:${{ user }}:${{parameters.permission}}
                    az storage fs access update-recursive --acl=$acl -p $(directory) -f  ${{parameters.fsname}} --account-name ${{parameters.storagename}} --auth-mode login
                ${{ if eq(parameters.subdirectory, ' ') }}:                  
                  inlineScript: |
                    acl=default:user:${{ user }}:${{parameters.permission}}
                    az storage fs access update-recursive --acl=$acl -p / -f  ${{parameters.fsname}} --account-name ${{parameters.storagename}} --auth-mode login                    
            - task: AzureCLI@2   
              displayName: 'setting Access permission for the given fs or Subdirectory for user ${{ user }}'
              inputs:
                azureSubscription: 'mysubs'
                scriptType: 'bash'
                scriptLocation: 'inlineScript'
                ${{ if ne(parameters.subdirectory, ' ') }}:                  
                  inlineScript: |
                    acl=user:${{ user }}:${{parameters.permission}}
                    az storage fs access update-recursive --acl=$acl -p $(directory) -f  ${{parameters.fsname}} --account-name ${{parameters.storagename}} --auth-mode login
                ${{ if eq(parameters.subdirectory, ' ') }}:                  
                  inlineScript: |
                    acl=user:${{ user }}:${{parameters.permission}}
                    az storage fs access update-recursive --acl=$acl -p / -f  ${{parameters.fsname}} --account-name ${{parameters.storagename}} --auth-mode login
jonie001 commented 2 years ago

Replied in 21959