dlpzx
commented
2 months ago Hi @gavclark, thanks for opening an issue! If I understood it correctly this is an issue of usability, you started using data.all with a team with a generic name and as your usage grows you want to use more representative team names. As you forecasted, it has a couple of complications as the team name defines the team IAM role and athena workgroup. As soon as the environment CloudFormation stack is updated (which happens daily), the previous IAM role and workgroup will be deleted and a new ones will be created.
Initial complexity assessment
For the team IAM role:
- the new IAM role won't have LakeFormation permissions to team-owned datasets ---> In the current dataset stack (v2.6) the permissions are also granted on stack update, so updating the dataset stacks should update the permissions. We need to verify that the dataset stack gets updated every time we click on "update dataset" in the UI.
- the new IAM role won't have LakeFormation permissions to shared datasets ---> If the team has not requested any dataset we won't face any issue. If there are share requests with this team as requested: the share health status is weekly checked and can also be checked on demand, it will appear as
UNHEALTHYand the team can reapply the share. Share verify and reapply features are available since 2.3. - the new IAM role won't have IAM permissions to some data.all team created AWS resources. We have to evaluate case by case.
- dataset IMPORTED S3 buckets and KMS keys: no problem, the new IAM role will be created alongside new IAM policies granting access to the buckets and KMS keys.
- dataset CREATED S3 buckets: in theory it should work without issue; not so sure about KMS keys
- other resources. In your case I think you are only interested in the old gluejobs/CDK pipelines. If I remember correctly, the pipelines and jobs use the team role as execution role. I am not sure what will happen when the role gets deleted. It could happen that the update environment fails when trying to delete the role because it is being used in a Glue job (for example). If the environment stack deletes the IAM role then the pipeline resources will be orphan and will fail as the defined execution role does not exist. I don't know if it is possible to update the pipeline stack or if it would require a manual update of all resources to use the new IAM role.
As for the Athena workgroup:
- no data.all AWS resources use the workgroup. So we do not need to worry about consumption tools, unless you have developed additional applications.
- the team IAM role needs to have IAM permissions to use the workgroup and to access the workgroup S3 location. This should happen automatically during the creation of the new IAM role (update environment stack).
Conclusion: it is a delicate operation with possible unseen issues. We strongly recommend some POC testing in a dev/test environment before carrying this out in prod.
Other options
Automated script using SDK/CLI
If I recognize your user correctly I believe your company is using an older version of data.all; which means this solution might not be available for you. Nevertheless, it is worth mentioning it.
We have implemented an SDK for data.all that allows users to call data.all APIs programmatically using Python. You could use the SDK to automate some of the operations needed:
for all shares:revokeShare, deleteShare, thenupdateEnvironment, updateDataset, for all old shares createShareRequest, addItems, submitShare...
Enhancement to data.all code
Instead of replacing the IAM role and the workgroup, we could use a team label to add the info for the sales retail group. The IAM role will still be 'arn:...."/role:sales' but in data.all it will appear as owned by the Sales Retail group. It might be easier than the actual migration of AWS resources.
gavclark
I'd like to create a new team in data.all and move a selection of assets from an existing team to the new team.
For example, we have a team called "Sales" and I want to create a new team called "Sales Retail" as it is more reflective of the business organisation . I want to then move some assets from the "Sales" team to the "Sales Retail" team via a simple process / method.
I am not sure if there is a documented process to do this or a tool we could use , My thoughts are that we could update the metadata database, but there'd still be a lot of remediation needed for other AWS services .
For example what do we need to do with: glue jobs / existing CKD pipelines and the IAM role assigned to them lakeformation admins / access athena workgroups s3 buckets / datasets glue databases / tables
I realise there may be more to consider, but I'd like is to have a robust process for this as we'd like to reorganise our existing teams.