pubsub storage notification management

[marcosrmendezthd]

Hi,

I'd like to be able to manage storage notifications the way we manage other DM resources. They should either exist or not. In our deployments we have multiple pubsub topics and use storage notifications as toggle to control what processes them.

         if pipeline == active:
             notification = {
                 'name': '{}-notification'.format(resource_name),
                 'type': 'gcp-types/storage-v1:notifications',
                 'properties': {
                     'bucket': bucket,
                     'event_types': ["OBJECT_FINALIZE"],
                     'payload_format': "none",
                     'topic': '//pubsub.googleapis.com/projects/{}/topics/{}'.format(project,resource_name),
                 },
         }

However, I've noticed that it only keeps track of the notification it creates at best. Has anyone run into this issue?

I tried to do this in terraform as well, and basically had to script it out with gsutil to determine if i had the wrong configuration; duplicates, or existing notifications that shouldn't exist.

Do I need to do the same here or can DM manage them?

[ocsig]

Hi Marcos, Let my check if I understand your question well. You have some storage-v1:notifications created by DM, some created manually or gcloud? Is you question, how to import existing notifications into a DM deployment to manage it's life-cycle from there? There is the DM Scaffolder code, which can be easily extended to support 'gcp-types/storage-v1:notifications'. This could generate your DM config and let you acquire them into your deployments. In terms of making sure, not resource created outside DM: you can limit it via IAM or you can monitor your admin logs and react on those.

Please let me know if I understood your question correctly.

[marcosrmendezthd]

Hi, in our case there shouldn't be any notifications created outside of DM. however that could happen, or maybe an update didn't work all the way through etc. We've wrapped the notifications in a python module so that the operator doesn't have to list out every resource; the resources get auto-generated based on the configuration yaml; -> python template -> list of resources, outputs.

So for gke clusters, if someone creates a gke cluster, the deployment can acquire it. When it gets removed from the config yaml, it gets removed.

I'm not sure how to manage the storage notification list as an actual entity with all its members/notifications being accounted for. So that if the notification list has entries that are not being managed (or are duplicates) by DM they get removed.

Using the DM scaffolder would mean that managing that resource separately from the python code that generates the resources. Does that make sense?

Are you thinking we could use the scaffolder to dynamically generate these notifications and then we could through scripting, remove the notifications that shouldn't be there and update that deployment separately? This would be rather difficult no?

[ocsig]

If you want a hard rule: Notifications should not be created outside the DM script, you can:

• Remove IAM rights from other users/groups/SAs so only the DM SA has the right to CRUD them.
• Create a logsink with a filter for notification CRUD, where the actor is not a DM SA. > This triggers an out of policy creation you can have a cron job cleaning these up, or a warning to your ops team.

DM cannot and should not delete resources which are NOT managed by the DM deployment. The scaffolder is an initial, bulk import if that's what you need. ( Helps your dev team to move a large amount of handcrafted resources into DM config, however it is a dump, not a high quality infra as code.)

[marcosrmendezthd]

understood. the line blurs a little bit since resources can be acquired if they exist already. :smile: