sandeephs1 commented 8 months ago

In catalog, we want to display search results limited to the organization user belongs to

Here is the scenario - Below table details, team, user, environment and associated organization <html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">

mgm and alexa are the 2 organization mgm has 2 environments, 8 dataset, 8 tables alexa has 2 envrionemnts, 6 dataset, 6 tables Below table details dataset and tables <html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">

When user 'mgm_u1' searches, returned results should not exceed (8 dataset + 8 tables)+searchcondition. He should not be displayed with the 'alexa' associated dataset and tables

similarly alexa user 'x_u1' should not be displayed with 'mgm' objects.

How this can be achieved

dlpzx commented 8 months ago

Hi @sandeephs1 this is a cool feature :) Basically you want a filtered version of the catalog based on the user's organizations. What is the motivation behind the feature? Do you need to restrict access to the metadata, or is it more of a usability problem?

sandeephs1 commented 8 months ago

Hi @dlpzx We have multiple products, our customers can subscribe to any of them.

customers -> Organization in Data.All product -> Environment in Data.All

Since multiple customers will be onboarded on "Data.All", should be able to meet the data governance. Currently Catalog search presents all the matching dataset/table irrespective of the Organization user belongs to, so it will be a databreach.

We want to avoid this situation by restricting catalog search results based on the "user-Organization" relation. In a way multi-tenancy feature

dlpzx commented 8 months ago

Hi @sandeephs1 thanks for the quick response! Based on your requirements, these are the high-level changes that we would need to implement:

Add another field "organizationUri" in the data catalog, for that we need to modify the mappings of the OpenSearch index
Add the information of the "organizationUri" to each item that is added to the catalog
Backfill existing items (we need to investigate on the best approach)
Modify the search API calls to filter by the user's organizations (list of "organizationUri")
Introduce a configuration parameter to enable or disable "organization_data_catalog_isolation" and make this feature configurable in the code -- probably only the part of limiting the search api calls.

Do you have the bandwidth to implement this feature? We are happy to provide guidance and coaching throughout the process. We will consider it in our roadmap, but other features might be prioritized

sandeephs1 commented 8 months ago

Thanks @dlpzx we were also thinking in the similar lines, your input is definitely of help. we will implement this feature and update you

dlpzx commented 7 months ago

Updates [offline discussion]

@sandeephs1 and his team have a proposed implementation in which:

manually, data.all admins will create Cognito groups for each of the Organizations. These groups need to follow the naming convention: <SOME-NAME>-accesscontrol-<organizationUri>
search_handler Lambda fetches the user's groups
search_handler Lambda looks for a particular Cognito group based on a naming convention (contains -accesscontrol-<organizationUri> and extracts the organizationUri from it
catalog search API query run_query filters the results based on the organizationUri field

Remarks from data.all team

First of all, we want to highlight that the by-design-purpose of data.all was to work in a single-tenancy scenario. The way to achieve multi-tenancy in the most secure way would be by deploying data.all multiple times, one for each tenant.

But we understand and are interested in the multi-tenancy scenario that you present. We cannot implement this feature in the next couple of weeks, but we can work together on designs and are happy to guide you to contribute back. If the feature is part of the open-source repository we will manage and take ownership of bugs, issues and enhancements.

Here are some remarks that we have pointed out during our internal discussions:

Since there will be multiple tenants sharing the same central infrastructure (data.all backend and frontend), we need to make sure that security-wise tenants cannot damage other tenants. For example, Could some tenant teams throttle the API Gateway for all other tenants? The answer is no because of the WAF rules implemented, but I wanted to use an example of the types of infrastructure security checks that we need to fulfill.
We need to verify all data.all API calls to ensure that all are decorated and are checking that users cannot access resources that do not belong to them. [We can help you with this task :) Findings will be added to this comment.]
We also need to identify if there are any API calls that need ADDITIONAL checks. You already implemented the changes for the searchCatalog API call to filter results based on your organzition. We also noticed that the createShareObject API call needs to be restricted to check that the target requested Dataset is part of your organization. Also, the inviteTeam APIs for Organization and Environment will need to restrict which groups can be added to each of them.
In your design we see potential improvements. We would like to avoid the string-manipulation involved in obtaining the user organizationUris. For that, we can use or at least take inspiration from the helper methods that are defined for the generic graphql APIs.

Let's keep on working on this!

data-dot-all / dataall

Limiting Catalog search results based on Organizations #965

Updates [offline discussion]

Remarks from data.all team