Separate user data residency for data privacy law compliance

[fungc-io]

Problem

In some countries, there are geographical limit on the registered server used to store the user data.

For example in China, the Personal Information Protection Law (PIPL) demand that the personal information of a Chinese Citizen including the mobile number the user used to log in should be stored in a server registered in China.

For an application that serves customers across countries, this means that according to a enduser’s identity

• their user infos should be stored into different locations
• The endusers will need to agree on a different set of terms and conditions

Research

In this section, we will look at how some MNCs approach this problem

NIKE International

• Show region in the signup/login screen
• Enduser should select the correct region by themselves
• If region is not China, +86 numbers cannot be registered

Apple

• Region of an enduser is determined based on their IP address at first signup, user can also change their region
• If the enduser’s region is in China, their info is stored in China

Solution

Overview

Let’s say a user want to serve both customers from both China and other countries.

For PIPL compliance, the user can set up two servers in China and, for example, HK respectively

• There will be 2 set up sign-up and login pages. One served from the HK server and one from China.
• Before signup, the app should determine which region the user located, this can be based on
  • Asking the user before triggering signup
  • IP Geolocation
  • Phone number country code prefix
• Then the user will be brought to the corresponding server for authentication
• In the HK server, +86 numbers should be blocked
• In the CN server, all phone numbers are allowed
• In the middle of signup, if the user is determined to be very likely from China, e.g. using email address ending by qq.com 163.com , if they are using the HK server, ask the user to restart the signup process in the CN server.
• If the user later change their region, their data should be migrated from the HK server to the China server. A Migration API will be available for this feature.
  • After data migration, all existing sessions of the user will expire and the user should re-login using the China Server.
• When signup, both servers should be check so no duplicated users will be created.
  • If the phone number is found in the another server, redirect them to that server for login.
• When login, after the user enters their phone number, both servers should be checked for looking up the user. And the user should be directed to the correct server.
• In the mobile app, 2 sets of endpoints and client ids are set in the SDK.

Note

If the Chinese users are expected to use the app from outside China, their network speed may be affected by the Chinese network. A tunnel is recommended communicate to the China server.

Benefits of this approach

• Guarantee the data are separated between HK and China server, including the cache, logs, phone numbers
  • Also it has low latency because DB is only query once. No need to query both DB for each operations.
  • No need a centralized storage for indexing and caching.
• The approach is simple and relatively easy to do. Which means it’s less error-prone.
• Easy to change the config in only one isolated server when the compliance requirements change in the future.

Support in Authgear

• In the Portal, add a page called User Data Residency for these configurations
  • In this page, user can set the endpoint for other servers, and specify in what criteria to redirect the user
  • Both Authgear projects should know the criteria of determining the region so they can redirect the users at the right moment
  • the projects should know the endpoints of the other server. So servers can talk to each other for
    • Checking if user exist
    • Migration
  • For example:
  • In the HK server, add a China server connection
  • Set up rules, if signup with email address ending by qq.com 163.com, redirect the user to the China Server
• In the Admin API:
  • Add the mutation to migrate the user from the current server to another connected server
• In the SDK:
  • Support redirect to another endpoint
• In the AuthUI:
  • support the relevant message for wrong region and redirection.

Rabbit Holes

• unless a 2-phase-commit-like protocol is implemented, otherwise it is possible that we will have race conditions between sign up, and there will be multiple users of identical login ID created in multiple regions. But a 2-phase-commit will significantly degrade signup throughput if it has to be coordinated between multiple region servers.

[fungc-io]

Hi @chpapa,

I've created the pitch based on our previous discussions few months ago. With elaboration in the "Solution" part, trying to give the solution more concrete shape.

What do you think about it?

One thing i'm not sure is the "redirection rules" under Support in Authgear > Portal, not sure if we should allow configuring these rules in Portal or only in custom defined workflows

[chpapa]

@fungc-io

A few thoughts on the concept/directions:

1. Maybe we can also think about if the address (country) field in the standard attributes could be useful at determining the location.
2. I think we should highlight unless a 2-phase-commit-like protocol is implemented, otherwise it is possible that we will have race conditions between sign up, and there will be multiple users of identical login ID created in multiple regions. But a 2-phase-commit will significantly degrade signup throughput if it has to be coordinated between multiple region servers.
3. Not quite sure about redirection rules yet... but I guess it is probably limited to the identify step; We do need to think about how passkey, oauth etc could be compatible with this?
4. Although it seems referencing other implementations, having multiple endpoint for multiple region seems the way to go, yet to complete our research and make sure, maybe we should still explore the idea of, for example, doing the redirection at API gateway.

[fungc-io]

1. It can be a redirection rules if the signup process include filling the profile? If the user's country is China, block the signup and redirect to the China server. If the user changes country later, the developer can either forbid this action, or run the migration API later

2. Let me put in this into Rabbit Hole

3. If the application uses other authenticators like passkey/oauth, the application should determine the user's region first before triggering the authentication. Authgear cannot redirect the user based on that.

---

4. Redirection at API gateway meaning the applications will always connect to the same endpoint?
   • Base on some criteria set in Authgear, the user will be created in either server.
   • This requires a centralized db to store the mapping of a user to their corresponding server location
   • For every requests, Authgear will get the user profile and data from their location

[louischan-oursky]

After the product meeting on 2023-08-02, I read skimmed through GDPR and PIPL.

Pseudonymized data is still considered as personal data by GDPR. Therefore even if we store pseudonymized identifiers (like email addresses and phone numbers) in a centralized database, this approach could violate the law.

Does this mean we can rule out the option of having a centralized database to ensure identities (e.g. email addresses and phone numbers) are unique across all regions for a given project?

Given that two-phase-commit protocol is complex and can degrade the performance, I think we can also rule out this option.

Then, can we conclude that identifier uniqueness is something that we need give up?

Since we do not have a centralized database, all servers in different regions must be known to each other. They have to ask each other if an identifier already exists in another region. We need to design a protocol for the servers in different regions to communicate.

[tung2744]

Trying out cockroach db

• [x] Try to setup cockroachdb in local with multi region nodes

• [x] Try to setup regional table

• [x] Run authgear server with regional table

• [ ] Stress test

• [ ] Observe behavior of failing / disconnected nodes (Check survival goals)

• [ ] Try to migrate some data

• [ ] Try to backup data

• [ ] Setup a real multi region cluster and try it

Findings

• Original migrations cannot be run, follow this to create a migration for cockroach db: https://www.cockroachlabs.com/docs/stable/migrate-from-postgres

• ALTER TABLE statements cannot be run inside a transaction with UPDATE / INSERT

• ALTER TABLE statements seems time consuming even for empty table

• Remove brin indexes, replace with btree

• jsonb is not indexable, use jsonb_col::text which change it to text

• pg_partman is not supported, see if we still need to partition _audit_log.

• Triggers are not implemented.

• pg_notify cannot be used. Possible replacement: https://www.cockroachlabs.com/docs/stable/change-data-capture-overview

  Steps

• Setting up multi-region database & table

  
  SHOW REGIONS FROM CLUSTER;
  ALTER DATABASE {database} SET PRIMARY REGION "{region}"; -- (Require enterprise license)
  ALTER DATABASE {database} ADD REGION "{region2}";
  SET enable_multiregion_placement_policy=on;
  ALTER DATABASE {database} PLACEMENT RESTRICTED;

ALTER TABLE {table} SET LOCALITY REGIONAL BY ROW;

### App server
- Multi app server (TBC)
- Multi redis instances (TBC)