Method to merge Duplicates

aficiomaquinas commented 1 month ago

Scope & Context

I'm submitting a feature request for enabling users to merge duplicate person records in their workspace. This feature is important for maintaining clean and accurate data, especially in cases where partial information about a person is entered separately, resulting in duplicate records. The primary focus is on consolidating all related data, including the full history, opportunities, tasks, notes, and other relationships associated with each person record.

Current behavior

Currently, there is no built-in functionality to merge two person records. While users can manually copy basic information from one record to another, the main problem lies in consolidating the associated data:

Historical data (e.g., interaction history, changes to the person record) is split between two records
Related entities such as opportunities, tasks, and notes are attached to separate person records
Timeline events and activities are distributed across multiple records
Relationships with other entities (e.g., companies, projects) may be incomplete on each record

This fragmentation of data leads to:

Incomplete views of a person's history and relationships
Potential loss of critical information if one record is deleted
Inefficiencies in managing customer relationships with external systems via API
Inaccurate reporting and analytics

Expected behavior

A comprehensive person merge feature that not only combines basic contact information but also consolidates all related data and maintains the full history of both records. This should be achievable through multiple interfaces: REST API, GraphQL, and User Interface.

The merge process should:

Combine all non-conflicting information from both records
Consolidate all related entities:
- Opportunities
- Tasks
- Notes
- Timeline events
- Activity history
- Relationships with other entities (e.g., companies, projects)
Preserve the full history of both records, including past changes and interactions
Update all related entities to reference the merged record

greptile-apps[bot] commented 1 month ago

Implementation Steps

Create mergePersons Method in PersonService:

Path: /packages/twenty-server/src/workspace/messaging/repositories/person/person.service.ts

Add a new method mergePersons to handle merging person records.

async mergePersons(primaryPersonId: string, secondaryPersonId: string, workspaceId: string, transactionManager?: EntityManager): Promise<void> {
// Fetch both person records
const primaryPerson = await this.getById(primaryPersonId, workspaceId, transactionManager);
const secondaryPerson = await this.getById(secondaryPersonId, workspaceId, transactionManager);

// Merge data from secondaryPerson into primaryPerson
// Implement logic to handle merging of related entities (opportunities, tasks, notes, etc.)
// Update references in related entities to point to primaryPerson

// Delete secondaryPerson record
await this.deletePerson(secondaryPersonId, workspaceId, transactionManager);
}

Update PersonRepository:
- Path: /packages/twenty-server/src/modules/person/repositories/person.repository.ts
- Add necessary methods to fetch, update, and delete person records.
```
async getById(personId: string, workspaceId: string, transactionManager?: EntityManager): Promise<PersonWorkspaceEntity> {
const dataSourceSchema = this.workspaceDataSourceService.getSchemaName(workspaceId);
return await this.workspaceDataSourceService.executeRawQuery(`SELECT * FROM ${dataSourceSchema}.person WHERE id = $1`, [personId], workspaceId, transactionManager);
}
```
async deletePerson(personId: string, workspaceId: string, transactionManager?: EntityManager): Promise { const dataSourceSchema = this.workspaceDataSourceService.getSchemaName(workspaceId); await this.workspaceDataSourceService.executeRawQuery(DELETE FROM ${dataSourceSchema}.person WHERE id = $1, [personId], workspaceId, transactionManager); }

Handle Merge Events in MessagingPersonListener:

Path: /packages/twenty-server/src/workspace/messaging/listeners/messaging-person.listener.ts

Add logic to handle events related to merging person records.

async handleMergeEvent(payload: MergePersonEvent): Promise<void> {
const { primaryPersonId, secondaryPersonId, workspaceId } = payload;
await this.personService.mergePersons(primaryPersonId, secondaryPersonId, workspaceId);
}

Update GraphQL and REST API Endpoints:
- Add new endpoints to trigger the merge functionality via API.
- Ensure the endpoints call the mergePersons method in PersonService.
Update Frontend to Support Merge Functionality:
- Add UI components to allow users to select and merge person records.
- Ensure the frontend calls the appropriate API endpoints to perform the merge.

References

/packages/twenty-server/src/workspace/messaging/repositories/person/person.service.ts /packages/twenty-server/src/modules/person/repositories/person.repository.ts /packages/twenty-server/src/modules/person/repositories/person/person.service.ts /packages/twenty-server/src/modules/connected-account/auto-companies-and-contacts-creation/services/create-company-and-contact.service.ts /packages/twenty-server/src/core/quick-actions/quick-actions.service.ts /packages/twenty-server/src/workspace/messaging/listeners/messaging-person.listener.ts /packages/twenty-chrome-extension/src/db/person.db.ts /packages/twenty-chrome-extension/src/graphql/person

#### About Greptile

This response provides a starting point for your research, not a precise solution. Help us improve! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. [Ask Greptile](https://app.greptile.com/chat/github/twentyhq/twenty/main) · [Edit Issue Bot Settings](https://app.greptile.com/apps/github)

aficiomaquinas commented 1 month ago

As I go further down the investigation on how to deduplicate persons in Twenty from my app through the API I think that it would be very useful to have company merging too.

FelixMalfait commented 1 month ago

Thanks for the suggestion! We always try to build features in the most agnostic way possible so if we build it for People, we would also build it for companies, opportunities or any custom object at the same time.

I think it's around 2 to 3 weeks or eng work to do this well. Backend is pretty simple (2 days?) but it would probably require significant frontend work (+ @Bonapara has not designed this, so it depends on how it'd look like)

twentyhq / twenty