Ability to perform custom trigger and filters configuration for Goonj Sync tasks in Avn-Goonj integration

[himeshr]

Goonj Aug 2024 change requirements sheet

Introduction to Goonj Integration:

Goonj has 2 separate jobs, to perform sync of data between Avni and Salesforce for Goonj org:

• Main Job: Triggered once every 30 minutes
• FullError Job: Triggered every 6th/12th/18th and 24th hour of every day

Within the Main Job, The are following sub-tasks that are executed one after another in a sequence:

• Demand
• Dispatch
• Activity
• DispatchReceipt
• Distribution
• ImplementationInventory

About going live in a new State(s):

For field-workers of the new state to start recording Activity and Distributions they need, Demand, Dispatch and ImplementationInventory of their state to be set up on Avni. The sync of Demand, Dispatch and ImplementationInventory from Goonj Salesforce system to Avni, needs the corresponding entity location (State->District) to be present.

Whenever, a new state has to go live for Goonj,

1. We first upload location entities for State
2. We then backdated-sync data for Demand, Dispatch and ImplementationInventory from a few months (3 to 6 months or more) back into the system, as they will only now get pulled, as the locations have been created just now.

Area of concern:

During this backdated-sync, we have observed that

• this overloads both the Salesforce system and Avni system due to us fetcing all records irrespective of their state, district or account
• Salesforce is unable to retrieve all the records for the specified timerange and recommended approach to overcome this is to make use of state or account filters in addition to the StartDateTime value
• this also results in delay in integration sync of data across the 2 systems as we block the rest of the goonj sync process

Therefore, the on-ground impact of this is

• Unnecessary load and wasteful processing on Salesforce and Avni
• Inability to fetch data from say more than 1 year for a specific account due to volume of data fetched
• Delay in sync of data from the Avni client devices of Goonj field workers to Goonj Salesforce systems and vice-versa

Technical proposal

To resolve this, we were asked to make use of the additional filtering capabilities (like State, accountName, etc..) provided by Salesforce GET APIs to optimize this back-dated sync. Therefore, the task ahead for us is to enhance the Goonj-Avni integration-service module to be able to perform "specific criteria back-dated syncs" whenever a new State has to go live.

### Tasks
- [ ] https://github.com/avniproject/Goonj/issues/122
- [ ] https://github.com/avniproject/Goonj/issues/121
- [ ] https://github.com/avniproject/Goonj/issues/120
- [ ] https://github.com/avniproject/Goonj/issues/119
- [ ] https://github.com/avniproject/Goonj/issues/118

[himeshr]

Go Live steps (WIP):

DB Updates

UPDATE public.error_type
SET comparison_value = '.*Address .* not found.*'
WHERE name = 'AddressNotFoundError';