Spike: Impact analysis of new async bulk API on Applaud connector

[ssarap]

Summary

1. As a integrations designer
2. I want to use the new Aysnc bulk API
3. so that we are giving time for Applaud cloud for data push to avoid server load on Cloud and wait times in Recipe

Acceptance Criteria

1. Analyse the changes that needs to made to the Applaud connector to use this new API, may be a new action with label Async
2. Analyse how this will impact the existing customers who are using Bulk API
3. Analyse how we are going to be sure of the data insertion and complete the recipe execution

Wireframes and mockups

[Title of wireframe/mockup 1]

[Image of wireframe/mockup 1]

[Title of wireframe/mockup 2]

[Image of wireframe/mockup 2]

Notes

• Here goes a quote from an email

• Here goes any other useful information…

Developer Testing

Requestor: leave this section blank Developer: complete this section during commit/pull request

Positive testing:

Add all the things that you have checked whilst completing this story here, it should be written in a non-technical way, and a lot of this can probably be taken from the acceptance criteria, here are two different examples:

• I checked that the language of the page changed to Spanish when I selected Spanish in the language dropdown
• I checked that the list filtered correctly when I added an isBefore date filter on list block

Negative testing:

Here you list the things you have checked where your code needs to show an error, handle unusual/bad input/config gracefully, or may impact separate features that use or depend on the same code. Pause and think -- they may not be obvious! Examples:

• I checked that there was no "Select language" dropdown option when there is only one language configured
• I checked that a nice error is shown when a language is selected, but the page isn't translated into that language yet
• I checked that my list filter code changes didn't cause problems when filters are used in other places like ACLs, workflow if/then, and select list filters
• I checked that in list filters, I can only select a date type property for the "Value" field, when the "Property" field is a date type property

e2e test links and description:

• http://www.Link-to-Ghost-inspector-test.com - and paste the test description here
• http://www.Link-to-another-Ghost-inspector-test.com - and paste the test description here

[duncancasemore]

@ssarap thanks. If the recipes are waiting, can we also understand whether Workato has a max runtime for recipes or async operations -- want to make sure it won't terminate incomplete recipes.

[uiguru]

@ssarap - the cloud team have had to start on the work for this on the cloud side of things. That story is being worked on now.

Having discussed with Lewis, the work on your team now will need to be on finding a way to integrate with the new this async bulk pattern.

As the story is a high priority on cloud, please keep the communication flowing between the teams so we can be made aware of any issues (on with side) as soon as possible.

The cloud story is here: https://github.com/ApplaudSolutions/applaud-cloud/issues/4470#issuecomment-955978699

[ssarap]

@uiguru We are working on this and we still have a few things to review from integrations point of view. So far below is our analysis :

1. Changes to Applaud connector to include a new bulk Async action
2. Review how can we process the data with 20K records without facing Recipe time out issues (Workato recipe cannot run more than 90 mins)
3. Populate the data pills of this action so that subsequent steps can use the generated ids
4. Ways to handle the failed records
5. How can we leverage the Workato triggers using Webhooks in combination to bulk Aysnc actions
6. Searching community recipes whether any thing similar has been used

Team has put up some time studying these things and we will plan for a call with Cloud team and post them queries so that this enhancement can be effecitively used by upcoming projects.

[uiguru]

Thank you @ssarap - there is a lot of conversation on this between @lewie6 and @duncancasemore - so please keep this issue updated as regularly as possible and keep in touch.

[ssarap]

Sure @uiguru

[ssarap]

Analysis:

1. Connector SDK would be changed to use multi-step action
2. This action will have two parameters Post URL and Time out
3. When Post URL is specified the action would send a request to cloud bulk async API by passing the specified value and another recipe based on webhook should be written which gets executed when the Cloud API posts data after the request completion.
4. When Time out value is specified the action would wait for specified milliseconds and again would recheck the status of the transaction and would be complete once the response is received from Cloud. We need to check with Cloud team on the timings for each 1000 records call.
5. Would request cloud to send list of failed records in a separate object mapping against secondary unique key (Not sure of the feasibility). This would help us in rerunning the request with failed records rather than retrying the complete request.

@duncancasemore Do you like to add anything here.

[hgajula-applaud]

@ssarap what happens on point 4 when the cloud does not respond ( or the transaction is still running on cloud) Does the recipe fail after 90 mins?

[ssarap]

@hgajula-applaud We will retry few times say 20 times and then we raise an error.

[ssarap]

@duncancasemore @hgajula-applaud @uiguru We had a discussion on this with @lewie6 and Hari over a call. Lewis suggested not to use polling instead use the call back URL to check the response. In Workato SDK we can currently use only polling mechanism but not the callback. We are further checking with Workato team and brainstorming internally to suffice this requirement. Will keep you posted on this as we get any clues and any pointers from anybody will be highly appreciated.

[hgajula-applaud]

@ssarap understand the support we have/going to get from workato. Based on that, we have to take a decision on our API support.

[ssarap]

Need to defer it to Jan 2022.