CSV File above 50 SoD-Policies: Fails to execute during Account Aggregation

[alan-cruz-sp]

Hello @mostafa-helmy-sp,

Hope all is going great, and thanks a lot for your effort and work here c:

Working arm-to-arm on this one. After testing the SaaS 2.0 Connector based on the Advanced Policy Management Framework, we found an error when someone uses a full list of policies above 50 of them (in this case 122). The connector, suddenly, stops working altogether.

Error message (after Account Aggregation):

"Unable to resolve Policy Owner. Type: IDENTITY, Value: policy.owner"]

• APMF Connector version: 0.3.0
• SoD CSV file: Policies_122.csv
• TypeScript source: Code segment

Standing-by if help and or support would be required, thanks again ! Alan

[mostafa-helmy-sp]

I think I found the root cause and I don't think it is directly tied to 50 policies. I built this tool to try and process all lines concurrently to save time, but when you have too many policies in there it seems to overload ISC and we get HTTP 429 errors (Too many requests).

I was under the impression that our TypeScript SDK (using Axios under the hood) was automatically handling HTTP 429 errors and backing off. Let me look into this. Worst case scenario, I remove the concurrent processing and have it process policies one by one.

[alan-cruz-sp]

Hello @mostafa-helmy-sp good day !,

Thanks for taking your time in this one.

Just to mention, I faced the same issue while trying to request data using filters. IdN API Rate Limit

If you find it useful, I add here a code snippet that I used to "avoid" the Axios with requests issue. It is tested, and it works, but as far as I know, the issue was "fixed" in the Connector SDK version: 1.1.4^

• Add to idn-client.ts
• Usage shown in function requestAndSetEntitlements

/**
     * Turns Array into chunk items for async processing (Promise.all)
     * 
     * @param items
     * @returns 
     */
    async chunkItems<T>(items: T[]): Promise<T[][]> {
        return items.reduce((chunks: T[][], item: T, index) => {
          const chunk = Math.floor(index / this.CHUNK_SIZE);
          chunks[chunk] = ([] as T[]).concat(chunks[chunk] || [], item);
          return chunks;
        }, []);
    }

    /**
     * Delay to avoid `HTTP 429` errors and ERR_STREAM_PREMATURE_CLOSE.
     * 
     * Fixed by @sailpoint/connector-sdk v1.1.4
     * 
     * https://developer.sailpoint.com/discuss/t/connectorerror-with-new-connector-that-uses-saas-connectivity-framework/15220/13
     */
    delay(): Promise<undefined> {
        return new Promise(r => setTimeout(r, this.CHUNK_DELAY))
    }

    /**
     * Retrieve filtered entitlements using chunks of items.
     * 
     * @returns any[]
     */
    async requestAndSetEntitlements(): Promise<any[]> {
        const searchedEntitlements = await this.searchEntitlementsFromSource()
        let results: any[] = []
        // Check Entitlements exist
        if (searchedEntitlements != null) {
            // Split searchedEntitlements to avoid HTTP 414 uri too long error
            const chunks = await this.chunkItems(searchedEntitlements)           
            for (const chunk of chunks) {
                let listEntitlements = []
                let filter = 'id in ("'
                chunk.forEach((element: any) => {
                    filter = filter + element.id + '","'
                });
                const lastCommaIndex = filter.lastIndexOf(',"')
                filter = filter.slice(0, lastCommaIndex) + ')' + filter.slice(lastCommaIndex + 2)
                // Get and set Source ID if not already set
                const entitlementsApi = new EntitlementsBetaApi(this.apiConfig)
                const entitlementsRequest: EntitlementsBetaApiListEntitlementsRequest = {
                    // filters: 'id in ("chunkNId1",..,"chunkNId#")'
                    filters: filter,
                    limit: this.CHUNK_SIZE
                }
                console.log(`Entitlement IDs. Current array chunk: ${chunks.indexOf(chunk)} length: ${chunk.length}`);
                try {
                    // https://eddieabbondanz.io/post/typescript/await-promise-all/
                    listEntitlements =
                        await entitlementsApi.listEntitlements(entitlementsRequest)
                            .then(ent => ent.data)
                    this.delay()
                } catch (err) {
                    let errorMessage = `Error retrieving Entitlements using Entitlements Beta API ${JSON.stringify(
                        err
                    )} with request: ${JSON.stringify(entitlementsRequest)}`
                    logger.error(errorMessage, err)
                    throw new ConnectorError(errorMessage)
                }
                results.push(listEntitlements)
            } // for
        }
        // await handled on function assignment
        return results
    } 

Let me know if something else would be needed. Thanks again !

[mostafa-helmy-sp]

This should fix the issue!