search

jetstreamapp / soql-parser-js

Javascript SOQL parser
https://jetstreamapp.github.io/soql-parser-js/
MIT License
79 stars 20 forks source link

question: are long IN clauses handled? #236

Closed techieshark closed 1 year ago

techieshark commented 1 year ago

Hi! Neither a bug nor feature but a question (apologies if not best forum for it) -

This is somewhat obscure, but I'm wondering how you handle composing a query where you have a very long "IN" clause. For example, imagine you are looking for Contact records and you are searching for say 100-200 email addresses.

In the past (using other libraries) this has been a problem as - if I recall correctly - the HTTP GET request to Salesforce becomes too long (the URL is too long).

Just wondering if you've run across that and have a solution for it in this library.

Thanks!

and p.s. thanks for suggesting this library (over on a jsforce github issue thread)

paustint commented 1 year ago

@techieshark - There are some use-cases that Jetstream handles programmatically to solve these types of use-cases. Basically the idea is to split up the query into multiple queries and then combine all the results as if it were just one query.

Nothing is built into this library specifically right now, but it is a good feature request to include something like this to automatically split up queries if they are too large.

Take a look here for an example: https://github.com/jetstreamapp/jetstream/blob/main/libs/shared/ui-utils/src/lib/hooks/useFetchPageLayouts.ts

Gist:

  1. Either calculate the maximum length of all the item or just set an upper bound on number of items if you have an idea of the possible length of items and know how large the rest of your query will be
  2. Build multiple queries (I have a helper function I used here splitArrayToMaxSize)
  3. Query each set of data, e.x. for (const query of getPageLayoutQueries(sObjects)) { - I use a for loop to make them consecutive, you could try concurrent if there are not too many queries
  4. And depending on if the results would be paginated or not, you may want to build a queryAll helper function like I did in Jetstream https://github.com/jetstreamapp/jetstream/blob/5b1caa2585f13392e2239d64ca87e9268be8cf1a/libs/shared/data/src/lib/client-data.ts#L305

Here is another example where Jetstream will fetch all child records for a given parent record, and to do this it builds subqueries dynamically, but also makes sure not to make the SOQL too large (just limits to 50 child relationships, but the number of fields is largely known - so this was a safe number I landed on) https://github.com/jetstreamapp/jetstream/blob/5b1caa2585f13392e2239d64ca87e9268be8cf1a/apps/jetstream/src/app/components/core/ViewChildRecords.tsx#L175

Hope this helps

techieshark commented 1 year ago

Thanks for the detailed response @paustint!

Yeah that final example here does look similar to what I try to do currently:

https://github.com/jetstreamapp/jetstream/blob/5b1caa2585f13392e2239d64ca87e9268be8cf1a/apps/jetstream/src/app/components/core/ViewChildRecords.tsx#L208-L214

composeQuery looks great!

I will close this issue now to keep things tidy