search

pepsico-ecommerce / snowflex

elixir snowflake client
Apache License 2.0
52 stars 10 forks source link

[WIP] Add a stream function to the api #35

Closed Ch4s3 closed 2 years ago

Ch4s3 commented 2 years ago

This function is build around :odbc.select_count/3 and the next/0 function.

A stream is built using Stream.resource/3 to do the following

  1. call seect_count/3 to get a count and bind the query to the :odbc context
  2. call next and process the data until the count is decremented to 0
  3. upon :halt check the poolboy connection back in

This is all necessary because sufficiently large queries can cause very large spikes in memory.

Ch4s3 commented 2 years ago

big improvement. i think we should add a test for this

Sadly, this doesn't work because :odbc is owned by the Worker and can't be called in an external context. There are 2 path's forward here:

  1. support select_count and next with explicit check_in/out
  2. hoist the streaming to the calling app and document how to do it.

I'm leaning towards 2.

Ch4s3 commented 2 years ago

closing this for now, to refocus efforts elsewhere