search

dremio / dbt-dremio

dbt (data build tool) adapter for the Dremio
Apache License 2.0
44 stars 21 forks source link

Optimize REST API workload by using Session and slowing down polling loop #220

Closed chulucninh09 closed 7 months ago

chulucninh09 commented 8 months ago

Summary

The cursor implementation was using while True loop to poll for result from REST API of Dremio. Without throttling, it will jam the server easily, especially when using a high number of thread count. Since most SQL jobs need a few second to run, looping too fast to get the result is unessesary.

Also, in REST client implementation, using get, post, delete directly from requests is inefficient because it will create new HTTP session with the server everytime the polling is called.

Description

  1. Use request.Session instead of vanila requests to utilize connection pooling, so optimize networking performance
  2. Add time.sleep in while True loop to slow down the polling

Changelog

chulucninh09 commented 8 months ago

Hi @ravjotbrar, can you comment on this PR and make this progress

ravjotbrar commented 7 months ago

Apologies for the late response @chulucninh09. I will try to get to this tomorrow.

chulucninh09 commented 7 months ago

Hi @ravjotbrar, can I get approval on this or any comment for this PR?