dremio-professional-services / dremio-cloner

27 stars 21 forks source link

Use request.Session to share http connection between calls #40

Closed jeff-99 closed 1 year ago

jeff-99 commented 1 year ago

When doing a GET or PUT synchronisation a lot of time is spent in setting up HTTPS connections. I added a shared session to the http clients so the connection can be reused. Improving the total runtime by 70% in one of our projects.

before:

Command being timed: "python .dremio-cloner/src/dremio_cloner.py .tmp.cloner.config.json"
User time (seconds): 25.73
System time (seconds): 0.61
Percent of CPU this job got: 16%
Elapsed (wall clock) time (h:mm:ss or m:ss): 2:39.81

after:

Done with 0 error(s). Please review log file for details.
Command being timed: "python .dremio-cloner/src/dremio_cloner.py .tmp.cloner.config.json"
User time (seconds): 16.19
System time (seconds): 0.40
Percent of CPU this job got: 36%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:45.08