When doing a GET or PUT synchronisation a lot of time is spent in setting up HTTPS connections. I added a shared session to the http clients so the connection can be reused. Improving the total runtime by 70% in one of our projects.
before:
Command being timed: "python .dremio-cloner/src/dremio_cloner.py .tmp.cloner.config.json"
User time (seconds): 25.73
System time (seconds): 0.61
Percent of CPU this job got: 16%
Elapsed (wall clock) time (h:mm:ss or m:ss): 2:39.81
after:
Done with 0 error(s). Please review log file for details.
Command being timed: "python .dremio-cloner/src/dremio_cloner.py .tmp.cloner.config.json"
User time (seconds): 16.19
System time (seconds): 0.40
Percent of CPU this job got: 36%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:45.08
When doing a GET or PUT synchronisation a lot of time is spent in setting up HTTPS connections. I added a shared session to the http clients so the connection can be reused. Improving the total runtime by 70% in one of our projects.
before:
after: