argonlaser
commented
7 years ago Good analysis. We must have a caching layer in FRONT or API so as to minimize the cloning that would help a lot to avoid cloning big repositories.
Closed scriptnull closed 7 years ago
argonlaser
commented
7 years ago Good analysis. We must have a caching layer in FRONT or API so as to minimize the cloning that would help a lot to avoid cloning big repositories.
scriptnull
commented
7 years ago SSHed into the box and exec bash on docker and git clone inside the docker command results in a usual memory usage
It started with
It went all the way up to
When I used the website to clone remacs, memory and CPU usage hikes to
Investigating more.
scriptnull
commented
7 years ago As I suspected, it was a discrepancy caused by the git clone library. Removed the dependency and used os/exec with git seem to solve the problem.
Now the server is able to clone multiple remacs at the same time.
scriptnull
commented
7 years ago hmm the initial clone went well. But I made the worker size to 4 and seems like we are missing something.
But, for now it should be ok. I think, i will try some kind of limiting like docker cpu limit, memory limit etc.
scriptnull
commented
7 years ago | Service | Qty | Memory | Cpu |
|---|---|---|---|
| rabbit | 1 | 150m | 1 |
| api | 1 | 50m | 1 |
| front | 1 | 200m | 2 |
| worker | 2 | 200m | 3 |
Results: Worker performance results
| Repo number | Size of Repo | Start time | End time |
|---|---|---|---|
| remacs 1 | ~250m | 2:24 | 2:33 |
| remacs 2 | ~250m | 2:24 | 2:33 |
go-wrk badgeit.today alone
scriptnull 📂 ~ ⚡️ : go-wrk -t 2 http://badgeit.today
==========================BENCHMARK==========================
URL: http://badgeit.today
Used Connections: 100
Used Threads: 2
Total number of calls: 1000
===========================TIMINGS===========================
Total time passed: 12.13s
Avg time per request: 1095.28ms
Requests per second: 82.43
Median time per request: 905.13ms
99th percentile time: 2464.57ms
Slowest time for request: 2469.00ms
=============================DATA=============================
Total response body sizes: 2470000
Avg response body per request: 2470.00ms
Transfer rate per second: 203592.69 Byte/s (0.20 MByte/s)
==========================RESPONSES==========================
20X Responses: 1000 (100.00%)
30X Responses: 0 (0.00%)
40X Responses: 0 (0.00%)
50X Responses: 0 (0.00%)
Errors: 0 (0.00%)
scriptnull 📂 ~ ⚡️ go-wrk -t 3 http://badgeit.today
==========================BENCHMARK==========================
URL: http://badgeit.today
Used Connections: 100
Used Threads: 3
Total number of calls: 1000
===========================TIMINGS===========================
Total time passed: 10.18s
Avg time per request: 909.04ms
Requests per second: 98.25
Median time per request: 875.53ms
99th percentile time: 1508.94ms
Slowest time for request: 1548.00ms
=============================DATA=============================
Total response body sizes: 2470000
Avg response body per request: 2470.00ms
Transfer rate per second: 242668.35 Byte/s (0.24 MByte/s)
==========================RESPONSES==========================
20X Responses: 1000 (100.00%)
30X Responses: 0 (0.00%)
40X Responses: 0 (0.00%)
50X Responses: 0 (0.00%)
Errors: 0 (0.00%)go-wrk badgeit.today + all workers are working
scriptnull 📂 ~ ⚡️ : go-wrk -t 2 http://badgeit.today
==========================BENCHMARK==========================
URL: http://badgeit.today
Used Connections: 100
Used Threads: 32
Total number of calls: 1000
===========================TIMINGS===========================
Total time passed: 12.69s
Avg time per request: 1141.27ms
Requests per second: 78.82
Median time per request: 1155.23ms
99th percentile time: 1786.83ms
Slowest time for request: 2285.00ms
=============================DATA=============================
Total response body sizes: 2470000
Avg response body per request: 2470.00ms
Transfer rate per second: 194688.81 Byte/s (0.19 MByte/s)
==========================RESPONSES==========================
20X Responses: 1000 (100.00%)
30X Responses: 0 (0.00%)
40X Responses: 0 (0.00%)
50X Responses: 0 (0.00%)
Errors: 0 (0.00%)
scriptnull 📂 ~ ⚡️ go-wrk -t 3 http://badgeit.today
==========================BENCHMARK==========================
URL: http://badgeit.today
Used Connections: 100
Used Threads: 3
Total number of calls: 1000
===========================TIMINGS===========================
Total time passed: 9.77s
Avg time per request: 874.93ms
Requests per second: 102.30
Median time per request: 846.44ms
99th percentile time: 1406.76ms
Slowest time for request: 1716.00ms
=============================DATA=============================
Total response body sizes: 2470000
Avg response body per request: 2470.00ms
Transfer rate per second: 252691.78 Byte/s (0.25 MByte/s)
==========================RESPONSES==========================
20X Responses: 1000 (100.00%)
30X Responses: 0 (0.00%)
40X Responses: 0 (0.00%)
50X Responses: 0 (0.00%)
Errors: 0 (0.00%)Things are pretty stable now. At least we are in a position where, our machine is not going down. The above configuration suits us and if some error happens, it should probably stop that container with error logs, as I have set memory, cpu-period, cpu-quota options in docker.
Recently @argonlaser tried cloning https://github.com/Wilfred/remacs on the worker. This git repo is 250+ MB in size. ( Good job on doing this🏅 🎖 )
After the cloning process starts, the following logs are found and the worker container stops. Also note that, during the cloning process, the CPU usage is very high and even SSH into the machine is being blocked by the absence of CPU availability, I guess.
Suspicion
I have following suspicions:
Logs