search

cirruslabs / tart

macOS and Linux VMs on Apple Silicon to use in CI and other automations
https://tart.run
Other
3.76k stars 106 forks source link

Poor performance pulling and pushing to ECR #279

Closed adrienkohlbecker closed 1 year ago

adrienkohlbecker commented 1 year ago

Hi!

I'm not getting great performance with ECR. That is an issue for us because we are integrating tart into our macOS CI, which looks like this: pull an image, do things to it, and push the result back. As it stands, tart adds quite a lot of latency before tests can run, due to the slow pulling speed. Pushing is also an issue that prevents the jobs from moving on after the builds are finished.

I recently updated to release 0.30.0 hoping to improve this, so the numbers below are on this release.

I'm getting around 14MB/s when uploading, the image below is 27GB compressed and uploaded in 33min

> tart push tag ACCOUNT_ID.dkr.ecr.eu-west-1.amazonaws.com/macos/repo:tag
21:13:11.446744Z   pushing tag to ACCOUNT_ID.dkr.ecr.eu-west-1.amazonaws.com/macos/repo:tag...
21:13:11.479768Z   pushing config...
21:13:12.239447Z   pushing disk... this will take a while...
21:13:12.240589Z   0%

...

21:36:53.809777Z   100%
21:37:01.597372Z   pushing NVRAM...
21:37:03.378103Z   pushing manifest for tag...

Here is an example of pulling a 37.7GB image that took 1h20, so pulling is slower at 8MB/s.

> tart pull ACCOUNT_ID.dkr.ecr.eu-west-1.amazonaws.com/macos/repo:tag
15:20:29.398440Z   pulling ACCOUNT_ID.dkr.ecr.eu-west-1.amazonaws.com/macos/repo:tag...
15:20:29.398715Z   pulling manifest...
15:20:30.011137Z   pulling disk (37.7 GB compressed)...
15:20:30.011326Z   0%

...

16:42:32.350457Z   100%
16:42:32.381668Z   pulling NVRAM...

My setup is an AWS mac2.metal (M1 mac mini), hosted in the eu-west-1 region and working with ECR in that same region. The disk is a 500GB EBS gp2 volume which should provide 250 MB/s of throughput, and the network is a 10Gb/s link.

I would appreciate any help to improve the performance. I'm also working with @saracen, who you've already been in touch with.

adrienkohlbecker commented 1 year ago

I ran s3-benchmark on my instance to get an idea of the raw performance that is attainable with S3 (ECR Layers are pulled from S3, I believe?). It only tests the download speed, but it shows that a much higher throughput is possible:

ec2-user@ip-172-31-3-61 s3-benchmark % ./s3-benchmark

--- SETUP --------------------------------------------------------------------------------------------------------------------

Uploading 1 KB objects
 100% |████████████████████████████████████████|  [0s:0s]
Uploading 2 KB objects
 100% |████████████████████████████████████████|  [0s:0s]
Uploading 4 KB objects
 100% |████████████████████████████████████████|  [0s:0s]
Uploading 8 KB objects
 100% |████████████████████████████████████████|  [0s:0s]
Uploading 16 KB objects
 100% |████████████████████████████████████████|  [0s:0s]
Uploading 32 KB objects
 100% |████████████████████████████████████████|  [0s:0s]
Uploading 64 KB objects
 100% |████████████████████████████████████████|  [0s:0s]
Uploading 128 KB objects
 100% |████████████████████████████████████████|  [0s:0s]
Uploading 256 KB objects
 100% |████████████████████████████████████████|  [1s:0s]
Uploading 512 KB objects
 100% |████████████████████████████████████████|  [1s:0s]

--- BENCHMARK ----------------------------------------------------------------------------------------------------------------

Download performance with 1 KB objects (mac2.metal)
                           +-------------------------------------------------------------------------------------------------+
                           |            Time to First Byte (ms)             |            Time to Last Byte (ms)              |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads |     Throughput |  avg   min   p25   p50   p75   p90   p99   max |  avg   min   p25   p50   p75   p90   p99   max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
|       8 |       0.4 MB/s |   17    11    15    16    17    18    46    72 |   17    11    15    16    17    19    46    72 |
|       9 |       0.5 MB/s |   17    11    15    16    17    20    54    74 |   17    11    15    16    17    20    54    74 |
|      10 |       0.6 MB/s |   17    11    15    16    18    20    36    64 |   17    11    15    16    18    20    36    64 |
|      11 |       0.6 MB/s |   17    11    15    16    17    19    34    43 |   17    11    15    16    17    19    34    43 |
|      12 |       0.7 MB/s |   17    11    15    16    18    20    41    63 |   17    11    15    16    18    20    41    63 |
|      13 |       0.7 MB/s |   17    11    15    16    17    20    49    73 |   17    11    15    16    17    20    49    73 |
|      14 |       0.9 MB/s |   15    11    13    14    15    17    43    59 |   15    11    13    14    15    17    43    59 |
|      15 |       0.9 MB/s |   16    11    13    15    17    19    50    63 |   16    11    13    15    17    19    50    63 |
|      16 |       1.0 MB/s |   15    11    13    14    16    17    34    43 |   15    11    13    14    16    17    34    43 |
+---------+----------------+------------------------------------------------+------------------------------------------------+

Download performance with 2 KB objects (mac2.metal)
                           +-------------------------------------------------------------------------------------------------+
                           |            Time to First Byte (ms)             |            Time to Last Byte (ms)              |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads |     Throughput |  avg   min   p25   p50   p75   p90   p99   max |  avg   min   p25   p50   p75   p90   p99   max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
|       8 |       1.0 MB/s |   15    11    13    14    16    17    36    58 |   15    11    13    14    16    17    36    58 |
|       9 |       1.1 MB/s |   16    11    14    15    16    18    42    47 |   16    11    14    15    16    18    42    47 |
|      10 |       1.2 MB/s |   16    12    14    15    16    19    41    66 |   16    12    14    15    16    19    41    66 |
|      11 |       1.3 MB/s |   16    11    14    15    17    20    42    49 |   16    11    14    15    17    20    42    49 |
|      12 |       1.5 MB/s |   15    11    14    15    16    18    32    36 |   15    11    14    15    16    18    32    36 |
|      13 |       1.4 MB/s |   17    11    14    15    17    20    45    68 |   17    11    14    15    17    20    45    68 |
|      14 |       1.7 MB/s |   16    12    14    15    16    19    42    65 |   16    12    14    15    16    19    42    65 |
|      15 |       1.7 MB/s |   17    12    14    15    17    21    51    57 |   17    12    14    15    17    21    51    57 |
|      16 |       1.9 MB/s |   16    12    14    15    16    19    34    42 |   16    12    14    15    16    19    34    42 |
+---------+----------------+------------------------------------------------+------------------------------------------------+

Download performance with 4 KB objects (mac2.metal)
                           +-------------------------------------------------------------------------------------------------+
                           |            Time to First Byte (ms)             |            Time to Last Byte (ms)              |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads |     Throughput |  avg   min   p25   p50   p75   p90   p99   max |  avg   min   p25   p50   p75   p90   p99   max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
|       8 |       1.9 MB/s |   16    12    14    15    16    18    33    46 |   16    12    14    15    16    18    33    47 |
|       9 |       2.2 MB/s |   16    11    14    15    17    19    36    47 |   16    11    14    15    17    19    36    47 |
|      10 |       2.4 MB/s |   16    12    14    15    17    20    39    50 |   16    12    14    15    17    20    39    50 |
|      11 |       2.8 MB/s |   15    12    13    14    16    18    32    49 |   15    12    13    14    16    18    32    49 |
|      12 |       2.9 MB/s |   16    11    13    14    16    19    42    92 |   16    11    13    15    16    19    42    92 |
|      13 |       3.3 MB/s |   15    11    13    14    16    17    34    59 |   15    11    13    14    16    17    34    59 |
|      14 |       3.4 MB/s |   16    11    14    15    16    19    31    49 |   16    11    14    15    16    19    31    49 |
|      15 |       3.7 MB/s |   15    11    13    14    16    18    35    64 |   16    11    13    14    16    18    35    64 |
|      16 |       3.6 MB/s |   17    11    14    16    18    22    43    49 |   17    11    14    16    18    22    43    49 |
+---------+----------------+------------------------------------------------+------------------------------------------------+

Download performance with 8 KB objects (mac2.metal)
                           +-------------------------------------------------------------------------------------------------+
                           |            Time to First Byte (ms)             |            Time to Last Byte (ms)              |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads |     Throughput |  avg   min   p25   p50   p75   p90   p99   max |  avg   min   p25   p50   p75   p90   p99   max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
|       8 |       3.4 MB/s |   17    12    15    16    17    22    40    73 |   17    12    15    16    17    22    40    74 |
|       9 |       4.1 MB/s |   17    12    14    15    17    21    45    73 |   17    12    14    15    17    21    45    73 |
|      10 |       4.7 MB/s |   16    12    14    15    17    19    50    65 |   16    12    14    15    17    19    50    66 |
|      11 |       5.2 MB/s |   16    11    14    15    17    19    38    58 |   16    11    14    15    17    19    38    58 |
|      12 |       5.4 MB/s |   17    12    14    15    17    19    54    71 |   17    12    14    15    17    19    54    71 |
|      13 |       6.1 MB/s |   16    11    14    15    17    19    44    54 |   16    11    14    15    17    20    44    54 |
|      14 |       6.4 MB/s |   16    11    14    15    17    20    43    49 |   17    11    14    15    17    20    43    49 |
|      15 |       7.1 MB/s |   16    11    14    15    16    18    47    54 |   16    11    14    15    16    18    47    54 |
|      16 |       8.1 MB/s |   15    11    13    14    15    18    36    54 |   15    11    13    14    16    18    36    55 |
+---------+----------------+------------------------------------------------+------------------------------------------------+

Download performance with 16 KB objects (mac2.metal)
                           +-------------------------------------------------------------------------------------------------+
                           |            Time to First Byte (ms)             |            Time to Last Byte (ms)              |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads |     Throughput |  avg   min   p25   p50   p75   p90   p99   max |  avg   min   p25   p50   p75   p90   p99   max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
|       8 |       7.9 MB/s |   15    11    13    14    16    18    31    51 |   15    11    13    15    16    19    32    51 |
|       9 |       8.9 MB/s |   15    11    14    14    16    18    32    51 |   16    11    14    15    16    19    33    51 |
|      10 |       9.6 MB/s |   16    12    14    15    16    19    36   155 |   16    12    14    15    16    19    36   156 |
|      11 |      11.1 MB/s |   15    11    13    14    15    17    42    54 |   15    11    13    14    16    17    42    54 |
|      12 |      11.9 MB/s |   15    11    13    14    15    17    45    49 |   15    11    13    14    16    18    45    49 |
|      13 |      12.4 MB/s |   16    11    13    15    16    19    37   116 |   16    11    14    15    17    19    38   117 |
|      14 |      14.0 MB/s |   15    11    13    14    16    18    33    39 |   15    11    14    15    16    18    33    39 |
|      15 |      14.8 MB/s |   16    11    13    14    15    17    44    69 |   16    11    13    14    15    17    45    69 |
|      16 |      15.3 MB/s |   16    11    13    14    16    18    48    60 |   16    12    14    14    16    18    48    60 |
+---------+----------------+------------------------------------------------+------------------------------------------------+

Download performance with 32 KB objects (mac2.metal)
                           +-------------------------------------------------------------------------------------------------+
                           |            Time to First Byte (ms)             |            Time to Last Byte (ms)              |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads |     Throughput |  avg   min   p25   p50   p75   p90   p99   max |  avg   min   p25   p50   p75   p90   p99   max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
|       8 |      16.1 MB/s |   14    11    13    14    14    16    37    40 |   15    11    13    14    15    17    39    41 |
|       9 |      17.8 MB/s |   15    11    13    14    15    17    36    67 |   16    11    14    15    16    19    37    68 |
|      10 |      19.5 MB/s |   15    11    13    14    15    17    37    62 |   16    12    14    15    16    18    38    62 |
|      11 |      20.8 MB/s |   16    11    13    14    16    19    39    55 |   16    12    14    15    17    20    40    56 |
|      12 |      23.6 MB/s |   15    11    13    14    15    17    32    55 |   16    12    14    15    16    18    32    56 |
|      13 |      23.3 MB/s |   16    11    13    14    16    20    52    69 |   17    12    14    15    17    21    53    70 |
|      14 |      24.3 MB/s |   16    11    13    14    16    20    46    70 |   17    11    14    15    17    21    48    71 |
|      15 |      27.0 MB/s |   16    11    13    14    16    20    51    68 |   17    12    14    15    17    21    52    69 |
|      16 |      30.0 MB/s |   15    10    13    14    15    17    52    93 |   16    11    14    15    16    19    53    93 |
+---------+----------------+------------------------------------------------+------------------------------------------------+

Download performance with 64 KB objects (mac2.metal)
                           +-------------------------------------------------------------------------------------------------+
                           |            Time to First Byte (ms)             |            Time to Last Byte (ms)              |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads |     Throughput |  avg   min   p25   p50   p75   p90   p99   max |  avg   min   p25   p50   p75   p90   p99   max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
|       8 |      28.6 MB/s |   16    12    14    15    17    20    35    45 |   17    13    15    16    18    22    36    46 |
|       9 |      33.3 MB/s |   16    11    14    15    16    19    28    78 |   17    12    15    16    17    20    30    79 |
|      10 |      35.8 MB/s |   16    11    14    15    17    20    38    64 |   17    12    15    16    18    22    39    65 |
|      11 |      41.7 MB/s |   15    10    13    14    16    18    36    40 |   16    11    14    15    17    19    38    41 |
|      12 |      43.6 MB/s |   16    10    14    15    17    19    33    56 |   17    11    15    16    18    20    34    58 |
|      13 |      47.2 MB/s |   16    11    14    15    16    18    35    54 |   17    12    15    16    17    20    36    55 |
|      14 |      48.9 MB/s |   17    11    14    15    17    20    40    45 |   18    12    15    16    18    22    41    50 |
|      15 |      50.7 MB/s |   17    11    14    15    17    21    43    73 |   18    12    15    17    19    23    44    74 |
|      16 |      53.4 MB/s |   17    12    14    15    17    21    40    67 |   18    12    15    16    19    22    42    68 |
+---------+----------------+------------------------------------------------+------------------------------------------------+

Download performance with 128 KB objects (mac2.metal)
                           +-------------------------------------------------------------------------------------------------+
                           |            Time to First Byte (ms)             |            Time to Last Byte (ms)              |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads |     Throughput |  avg   min   p25   p50   p75   p90   p99   max |  avg   min   p25   p50   p75   p90   p99   max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
|       8 |      51.3 MB/s |   18    13    15    16    18    21    42    60 |   19    14    16    18    19    24    43    62 |
|       9 |      55.3 MB/s |   18    12    15    17    18    21    38    76 |   19    13    17    18    20    23    40    76 |
|      10 |      62.9 MB/s |   18    13    15    17    18    21    45    61 |   20    14    17    18    20    23    47    64 |
|      11 |      72.2 MB/s |   17    12    15    16    17    20    45   149 |   19    13    16    17    19    22    48   151 |
|      12 |      79.5 MB/s |   17    12    15    16    17    20    44    57 |   19    13    16    17    19    22    45    59 |
|      13 |      87.7 MB/s |   17    12    15    16    17    20    39    75 |   18    13    16    17    19    21    41    77 |
|      14 |      92.4 MB/s |   17    12    15    16    17    19    45    97 |   19    13    16    17    19    21    47    98 |
|      15 |      90.5 MB/s |   19    12    16    17    18    19    67   117 |   20    13    17    18    19    21    69   121 |
|      16 |     100.1 MB/s |   18    12    16    17    18    20    43    50 |   19    13    17    18    20    21    46    52 |
+---------+----------------+------------------------------------------------+------------------------------------------------+

Download performance with 256 KB objects (mac2.metal)
                           +-------------------------------------------------------------------------------------------------+
                           |            Time to First Byte (ms)             |            Time to Last Byte (ms)              |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads |     Throughput |  avg   min   p25   p50   p75   p90   p99   max |  avg   min   p25   p50   p75   p90   p99   max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
|       8 |      97.4 MB/s |   18    11    16    17    18    20    36    51 |   20    14    18    19    21    23    41    66 |
|       9 |     107.2 MB/s |   18    15    17    18    19    21    33    47 |   21    16    19    20    21    23    40    49 |
|      10 |     116.5 MB/s |   19    13    17    18    19    21    38    71 |   21    14    19    20    21    23    43    73 |
|      11 |     137.1 MB/s |   18    13    16    17    18    19    43    65 |   20    14    17    19    20    23    51    68 |
|      12 |     152.8 MB/s |   17    12    16    16    17    19    32    55 |   19    14    17    18    20    21    37    63 |
|      13 |     164.1 MB/s |   17    13    16    17    17    19    35    57 |   19    14    17    18    20    21    41    65 |
|      14 |     179.9 MB/s |   17    12    16    17    17    19    33    36 |   19    13    17    18    20    22    37    42 |
|      15 |     186.9 MB/s |   18    14    16    17    18    20    37    60 |   19    15    17    18    19    22    41    60 |
|      16 |     202.7 MB/s |   18    13    16    17    18    19    37    64 |   19    14    17    18    20    22    44    65 |
+---------+----------------+------------------------------------------------+------------------------------------------------+

Download performance with 512 KB objects (mac2.metal)
                           +-------------------------------------------------------------------------------------------------+
                           |            Time to First Byte (ms)             |            Time to Last Byte (ms)              |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads |     Throughput |  avg   min   p25   p50   p75   p90   p99   max |  avg   min   p25   p50   p75   p90   p99   max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
|       8 |     182.7 MB/s |   18    13    16    17    18    20    41    49 |   21    15    18    20    22    25    48    64 |
|       9 |     208.1 MB/s |   18    13    16    17    18    20    35    69 |   21    15    19    20    21    25    47    71 |
|      10 |     236.8 MB/s |   18    15    17    17    18    19    36    52 |   21    16    19    20    21    23    40    65 |
|      11 |     257.8 MB/s |   18    14    16    17    18    19    45    75 |   21    16    18    19    21    24    50    78 |
|      12 |     292.5 MB/s |   18    13    16    17    18    19    34    48 |   20    15    18    19    21    23    41    60 |
|      13 |     307.2 MB/s |   18    13    16    17    18    20    51    81 |   21    15    19    20    21    23    53    87 |
|      14 |     338.1 MB/s |   18    13    16    17    18    20    40    59 |   20    15    18    19    20    23    45    62 |
|      15 |     331.8 MB/s |   18    11    14    15    17    24    57    70 |   22    14    17    19    22    31    64    89 |
|      16 |     374.0 MB/s |   17    11    14    15    17    20    56    66 |   21    13    17    19    21    26    62    81 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
fkorotkov commented 1 year ago

We also identified the regression. It seem #260 which was released in 0.30.0 helped in many cases but actually drastically regressed in some. We are investigating and will provide a fix this week.

In the meantime a workaround is to downgrade to 0.29.0.

adrienkohlbecker commented 1 year ago

Interesting! I went back and looked at past pipelines we had for this. It seems like the upload performance slightly worsened indeed, but the download stayed the same.

On that release, I was getting 7.8MB/s download and 19MB/s upload.