Server cold start issue

[denismakogon]

Describe the bug

We’ve (me and my folks) been trying to solve the cold start issue of our pet project (https://github.com/fnproject/fdk-python). At this moment we use aiohttp as the base unix http server. However, when we started to do a solid performance testing, we’ve figured out that aiohttp takes almost 9 seconds to start within 128Mb RAM (this is our criteria: a server must start within 3 seconds, not longer). Then I decided to look around to see what else we can use and then i came across sanic. I was able to reduce the cold start time up to 6.5 seconds, however, this is still x2 of the required cold start time.

I figured out that sanic doesn’t have any importing rules and when i attempt to do things like "from sanic. import "tons of unnecessary imports being executed (for instance, if i don’t use blueprints - i don’t want the import to be loaded; if I don’t use websocket - i don’t want it to affect the cold start, etc.)

import time: self [us] | cumulative | imported package
import time:     75054 |      75054 | zipimport
import time:       971 |        971 | _frozen_importlib_external
import time:       166 |        166 |     _codecs
import time:     91712 |      91877 |   codecs
import time:      4969 |       4969 |   encodings.aliases
import time:     11474 |     108320 | encodings
import time:       810 |        810 | encodings.utf_8
import time:       501 |        501 | _signal
import time:      4961 |       4961 | encodings.latin_1
import time:       170 |        170 |     _abc
import time:      1861 |       2031 |   abc
import time:      5073 |       7104 | io
import time:     73518 |      73518 |       _stat
import time:      2963 |      76480 |     stat
import time:      1404 |       1404 |       genericpath
import time:    100572 |     101976 |     posixpath
import time:     99712 |      99712 |     _collections_abc
import time:      8394 |     286560 |   os
import time:      4962 |       4962 |   _sitebuiltins
import time:       797 |        797 |   sitecustomize
import time:       465 |        465 |   usercustomize
import time:      5821 |     298603 | site
import time:      2182 |       2182 |     types
import time:     87238 |      87238 |     _collections
import time:      8005 |      97424 |   enum
import time:       154 |        154 |     _sre
import time:      1343 |       1343 |       sre_constants
import time:      7231 |       8573 |     sre_parse
import time:      7200 |      15925 |   sre_compile
import time:       120 |        120 |     _functools
import time:       178 |        178 |         _operator
import time:     93176 |      93353 |       operator
import time:      1744 |       1744 |       keyword
import time:       372 |        372 |         _heapq
import time:      3370 |       3741 |       heapq
import time:     96271 |      96271 |       itertools
import time:      3532 |       3532 |       reprlib
import time:    103500 |     302139 |     collections
import time:     90438 |     392697 |   functools
import time:       233 |        233 |   _locale
import time:      1529 |       1529 |   copyreg
import time:     86535 |     594341 | re
import time:       477 |        477 |           collections.abc
import time:      2708 |       2708 |             concurrent
import time:       244 |        244 |                 time
import time:      1845 |       1845 |                       token
import time:    102814 |     104658 |                     tokenize
import time:      1602 |     106260 |                   linecache
import time:     90588 |     196847 |                 traceback
import time:     92023 |      92023 |                 warnings
import time:      2348 |       2348 |                   _weakrefset
import time:     95134 |      97482 |                 weakref
import time:       187 |        187 |                   _string
import time:     99503 |      99690 |                 string
import time:     13946 |      13946 |                 threading
import time:       130 |        130 |                 atexit
import time:     96707 |     597065 |               logging
import time:     97216 |     694281 |             concurrent.futures._base
import time:      3781 |     700769 |           concurrent.futures
import time:       494 |        494 |             _socket
import time:       324 |        324 |               math
import time:       833 |        833 |               select
import time:     90818 |      91975 |             selectors
import time:       327 |        327 |             errno
import time:     12336 |     105132 |           socket
import time:      3348 |       3348 |             signal
import time:       527 |        527 |             _posixsubprocess
import time:     98387 |     102261 |           subprocess
import time:      2003 |       2003 |             _ssl
import time:       315 |        315 |                 _struct
import time:     94578 |      94892 |               struct
import time:       894 |        894 |               binascii
import time:      3675 |      99460 |             base64
import time:    106677 |     208140 |           ssl
import time:       698 |        698 |           asyncio.constants
import time:       240 |        240 |                   _opcode
import time:     81569 |      81809 |                 opcode
import time:      3580 |      85388 |               dis
import time:      2616 |       2616 |                 importlib
import time:      1582 |       4197 |               importlib.machinery
import time:    299663 |     389247 |             inspect
import time:      1132 |       1132 |               asyncio.format_helpers
import time:      2138 |       3269 |             asyncio.base_futures
import time:       978 |        978 |             asyncio.log
import time:      2852 |     396346 |           asyncio.coroutines
import time:       275 |        275 |               _contextvars
import time:       499 |        774 |             contextvars
import time:      1275 |       1275 |               asyncio.base_tasks
import time:       775 |       2049 |             _asyncio
import time:     92092 |      94914 |           asyncio.events
import time:      3021 |       3021 |           asyncio.futures
import time:      4722 |       4722 |           asyncio.protocols
import time:      2089 |       2089 |             asyncio.transports
import time:     91472 |      93560 |           asyncio.sslproto
import time:      5257 |       5257 |           asyncio.tasks
import time:    540759 |    2256050 |         asyncio.base_events
import time:      4277 |       4277 |         asyncio.locks
import time:      1357 |       1357 |         asyncio.runners
import time:      2318 |       2318 |         asyncio.queues
import time:     38923 |      38923 |         asyncio.streams
import time:      6261 |       6261 |         asyncio.subprocess
import time:     97375 |      97375 |           asyncio.base_subprocess
import time:      8924 |       8924 |           asyncio.selector_events
import time:    181920 |     288218 |         asyncio.unix_events
import time:      1568 |    2598969 |       asyncio
import time:      1750 |       1750 |       fdk.constants
import time:      2429 |       2429 |           importlib.abc
import time:      3968 |       3968 |           contextlib
import time:     91994 |      98390 |         importlib.util
import time:       618 |      99007 |       fdk.customer_code
import time:      1332 |       1332 |       fdk.log
import time:      1169 |       1169 |       fdk.http
import time:       526 |        526 |             _datetime
import time:    202196 |     202721 |           datetime
import time:      2083 |       2083 |                   numbers
import time:     89055 |      91137 |                 _decimal
import time:       640 |      91777 |               decimal
import time:      4485 |      96262 |             iso8601.iso8601
import time:       976 |      97237 |           iso8601
import time:       217 |        217 |             fdk.headers
import time:       933 |       1150 |           fdk.context
import time:       269 |        269 |             fdk.response
import time:      1760 |       2029 |           fdk.errors
import time:      1733 |     304869 |         fdk.runner
import time:      1499 |     306368 |       fdk.http.event_handler
import time:      1091 |       1091 |         fdk.async_http
import time:      1689 |       1689 |                         _compat_pickle
import time:       557 |        557 |                             org
import time:        91 |        648 |                           org.python
import time:        87 |        735 |                         org.python.core
import time:      1077 |       1077 |                         _pickle
import time:    197836 |     201336 |                       pickle
import time:       414 |        414 |                         _queue
import time:      3558 |       3972 |                       queue
import time:    101548 |     306855 |                     logging.handlers
import time:      3928 |       3928 |                     socketserver
import time:     92155 |     402937 |                   logging.config
import time:       584 |        584 |                     urllib
import time:     91224 |      91807 |                   urllib.parse
import time:      2653 |       2653 |                         multiprocessing.process
import time:     89237 |      89237 |                           array
import time:      2803 |      92039 |                         multiprocessing.reduction
import time:      4185 |      98877 |                       multiprocessing.context
import time:     97875 |     196751 |                     multiprocessing
import time:       386 |     197137 |                   sanic.reloader_helpers
import time:       510 |        510 |                       sanic.helpers
import time:      1181 |       1690 |                     sanic.exceptions
import time:       636 |       2326 |                   sanic.config
import time:       481 |        481 |                   sanic.constants
import time:       600 |        600 |                     sanic.log
import time:       212 |        212 |                         winreg
import time:      3791 |       4003 |                       mimetypes
import time:       725 |        725 |                             aiofiles.base
import time:       432 |        432 |                             aiofiles.threadpool.utils
import time:      1017 |       2173 |                           aiofiles.threadpool.binary
import time:       654 |        654 |                           aiofiles.threadpool.text
import time:       683 |       3510 |                         aiofiles.threadpool
import time:       517 |       4026 |                       aiofiles
import time:    192941 |     192941 |                           platform
import time:       550 |     193490 |                         multidict._compat
import time:     90186 |      90186 |                         multidict._abc
import time:       271 |        271 |                           multidict._istr
import time:       806 |        806 |                           multidict._multidict_base
import time:      2341 |       3418 |                         multidict._multidict
import time:       739 |     287832 |                       multidict
import time:      1774 |       1774 |                       sanic.cookies
import time:      1066 |       1066 |                       ujson
import time:      1382 |     300081 |                     sanic.response
import time:       832 |     301512 |                   sanic.handlers
import time:       179 |        179 |                       _uuid
import time:      6732 |       6910 |                     uuid
import time:     83760 |      83760 |                     sanic.views
import time:      2707 |      93376 |                   sanic.router
import time:       328 |        328 |                           httptools.parser.errors
import time:       443 |        771 |                         httptools.parser.parser
import time:       336 |       1107 |                       httptools.parser
import time:       357 |       1464 |                     httptools
import time:       344 |        344 |                             _json
import time:      2472 |       2815 |                           json.scanner
import time:      3829 |       6644 |                         json.decoder
import time:    101125 |     101125 |                         json.encoder
import time:      2216 |     109984 |                       json
import time:       571 |        571 |                           email
import time:      1324 |       1324 |                             email.errors
import time:     88202 |      88202 |                                 email.quoprimime
import time:      2078 |       2078 |                                 email.base64mime
import time:      2011 |       2011 |                                     quopri
import time:       920 |       2930 |                                   email.encoders
import time:     97197 |     100127 |                                 email.charset
import time:     11005 |     201410 |                               email.header
import time:     86363 |      86363 |                                     _hashlib
import time:      2831 |       2831 |                                     _blake2
import time:     94555 |      94555 |                                     _sha3
import time:      2362 |     186109 |                                   hashlib
import time:       325 |        325 |                                     _bisect
import time:      4002 |       4327 |                                   bisect
import time:      1386 |       1386 |                                   _random
import time:      4747 |     196568 |                                 random
import time:    182213 |     182213 |                                     locale
import time:      6868 |     189080 |                                   calendar
import time:      3906 |     192986 |                                 email._parseaddr
import time:      5391 |     394943 |                               email.utils
import time:      4944 |     601296 |                             email._policybase
import time:      5907 |     608526 |                           email.feedparser
import time:     86297 |     695393 |                         email.parser
import time:      1697 |       1697 |                           uu
import time:     94938 |      94938 |                           email._encoded_words
import time:      3554 |       3554 |                           email.iterators
import time:     97739 |     197928 |                         email.message
import time:     95412 |      95412 |                           html.entities
import time:      4934 |     100346 |                         html
import time:       977 |        977 |                             fnmatch
import time:       912 |        912 |                             zlib
import time:      3975 |       3975 |                               _compression
import time:      1358 |       1358 |                               _bz2
import time:      4876 |      10208 |                             bz2
import time:       840 |        840 |                               _lzma
import time:     94300 |      95139 |                             lzma
import time:       572 |        572 |                             pwd
import time:      1476 |       1476 |                             grp
import time:     98194 |     207475 |                           shutil
import time:     96806 |     304281 |                         tempfile
import time:      7428 |    1305373 |                       cgi
import time:     84357 |      84357 |                         http
import time:     12879 |      97235 |                       http.cookies
import time:     95820 |    1608411 |                     sanic.request
import time:       125 |        125 |                         gc
import time:       893 |       1017 |                       uvloop.includes
import time:      1868 |       1868 |                       uvloop._patch
import time:       529 |        529 |                         backports_abc
import time:       862 |        862 |                         uvloop._noop
import time:      1559 |       1559 |                         concurrent.futures.thread
import time:      6020 |       8970 |                       uvloop.loop
import time:     91527 |     103381 |                     uvloop
import time:      1782 |    1715036 |                   sanic.server
import time:       424 |        424 |                     aiofiles.os
import time:     86435 |      86858 |                   sanic.static
import time:       454 |        454 |                   sanic.testing
import time:       605 |        605 |                         websockets.exceptions
import time:       685 |        685 |                           websockets.extensions
import time:       215 |        215 |                             websockets.speedups
import time:       611 |        826 |                           websockets.framing
import time:      2180 |       3690 |                         websockets.extensions.permessage_deflate
import time:      1114 |       1114 |                           websockets.headers
import time:       683 |       1797 |                         websockets.handshake
import time:       190 |        190 |                           websockets.version
import time:     87595 |      87785 |                         websockets.http
import time:       316 |        316 |                           websockets.compatibility
import time:       295 |        295 |                             websockets.py36
import time:       318 |        613 |                           websockets.py36.protocol
import time:      3073 |       4001 |                         websockets.protocol
import time:       828 |        828 |                         websockets.uri
import time:       616 |        616 |                           websockets.py35
import time:       503 |       1119 |                         websockets.py35.client
import time:      1806 |     101626 |                       websockets.client
import time:       234 |        234 |                         websockets.py35.server
import time:      1026 |       1259 |                       websockets.server
import time:       984 |     103868 |                     websockets
import time:       980 |     104847 |                   sanic.websocket
import time:      1287 |    2998054 |                 sanic.app
import time:      1709 |       1709 |                 sanic.blueprints
import time:       441 |    3000203 |               sanic
import time:        75 |    3000277 |             sanic.request
import time:       441 |    3000718 |           fdk.async_http.protocol
import time:       305 |    3001022 |         fdk.async_http.serve
import time:      1171 |    3003283 |       fdk.async_http.app
import time:      1321 |    6013195 |     fdk
import time:     87309 |    6100503 |   fdk.scripts
import time:      3385 |    6103888 | fdk.scripts.fdk

Code snippet Take whatever code sample that is available in this repo and package that into a docker container.

Desired behavior Run it with:

docker run --rm -ti --cpu-quota=10000 --cpu-period=100000 --cpu-shares=128 --memory=128m --kernel-memory=128m --memory-swap=128m --entrypoint /bin/bash -e PYTHONPROFILEIMPORTTIME=1 <your-image>

I expect to have server start within 3 seconds at maximum.

The problem is that sanic (similar to aiohttp) doesn't really care about imports, ordering as well as executing code in __init__.py.

What sanic should do

First of all, import things only if they are needed.

For instance:

• if my server doesn't need websockets - i don't want them to be ever loaded into my syspath
• if my server doesn't need blueprints - i don't want them to be loaded.

Similar to aiohttp, I'd like to see sanic to provide very low-level API to create a simple http server, without any sugar around.

[denismakogon]

So, here's some things that sanic needs to adopt:

1. No more code in init.py
2. optional imports for websockets, blueprints, etc.
3. Move protocol, HTTP request, HTTP response, etc. into a sub-package. This code supposes to be the skeleton of the simplest HTTP server, and the rest is just a lot of wrappers on top of that.

[sjsadowski]

Hi Denis - thanks for your interest in Sanic. We're working through a list of open issues and PRs currently for out next release (19.03) but if you and your team would like to help contribute to refactoring things to make them faster, better, and easier to use we will certainly not turn down the assistance. Feel free to submit PRs that address your concerns. They will generally be debated if they make major changes to the codebase - you can review some of the current PRs under review for an idea of what that looks like but we are pretty good about accepting them in the end.

[ahopkins]

In general, I would say start up time is not a huge focus for the project. The main goal is the speed of the request/response cycle. Not to say start up time is unimportant, but it is sort of non-consequential for most web service applications.

Of course we welcome the PR for debate.

As far as things like websockets, while it is not a final decision, we are looking into the possibility of removing that dependency in the future.

[denismakogon]

Well, that’s true. Cold start was never a concern to most of py frameworks.

However, the area of my interest is a serverless applications where the cold start is the biggest problem. So far I wasn’t able to find any proper technology that really takes cold start issue seriously, maybe because nobody ever talked about that.

[c-goosen]

@denismakogon Look at: https://github.com/Miserlou/Zappa.

[c-goosen]

Maybe the question to ask is what kind of application do you want to build in serverless? API/Script?

[c-goosen]

Ive done an API in AWS Lambda before, but state/sessions/etc get tricky.

[denismakogon]

@c-goosen, I’m not using AWS, we’re building our own platform, so, thanks, but i’m not kind interested in that.

[denismakogon]

Take a look at https://github.com/fnproject/fn

[c-goosen]

@denismakogon thanks will take a look, have you seen: https://openwhisk.apache.org/

[denismakogon]

Hi! As I mentioned before I’m not interested in other FaaS solutions because I personally developing serverless platform: Fn Project.

As part of that I developed Python binding for an HTTP triggers. That’s why I am here. Because I’d like to improve sanic in order to make it work for my needs, for instance I don’t want to deal with hypercorn (despite that I developed pretty simple ASGI HTTP server with that: https://github.com/denismakogon/simple-asgi) because it takes a lot time to initialize. So, sanic still be a best option to use for now.

[denismakogon]

Why is it closed? Problem is still there.

[yunstanford]

per the discussion moving to https://community.sanicframework.org/t/cold-start-issue/182

[yunstanford]

we can come back here for tracking whenever having clear action items