search

ymyzk / mypy-playground

The mypy playground. Try mypy with your web browser.
https://mypy-play.net
MIT License
69 stars 9 forks source link

(šŸŽ) warm the cache / pre-cache the stdlib #228

Open KotlinIsland opened 2 years ago

KotlinIsland commented 2 years ago

mypy-playground can be slow, mostly due to checking the stdlib:

playground

Succeeded!! (2047 ms) Success: no issues found in 1 source file LOG: Could not load plugins snapshot: @plugins_snapshot.json LOG: Mypy Version: 0.971 LOG: Config File: Default LOG: Configured Executable: None LOG: Current Executable: /opt/python3.10/bin/python3 LOG: Cache Dir: /dev/null LOG: Compiled: True LOG: Exclude: [] LOG: Found source: BuildSource(path='main.py', module='__main__', has_text=False, base_dir=None) LOG: Could not load cache for __main__: __main__.meta.json LOG: Metadata not found for __main__ LOG: Parsing main.py (__main__) LOG: Could not load cache for builtins: builtins.meta.json LOG: Metadata not found for builtins LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/builtins.pyi (builtins) LOG: Could not load cache for sys: sys.meta.json LOG: Metadata not found for sys LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/sys.pyi (sys) LOG: Could not load cache for types: types.meta.json LOG: Metadata not found for types LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/types.pyi (types) LOG: Could not load cache for _ast: _ast.meta.json LOG: Metadata not found for _ast LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/_ast.pyi (_ast) LOG: Could not load cache for _collections_abc: _collections_abc.meta.json LOG: Metadata not found for _collections_abc LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/_collections_abc.pyi (_collections_abc) LOG: Could not load cache for _typeshed: _typeshed/__init__.meta.json LOG: Metadata not found for _typeshed LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/_typeshed/__init__.pyi (_typeshed) LOG: Could not load cache for collections.abc: collections/abc.meta.json LOG: Metadata not found for collections.abc LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/collections/abc.pyi (collections.abc) LOG: Could not load cache for io: io.meta.json LOG: Metadata not found for io LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/io.pyi (io) LOG: Could not load cache for typing: typing.meta.json LOG: Metadata not found for typing LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/typing.pyi (typing) LOG: Could not load cache for typing_extensions: typing_extensions.meta.json LOG: Metadata not found for typing_extensions LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/typing_extensions.pyi (typing_extensions) LOG: Could not load cache for importlib.abc: importlib/abc.meta.json LOG: Metadata not found for importlib.abc LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/importlib/abc.pyi (importlib.abc) LOG: Could not load cache for importlib.machinery: importlib/machinery.meta.json LOG: Metadata not found for importlib.machinery LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/importlib/machinery.pyi (importlib.machinery) LOG: Could not load cache for array: array.meta.json LOG: Metadata not found for array LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/array.pyi (array) LOG: Could not load cache for ctypes: ctypes/__init__.meta.json LOG: Metadata not found for ctypes LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/ctypes/__init__.pyi (ctypes) LOG: Could not load cache for mmap: mmap.meta.json LOG: Metadata not found for mmap LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/mmap.pyi (mmap) LOG: Could not load cache for pickle: pickle.meta.json LOG: Metadata not found for pickle LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/pickle.pyi (pickle) LOG: Could not load cache for os: os/__init__.meta.json LOG: Metadata not found for os LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/os/__init__.pyi (os) LOG: Could not load cache for collections: collections/__init__.meta.json LOG: Metadata not found for collections LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/collections/__init__.pyi (collections) LOG: Could not load cache for codecs: codecs.meta.json LOG: Metadata not found for codecs LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/codecs.pyi (codecs) LOG: Could not load cache for abc: abc.meta.json LOG: Metadata not found for abc LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/abc.pyi (abc) LOG: Could not load cache for importlib: importlib/__init__.meta.json LOG: Metadata not found for importlib LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/importlib/__init__.pyi (importlib) LOG: Could not load cache for importlib.metadata: importlib/metadata/__init__.meta.json LOG: Metadata not found for importlib.metadata LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/importlib/metadata/__init__.pyi (importlib.metadata) LOG: Could not load cache for os.path: os/path.meta.json LOG: Metadata not found for os.path LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/os/path.pyi (os.path) LOG: Could not load cache for contextlib: contextlib.meta.json LOG: Metadata not found for contextlib LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/contextlib.pyi (contextlib) LOG: Could not load cache for subprocess: subprocess.meta.json LOG: Metadata not found for subprocess LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/subprocess.pyi (subprocess) LOG: Could not load cache for _codecs: _codecs.meta.json LOG: Metadata not found for _codecs LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/_codecs.pyi (_codecs) LOG: Could not load cache for pathlib: pathlib.meta.json LOG: Metadata not found for pathlib LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/pathlib.pyi (pathlib) LOG: Could not load cache for email.message: email/message.meta.json LOG: Metadata not found for email.message LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/email/message.pyi (email.message) LOG: Could not load cache for importlib.metadata._meta: importlib/metadata/_meta.meta.json LOG: Metadata not found for importlib.metadata._meta LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/importlib/metadata/_meta.pyi (importlib.metadata._meta) LOG: Could not load cache for posixpath: posixpath.meta.json LOG: Metadata not found for posixpath LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/posixpath.pyi (posixpath) LOG: Could not load cache for email: email/__init__.meta.json LOG: Metadata not found for email LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/email/__init__.pyi (email) LOG: Could not load cache for email.charset: email/charset.meta.json LOG: Metadata not found for email.charset LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/email/charset.pyi (email.charset) LOG: Could not load cache for email.contentmanager: email/contentmanager.meta.json LOG: Metadata not found for email.contentmanager LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/email/contentmanager.pyi (email.contentmanager) LOG: Could not load cache for email.errors: email/errors.meta.json LOG: Metadata not found for email.errors LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/email/errors.pyi (email.errors) LOG: Could not load cache for email.policy: email/policy.meta.json LOG: Metadata not found for email.policy LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/email/policy.pyi (email.policy) LOG: Could not load cache for genericpath: genericpath.meta.json LOG: Metadata not found for genericpath LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/genericpath.pyi (genericpath) LOG: Could not load cache for email.header: email/header.meta.json LOG: Metadata not found for email.header LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/email/header.pyi (email.header) LOG: Loaded graph with 38 nodes (0.358 sec) LOG: Found 2 SCCs; largest has 37 nodes LOG: Processing SCC of size 37 (typing_extensions typing types sys subprocess posixpath pickle pathlib os.path os mmap io importlib.metadata._meta importlib.metadata importlib.machinery importlib.abc importlib genericpath email.policy email.message email.header email.errors email.contentmanager email.charset email ctypes contextlib collections.abc collections codecs array abc _typeshed _collections_abc _codecs _ast builtins) as inherently stale LOG: Processing SCC singleton (__main__) as inherently stale LOG: No fresh SCCs left in queue LOG: Build finished in 1.761 seconds with 38 modules, and 1 errors

Is it possible to include the cache files in the cloud_function? I'm not quite sure how they work.

ymyzk commented 2 years ago

Thanks for filing the ticket. I had a similar idea before. One idea I had is that running mypy against a Python script like the following at build time to populate the cache.

import abc
import <other stdlibs>

Although this will obviously increase the size of sandbox environments (Docker images, Cloud Functions, etc.) which can affect start up time and hosting cost. We need to create a PoC and see how it works.