TS server crashes with 1,000,000 line in a single d.ts file

[oriori1703]

Type: Bug

I have a big javascript project with a large api surface. I wrote a tool to generate typescript type definitions automatically. When the TypeScript server tries to initialize the project it crashes repeatedly. I worked fine when the definition file was only 100,000 lines.

VS Code version: Code 1.74.2 (e8a3071ea4344d9d48ef8a4df2c097372b0c5161, 2022-12-20T10:29:14.590Z) OS version: Windows_NT x64 10.0.22621 Modes: Sandboxed: No

System Info

|Item|Value| |---|---| |CPUs|AMD Ryzen 7 5800X 8-Core Processor (16 x 4200)| |GPU Status|2d_canvas: enabled
canvas_oop_rasterization: disabled_off
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_renderer: enabled_on
video_decode: enabled
video_encode: enabled
vulkan: disabled_off
webgl: enabled
webgl2: enabled
webgpu: disabled_off| |Load (avg)|undefined| |Memory (System)|31.92GB (9.62GB free)| |Process Argv|C:\\Users\\orior\\Desktop\\Projects\\Evrit\\Research --crash-reporter-id 82dc2225-5bce-4996-a106-e3cdf5cbb8d4| |Screen Reader|yes| |VM|25%|

Extensions (31)

Extension|Author (truncated)|Version ---|---|--- grok|Ada|1.0.0 vscode-frida|cod|0.4.10 vscode-eslint|dba|2.2.6 php-intellisense|fel|2.3.14 gc-excelviewer|Gra|4.2.56 vscode-graphviz|joa|0.0.6 vscode-docker|ms-|1.23.3 isort|ms-|2022.8.0 python|ms-|2022.20.1 vscode-pylance|ms-|2023.1.10 jupyter|ms-|2022.11.1003412109 jupyter-keymap|ms-|1.0.0 jupyter-renderers|ms-|1.0.12 vscode-jupyter-cell-tags|ms-|0.1.6 vscode-jupyter-slideshow|ms-|0.1.5 remote-containers|ms-|0.266.1 remote-wsl|ms-|0.72.0 cpptools|ms-|1.13.9 vsliveshare|ms-|1.0.5797 material-icon-theme|PKi|4.23.1 java|red|1.13.0 LiveServer|rit|5.7.9 intellicode-api-usage-examples|Vis|0.2.6 vscodeintellicode|Vis|1.2.30 vscode-java-debug|vsc|0.47.0 vscode-java-dependency|vsc|0.21.1 vscode-java-test|vsc|0.37.1 vscode-maven|vsc|0.40.2 php-debug|xde|1.30.0 php-pack|xde|1.0.3 php-intellisense|zob|1.0.13 (2 theme extensions excluded)

A/B Experiments

``` vsliv368cf:30146710 vsreu685:30147344 python383cf:30185419 vspor879:30202332 vspor708:30202333 vspor363:30204092 vslsvsres303:30308271 pythonvspyl392:30443607 vserr242cf:30382550 pythontb:30283811 vsjup518:30340749 pythonptprofiler:30281270 vsdfh931cf:30280410 vshan820:30294714 vstes263:30335439 pythondataviewer:30285071 vscod805:30301674 binariesv615:30325510 bridge0708:30335490 bridge0723:30353136 cmake_vspar411:30581797 vsaa593cf:30376535 pythonvs932:30410667 cppdebug:30492333 vsclangdc:30486549 c4g48928:30535728 dsvsc012:30540252 azure-dev_surveyone:30548225 vscccc:30610679 pyindex848:30577860 nodejswelcome1cf:30587006 3biah626:30602489 iaj6b796:30613358 f6dab269:30613381 fim-prod:30623723 vscsbc:30628655 ```

[mjbvz]

Please share the file that causes the issue

[oriori1703]

Here is the file: test.d.ts

[mjbvz]

Thanks!

large api surface

No kidding, the source text of the file 300 MB! If you open the file directly, I don't think VS Code even sends this file over to TypeScript because it's too big

However TS does try loading it if it is part of a project. Maybe we need to ignore files of this size by default

[yotamN]

I will chime in as the one who actually wrote the code generation. As can be seen in the project page, this is a code generator for Frida, which expose a large API surface. Sadly there is not much that can be done about it, as it expose a whole apk with all of its classes which are in the range of thousands and can actually reach hundred of thousands.

Would it be helpful to split it into multiple files? I've skimmed the performance guide and tried to apply it but it didn't help enough. Any other suggestions would be welcome.

[RyanCavanaugh]

We do have upper limits and, uh, this is one of them.

I don't really believe that an API surface exists that is nominally 300 MB but can't be reduced, a lot, by some kind of reuse or higher-order representation. Splitting into multiple files will help a little but fundamentally something must be happening that is resulting in an enormous amount of repetition. If you post some snippets (somehow) of relevant code, we can pitch some ideas.

[yotamN]

Frida is a tool to inject JavaScript code into a running process, in the case of the generated file, into a running Android application. We try to define two APIs that Frida provides: Java.use(className: str) to access a class and Java.choose(className: str) to access objects of that class.

The problem is, that a normal Android application have around 10,000 classes which mean that at least I will have 20,000 lines of Java.use and Java.choose for each one of them, where I define the string literal of the parameter and the return type will be a subinterface of a generic class. Of course that means for each class I define its interface from the APK which is why the file size is so big, I just define all of the classes from the APK.

I would love some help if you can think we can simplify it but I don't think there is much room to minimize it, maybe only writing more easy to parse code.

[RyanCavanaugh]

Sure. Looking at the first screenful of the provided .d.ts:

        interface Z extends Java.Wrapper {
                clone: {
                        overloads: [Java.Method<Packages.Z, [], Java.Wrapper<Packages.java.lang.Object>>],
                        overload(): Java.Method<Packages.Z, [], Java.Wrapper<Packages.java.lang.Object>>
                };
        }

        interface J extends Java.Wrapper {
                clone: {
                overloads: [Java.Method<Packages.J, [], Java.Wrapper<Packages.java.lang.Object>>],
                overload(): Java.Method<Packages.J, [], Java.Wrapper<Packages.java.lang.Object>>
        };
        }

        interface F extends Java.Wrapper {
                clone: {
                overloads: [Java.Method<Packages.F, [], Java.Wrapper<Packages.java.lang.Object>>],
                overload(): Java.Method<Packages.F, [], Java.Wrapper<Packages.java.lang.Object>>
        };
        }

        interface D extends Java.Wrapper {
                clone: {
                overloads: [Java.Method<Packages.D, [], Java.Wrapper<Packages.java.lang.Object>>],
                overload(): Java.Method<Packages.D, [], Java.Wrapper<Packages.java.lang.Object>>
        };
        }

We can rewrite this as

interface StdClass<X> extends Java.Wrapper {
    clone: {
        overloads: [Java.Method<X, [], Java.Wrapper<Packages.java.lang.Object>>];
        overload(): Java.Method<X, [], Java.Wrapper<Packages.java.lang.Object>>;
    }
}

interface Z extends StdClass<Packages.Z> { }
interface J extends StdClass<Packages.J> { }
interface F extends StdClass<Packages.F> { }
interface D extends StdClass<Packages.D> { }

without loss of generality. That's 11 (much shorter) lines instead of 27, and the ratio only improves as more types get declared.

[RyanCavanaugh]

Looking later in the file, there are tons of just textually-duplicated overload signatures:

                overload(arg0: "java.lang.String", arg1: "long", arg2: "bool"): Java.Method<Packages.X._5Wn, [Java.Wrapper<Packages.java.lang.String>, number, boolean], boolean>,
                overload(arg0: "java.lang.String", arg1: "long", arg2: "bool"): Java.Method<Packages.X._5Wn, [Java.Wrapper<Packages.java.lang.String>, number, boolean], boolean>,
                overload(arg0: "java.lang.String", arg1: "bool"): Java.Method<Packages.X._5Wn, [Java.Wrapper<Packages.java.lang.String>, boolean], boolean>,
                overload(arg0: "org.npci.commonlibrary.widget.FormItemEditText"): Java.Method<Packages.X._5Wn, [Java.Wrapper<Packages.org.npci.commonlibrary.widget.FormItemEditText>], boolean>,
                overload(arg0: "int"): Java.Method<Packages.X._5Wn, [number], Array<number>>,
                overload(arg0: "int"): Java.Method<Packages.X._5Wn, [number], Array<number>>,

Other overloads are unreachable due to prior overloads that would always be selected first, likely indicating something is wrong with the generator. Every line except the first one here isn't doing anything:

                overload(): Java.Method<Packages.X._5Xb, [], boolean>,
                overload(): Java.Method<Packages.X._5Xb, [], Java.Wrapper<Packages.java.lang.Object>>,
                overload(): Java.Method<Packages.X._5Xb, [], Java.Wrapper<Packages.org.npci.commonlibrary.widget.FormItemEditText>>,
                overload(): Java.Method<Packages.X._5Xb, [], Java.Wrapper<Packages.X._69s>>,
                overload(): Java.Method<Packages.X._5Xb, [], number>,
                overload(): Java.Method<Packages.X._5Xb, [], Java.Wrapper<Packages.java.lang.String>>,

Other properties have textually-identical declarations and don't re-use them:

                setAlertIconTint: {
                overloads: [Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.android.content.Context>], null>, Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.android.graphics.drawable.Drawable>], null>, Java.Method<Packages.X._5XW, [number], null>, Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.java.lang.String>], null>, Java.Method<Packages.X._5XW, [], Java.Wrapper<Packages.java.lang.Object>>, Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.android.view.View$OnClickListener>], null>, Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.java.lang.String>], null>, Java.Method<Packages.X._5XW, [number], null>, Java.Method<Packages.X._5XW, [number], null>, Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.java.lang.String>], null>, Java.Method<Packages.X._5XW, [number], null>],
                overload(arg0: "android.content.Context"): Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.android.content.Context>], null>,
                overload(arg0: "android.graphics.drawable.Drawable"): Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.android.graphics.drawable.Drawable>], null>,
                overload(arg0: "int"): Java.Method<Packages.X._5XW, [number], null>,
                overload(arg0: "java.lang.String"): Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.java.lang.String>], null>,
                overload(): Java.Method<Packages.X._5XW, [], Java.Wrapper<Packages.java.lang.Object>>,
                overload(arg0: "android.view.View$OnClickListener"): Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.android.view.View$OnClickListener>], null>,
                overload(arg0: "java.lang.String"): Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.java.lang.String>], null>,
                overload(arg0: "int"): Java.Method<Packages.X._5XW, [number], null>,
                overload(arg0: "int"): Java.Method<Packages.X._5XW, [number], null>,
                overload(arg0: "java.lang.String"): Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.java.lang.String>], null>,
                overload(arg0: "int"): Java.Method<Packages.X._5XW, [number], null>
        };
                setAlertType: {
                overloads: [Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.android.content.Context>], null>, Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.android.graphics.drawable.Drawable>], null>, Java.Method<Packages.X._5XW, [number], null>, Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.java.lang.String>], null>, Java.Method<Packages.X._5XW, [], Java.Wrapper<Packages.java.lang.Object>>, Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.android.view.View$OnClickListener>], null>, Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.java.lang.String>], null>, Java.Method<Packages.X._5XW, [number], null>, Java.Method<Packages.X._5XW, [number], null>, Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.java.lang.String>], null>, Java.Method<Packages.X._5XW, [number], null>],
                overload(arg0: "android.content.Context"): Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.android.content.Context>], null>,
                overload(arg0: "android.graphics.drawable.Drawable"): Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.android.graphics.drawable.Drawable>], null>,
                overload(arg0: "int"): Java.Method<Packages.X._5XW, [number], null>,
                overload(arg0: "java.lang.String"): Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.java.lang.String>], null>,
                overload(): Java.Method<Packages.X._5XW, [], Java.Wrapper<Packages.java.lang.Object>>,
                overload(arg0: "android.view.View$OnClickListener"): Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.android.view.View$OnClickListener>], null>,
                overload(arg0: "java.lang.String"): Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.java.lang.String>], null>,
                overload(arg0: "int"): Java.Method<Packages.X._5XW, [number], null>,
                overload(arg0: "int"): Java.Method<Packages.X._5XW, [number], null>,
                overload(arg0: "java.lang.String"): Java.Method<Packages.X._5XW, [Java.Wrapper<Packages.java.lang.String>], null>,
                overload(arg0: "int"): Java.Method<Packages.X._5XW, [number], null>
        };

10,000 classes with a few dozen methods each, defined reasonably, is no problem for TypeScript - the built-in DOM .d.ts alone declares ~1,500 types. Writing all of them out as a massive in-place structural expansion with almost no re-use definitely is a problem, though.

[typescript-bot]

This issue has been marked as "Won't Fix" and has seen no recent activity. It has been automatically closed for house-keeping purposes.