Taking too much memory with (relatively) larger files

[sudoCss]

Describe the bug I'm not sure if this is normal/expected or a known issue so sorry in advance about my ignorance. When I try to templ generate a single (programmatically generated) 6.4MiB .templ file which contains 4048 templ definitions each one is an svg from Game Icons, it takes all the available memory and the laptop starts lagging and even if I leave it like this for a long time it won't finish and the memory consumption will lower sometimes to like 2GB then go back to taking all the available memory.. Also the LSP(vscode with the templ extencion) does the same thing when I open the same file in vscode.

To Reproduce Clone this repo and run templ generate in it.

Expected issue cause I'm not familiar with the templ codebase but I think it may be in issue in a parser of some kind (may be specific to SVG parsing).

Screenshots

templ info output

❯ templ info
(✓) os [ goos=linux goarch=amd64 ]
(✓) go [ location=/usr/bin/go version=go version go1.23.1 linux/amd64 ]
(✓) gopls [ location=/home/sudocss/.go/bin/gopls version=golang.org/x/tools/gopls v0.16.2 ]
(✓) templ [ location=/home/sudocss/.go/bin/templ version=v0.2.778 ]

Desktop:

• OS: Linux(EndeavourOS)
• templ CLI version: v0.2.778
  • Go version: go1.23.1 linux/amd64
  • gopls version: golang.org/x/tools/gopls v0.16.2

[a-h]

Thanks for raising that. I'd have to take a look at a memory profile to see where all that RAM is being used, and whether any of it can be reclaimed more aggressively.

For your library (cool idea, by the way!), the best approach might be to use the Go embed feature to load the files in directly, and use Go's type system to implement the templ.Component interface in code.

If you drop the svg files into an icons directory, you can use embed to embed the file contents into the Go package, then implement the templ.Component interface in code.

package templicons

import (
    "context"
    _ "embed"
    "io"
)

type IconComponent []byte

func (ic IconComponent) Render(ctx context.Context, w io.Writer) (err error) {
    _, err = w.Write(ic)
    return err
}

//go:embed icons/3dGlasses.svg
var Glasses3D IconComponent

//go:embed icons/3dHammer.svg
var Hammer3D IconComponent

[sudoCss]

Thank you for your time and your grate work!

(cool idea, by the way!)

Thanks but it is unfortunately not my idea actually🫣 I'm just trying to replicate React Icons in go and templ

the best approach might be...

I really appreciate the suggestion! I'll definitely consider that and see how it goes.

[sudoCss]

Unfortunately I couldn't figure out a good way using your suggested approach so I went back to my original (dumber) approach😅...

For anyone interested, the library is now somehow usable.. GitHub repo | GitLab repo, and the really bad code that generates it all can be found here

P.S: Even the gitlab ci/cd machine(same 8GB RAM but no desktop/browser/editor/etc.. running like my laptop) couldn't handle the larger icon sets(3k+ icons) and the process keep getting killed after about half an hour or so

[utrack]

I just ran pprof and collected a heap dump... seems like the issue is that goexpression.Func() tries to parse the same file over and over again.

The current logic seems to be:

• the parser takes one file,
• parses it as a whole,
• gets the first function and
• discards the rest of the data.

So in the sample file, the first run of goexpression.Func() gets the whole file, then the whole file minus first template, then whole file minus first two templates etc.

On the screenshot, the root cause seems to be the src escaping to heap - expr = src[start:end] takes a slice out of string and returns it; but the underlying slice stays the same.

When you combine the above, the memory consumption becomes n! - the heap contains n+(n-3)+(n-3-3)+(n-3-3-3)... lines; assuming every icon is 3 lines long.

The logic is a bit wasteful for the CPU (it's best to astparse one file exactly once), but there is a hotfix for the mem leak itself. I'll push the MR in a second.

[utrack]

Just tested it - the sample repo takes 47 seconds and near-constant RAM on my laptop :)