Memory leak when converting from terraform to pcl

brandonpollack23 commented 1 month ago

What happened?

When convertering

the program executes until it is killed by the OS for consuming too much memory.

Upon profiling, increasing memory pressure on the heap can be observed, indicating a memory leak.

trace

(open with go tool pprof -http :8080 FILE)

Example

NA

Output of `pulumi about`

NA

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

This is a related bug to #21

brandonpollack23 commented 1 month ago

Notes:

If I exit the program and do not generate the pp files into a tempdir before running binding/conversion it converts from terraform just fine. The issue is in our binding (perhaps in fetching/converting deps).

Here is the pcl project code which results in an oom if trying to convert it to anything (even just into pcl again!)

https://drive.google.com/open?id=1-2zY0BuTO3tTYeX7Ij6rbVZmat5b7_VV&usp=drive_fs

Here is the profile:

Zaid-Ajaj commented 1 month ago

As described in https://github.com/pulumi/pulumi/pull/17538, the problem was not really a memory leak but rather an infinite recursion because of the the CLI didn't check if a component was binding the same directory it was in and the converter is unable to convert referenced modules that live outside of its entry point. Consider this structure:

| -- parentFolder (dir)
      | -- main.tf
      | -- examples (dir)
               | -- first_example (dir)
                          |-- main.tf

if you were running the converter from first_example and that example had a module reference that looked like this:

module first_example {
  source "../.."
}

then we fail to convert the top-level module because the the destination directory starts at / (current directory). Honestly I am not sure how we supported this before because that would require copying the top-evel structure into the cwd of the converter (where pulumi convert is running from.

brandonpollack23 commented 1 month ago

As Zaid mentioned, #17538 fixes the out of resources issue but now has the following error:

================================================================================
                The Pulumi CLI encountered a code generation error. This is a bug!
                We would appreciate a report: https://github.com/pulumi/pulumi/issues/
                Please provide all of the below text in your report.
                ================================================================================
                Pulumi Version:   3.136.2-dev.0
                error: main.pp:18,1-21,2: /tmp/pulumi-convert196531652/vpc_5.13.0/main.pp:97,5-115: cannot iterate over a value of type number; , and 43 other diagnostic(s); /tmp/pulumi-convert196531652/vpc_5.13.0/main.pp:97,5-115: cannot iterate over a value of type number; , and 43 other diagnostic(s)
                error: main.pp:27,1-33,2: cannot bind component mainSg from the same directory as the parent program; cannot bind component mainSg from the same directory as the parent program
                error: main.pp:39,1-236,2: cannot bind component completeSg from the same directory as the parent program; cannot bind component completeSg from the same directory as the parent program
                error: main.pp:242,1-274,2: cannot bind component ipv4Ipv6Example from the same directory as the parent program; cannot bind component ipv4Ipv6Example from the same directory as the parent program
                error: main.pp:280,1-288,2: cannot bind component fixedNameSg from the same directory as the parent program; cannot bind component fixedNameSg from the same directory as the parent program
                error: main.pp:294,1-302,2: cannot bind component onlyRules from the same directory as the parent program; cannot bind component onlyRules from the same directory as the parent program
                error: main.pp:322,1-338,2: cannot bind component prefixList from the same directory as the parent program; cannot bind component prefixList from the same directory as the parent program
                error: main.pp:355,1-371,2: cannot bind component prefixListSg from the same directory as the parent program; cannot bind component prefixListSg from the same directory as the parent program
                error: could not generate output program

So the next step is to execute the copying Zaid mentioned the parent module dirs.

pulumi / pulumi-converter-terraform