scroll-tech / halo2

Other
43 stars 39 forks source link

Fix mockprover memory #85

Closed DreamWuGit closed 8 months ago

DreamWuGit commented 8 months ago

issue #86 in MockProver fork method, there are some fields' clone operation happens in following code snippets.
among them self.cs.clone() and self.instance.clone() increase about 90M memory costs each time.
in very complex test tool tests(like create2_recursive_xxx) , there are > 4000 regions being generated and result in more than 4000 times clone operations. this increase the peak memory remarkably. for normal test, there are only about 100 sub regions used, so no much memory going up.

sub_cs.push(Self {
                k: self.k,
                n: self.n,
                cs: self.cs.clone(),
                regions: vec![],
                current_region: None,
                fixed_vec: self.fixed_vec.clone(),
                fixed,
                advice_vec: self.advice_vec.clone(),
                advice,
                advice_prev: self.advice_prev.clone(),
                instance: self.instance.clone(),
                selectors_vec: self.selectors_vec.clone(),
                selectors,
                challenges: self.challenges.clone(),
                permutation: None,
                rw_rows: sub_range.clone(),
                usable_rows: self.usable_rows.clone(),
                current_phase: self.current_phase,
            });
        }

fix solution:
change both self.cs and self.instance to Arc type, after testing against latest zkevm develop branch(disabled batch_inv feature) peak memory size down to less than 100G while old test run take >300 G

[2024-03-13T02:16:08Z DEBUG halo2_proofs::dev] memory usage of mockprover fork_4107 ends vm_size 81 GB : vm_rss: 68GB

Velaciela commented 8 months ago

double checked by using real trace and halo2-gpu, peak mem < 120GB after this PR