Tuning for time + space

[evanberkowitz]

In the simulation pictured (κ=0.74 W=1 N=35 worldline) the autocorrelation time is about 120 MC steps (generated 10000 samples, after thermalization and decorrelation 83 samples).

The whole calculation on my new laptop took about 32 minutes. As you can see in the log, the generation was ~3 minutes, while the Spin_Spin observable was about 30 minutes! If we had only measured on 83 samples, it would have taken less than half a minute.

Importantly, the expensive Spin_Spin fluctuates well while the slowest observables ({X,T}Wrapping) are cheap. So, we could have generated all of the data, measured only a subset of the observables, decorrelated, and then saved an enormous amount of time.

It would be enormously beneficial to have a routine that attempts to balance these costs. By finding the autocorrelation time of the slow observables and using KeepEvery we could produce an ensemble requiring much less storage AND much less measurement time.

Generation: 100%|████████████████████████████████████████████████| 9999/9999 [02:40<00:00, 62.33it/s]
Links                       : 100%|█████████████████████████| 10000/10000 [00:00<00:00, 30773.27it/s]
InternalEnergyDensity       : 100%|████████████████████████| 10000/10000 [00:00<00:00, 291046.76it/s]
InternalEnergyDensitySquared: 100%|████████████████████████| 10000/10000 [00:00<00:00, 232195.18it/s]
ActionDensity               : 100%|████████████████████████| 10000/10000 [00:00<00:00, 426003.64it/s]
ActionTwoPoint              : 100%|█████████████████████████| 10000/10000 [00:00<00:00, 14955.82it/s]
WindingSquared              : 100%|█████████████████████████| 10000/10000 [00:00<00:00, 71435.70it/s]
Winding_Winding             : 100%|██████████████████████████| 10000/10000 [00:01<00:00, 9789.10it/s]
TorusWrapping               : 100%|████████████████████████| 10000/10000 [00:00<00:00, 336837.78it/s]
TWrapping                   : 100%|███████████████████████| 10000/10000 [00:00<00:00, 5224593.92it/s]
XWrapping                   : 100%|███████████████████████| 10000/10000 [00:00<00:00, 5141338.56it/s]
Spin_Spin                   : 100%|████████████████████████████| 10000/10000 [29:12<00:00,  5.71it/s]
SpinSusceptibility          : 100%|████████████████████████| 10000/10000 [00:00<00:00, 404515.90it/s]
SpinSusceptibilityScaled    : 100%|███████████████████████| 10000/10000 [00:00<00:00, 3373796.65it/s]
Vortex_Vortex               : 100%|█████████████████████████| 10000/10000 [00:00<00:00, 12278.90it/s]
VortexSusceptibility        : 100%|████████████████████████| 10000/10000 [00:00<00:00, 296532.50it/s]
VortexSusceptibilityScaled  : 100%|███████████████████████| 10000/10000 [00:00<00:00, 3323273.91it/s]

[alcherman]

Hi, I have a naive suggestion on this. I think we should NOT use Spin_Spin in the default list of observables when determining our estimate for the autocorrelation time, precisely because it takes a huge amount of time to evaluate. Instead, we can use the fast observables to estimate autocorrelation times. Once we have the (putatively) thermalized and decorrelated MC chain, we can compute Spin_Spin using that data, and calculate the autocorrelation from that. If it is roughly 1-5, then our original estimate of the autocorrelation time was good. If not, then we'll have to reexamine things manually.... But my bet is that having Spin_Spin be the parametrically slowest variable is likely to be unusual.

[evanberkowitz]

Absolutely agreed. Let me see if I can rephrase to make sure I really get what you're saying.

1. Have some initial chain where we only measure the known-slowly-changing observables (Wrapping seems likely culprits), and (may as well) the extremely cheap observables.
2. Compute the autocorrelation time τ.
3. Only keep every τth steep and measure the expensive observables there, and recompute τ (which if there is justice should now be ~1). If there's enough statistics just declare victory.
4. If there ain't enough statistics (τ is a nontrivial fraction of the whole initial chain) we can generate a real production chain where we only even ever store every τth step, since there's no sense in measuring more often than that anyway.

Does that match what you're saying?

[evanberkowitz]

(Also worth remembering that the wrapping observables are likely to tunnel faster with @SethBuesing's worm.)

[alcherman]

Yes, that's what I mean! (But as usual you made it way more precise than my vague handwaving =) )

On Tue, Jan 16, 2024 at 11:27 AM Evan Berkowitz @.***> wrote:

Absolutely agreed. Let me see if I can rephrase to make sure I really get what you're saying.

1. Have some initial chain where we only measure the known-slowly-changing observables (Wrapping seems likely culprits), and (may as well) the extremely cheap observables.
2. Compute the autocorrelation time τ.
3. Only keep every τth steep and measure the expensive observables there, and recompute τ (which if there is justice should now be ~1). If there's enough statistics just declare victory.
4. If there ain't enough statistics (τ is a nontrivial fraction of the whole initial chain) we can generate a real production chain where we only even ever store every τth step, since there's no sense in measuring more often than that anyway.

Does that match what you're saying?

— Reply to this email directly, view it on GitHub https://github.com/evanberkowitz/supervillain/issues/102#issuecomment-1894191073, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGTKMLL74M36MSYT6IUNIF3YO2Z6LAVCNFSM6AAAAABB2NMOJWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJUGE4TCMBXGM . You are receiving this because you commented.Message ID: @.***>

[SethBuesing]

Would we want this to be a notebook, or perhaps a class of its own that handles such a routine?

[evanberkowitz]

Notebooks won't be good for large-scale production, so probably a script of some kind. But we should think about the production scripts pretty carefully before jumping in. You can look at a very simplified example in the current main branch in example/production scripts.

[SethBuesing]

Alright excellent.