RuntimeError LLVM backend supports up to 512 snode trees

Describe the bug

I am running a series of taichi kernels in a for loop. 63 of these loops run before I run into this error.

RuntimeError: [snode_tree_buffer_manager.cpp:taichi::lang::SNodeTreeBufferManager::allocate@44] LLVM backend supports up to 512 snode trees

To Reproduce Unfortunately my code is quite long.

Log/Screenshots Please post the full log of the program (instead of just a few lines around the error message, unless the log is > 1000 lines). This will help us diagnose what's happening. For example:

[E 09/22/23 14:00:00.722 16516] [snode_tree_buffer_manager.cpp:taichi::lang::SNodeTreeBufferManager::allocate@44] LLVM backend supports up to 512 snode trees
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.1.3\plugins\python-ce\helpers\pydev\pydevconsole.py", line 364, in runcode
    coro = func()
  File "<input>", line 1, in <module>
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.1.3\plugins\python-ce\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.1.3\plugins\python-ce\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:\Users\krhee\PycharmProjects\shade-engine\main.py", line 121, in <module>
    system_new_basis_orthogonal, system_2d_orthogonal, a_orthogonal = change_basis_to_2d_plane(
  File "C:\Users\krhee\PycharmProjects\shade-engine\controllers\change_basis.py", line 89, in change_basis_to_2d_plane
    a = change_basis()
  File "C:\Users\krhee\PycharmProjects\shade-engine\venv\lib\site-packages\taichi\lang\kernel_impl.py", line 974, in wrapped
    return primal(*args, **kwargs)
  File "C:\Users\krhee\PycharmProjects\shade-engine\venv\lib\site-packages\taichi\lang\kernel_impl.py", line 905, in __call__
    key = self.ensure_compiled(*args)
  File "C:\Users\krhee\PycharmProjects\shade-engine\venv\lib\site-packages\taichi\lang\kernel_impl.py", line 873, in ensure_compiled
    self.materialize(key=key, args=args, arg_features=arg_features)
  File "C:\Users\krhee\PycharmProjects\shade-engine\venv\lib\site-packages\taichi\lang\kernel_impl.py", line 560, in materialize
    self.runtime.materialize()
  File "C:\Users\krhee\PycharmProjects\shade-engine\venv\lib\site-packages\taichi\lang\impl.py", line 459, in materialize
    self.materialize_root_fb(not self.materialized)
  File "C:\Users\krhee\PycharmProjects\shade-engine\venv\lib\site-packages\taichi\lang\impl.py", line 394, in materialize_root_fb
    root.finalize(raise_warning=not is_first_call)
  File "C:\Users\krhee\PycharmProjects\shade-engine\venv\lib\site-packages\taichi\_snode\fields_builder.py", line 170, in finalize
    return self._finalize(raise_warning, compile_only=False)
  File "C:\Users\krhee\PycharmProjects\shade-engine\venv\lib\site-packages\taichi\_snode\fields_builder.py", line 182, in _finalize
    return SNodeTree(_ti_core.finalize_snode_tree(_snode_registry, self.ptr, impl.get_runtime().prog, compile_only))
RuntimeError: [snode_tree_buffer_manager.cpp:taichi::lang::SNodeTreeBufferManager::allocate@44] LLVM backend supports up to 512 snode trees

$ python my_sample_code.py
    ti.init(
        arch=arch,                  # compiled for which architecture
        default_fp=ti.float32,      # default float precision
        device_memory_fraction=0.9
    )
[Taichi] version 1.6.0
...

Additional comments If possible, please also consider attaching the output of command ti diagnose. This produces the detailed environment information and hopefully helps us diagnose faster.

If you have local commits (e.g. compile fixes before you reproduce the bug), please make sure you first make a PR to fix the build errors and then report the bug.

(venv) C:\Users\krhee\PycharmProjects\shade-engine\_shade>ti diagnose
[Taichi] version 1.6.0, llvm 15.0.1, commit f1c6fbbd, win, python 3.10.4
cpu: True
metal: False
opengl: True
cuda: True
vulkan: True

`glewinfo` not available: [WinError 2] The system cannot find the file specified
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA RTX A200... WDDM  | 00000000:01:00.0  On |                  N/A |
| N/A   48C    P0    12W /  40W |   7945MiB /  8192MiB |     46%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2456      C   ...thon\Python310\python.exe    N/A      |
|    0   N/A  N/A      3484    C+G   ...zilla Firefox\firefox.exe    N/A      |
|    0   N/A  N/A      3828    C+G   ...wekyb3d8bbwe\ms-teams.exe    N/A      |
|    0   N/A  N/A      6224    C+G   ...ge\Application\msedge.exe    N/A      |
|    0   N/A  N/A      6564    C+G   ...zilla Firefox\firefox.exe    N/A      |
|    0   N/A  N/A      7376    C+G   ...e\PhoneExperienceHost.exe    N/A      |
|    0   N/A  N/A      8244    C+G   C:\Windows\explorer.exe         N/A      |
|    0   N/A  N/A     12660    C+G   ...n1h2txyewy\SearchHost.exe    N/A      |
|    0   N/A  N/A     12684    C+G   ...artMenuExperienceHost.exe    N/A      |
|    0   N/A  N/A     13664    C+G   ...cw5n1h2txyewy\LockApp.exe    N/A      |
|    0   N/A  N/A     17836    C+G   ...2txyewy\TextInputHost.exe    N/A      |
|    0   N/A  N/A     18152    C+G   ...045.36\msedgewebview2.exe    N/A      |
|    0   N/A  N/A     19764    C+G   ...045.31\msedgewebview2.exe    N/A      |
|    0   N/A  N/A     21332    C+G   ...045.31\msedgewebview2.exe    N/A      |
|    0   N/A  N/A     22532    C+G   ...y\ShellExperienceHost.exe    N/A      |
|    0   N/A  N/A     25168    C+G   ...oft OneDrive\OneDrive.exe    N/A      |
|    0   N/A  N/A     30016    C+G   ...logioptionsplus_agent.exe    N/A      |
|    0   N/A  N/A     32288    C+G   ...lPanel\SystemSettings.exe    N/A      |
+-----------------------------------------------------------------------------+

[Taichi] version 1.6.0, llvm 15.0.1, commit f1c6fbbd, win, python 3.10.4

[Taichi] version 1.6.0, llvm 15.0.1, commit f1c6fbbd, win, python 3.10.4
[Taichi] Starting on arch=x64

[Taichi] version 1.6.0, llvm 15.0.1, commit f1c6fbbd, win, python 3.10.4
[Taichi] Starting on arch=opengl

[Taichi] version 1.6.0, llvm 15.0.1, commit f1c6fbbd, win, python 3.10.4
[Taichi] Starting on arch=cuda

[Taichi] version 1.6.0, llvm 15.0.1, commit f1c6fbbd, win, python 3.10.4

*******************************************
**      Taichi Programming Language      **
*******************************************

Docs:   https://docs.taichi-lang.org/
GitHub: https://github.com/taichi-dev/taichi/
Forum:  https://forum.taichi.graphics/

                                TAICHI EXAMPLES
+-----------------------------------------------------------------------------+
| 0: ad_gravity           | 25: keyboard            | 50: pbf2d               |
| 1: circle_packing_image | 26: laplace             | 51: physarum            |
| 2: comet                | 27: laplace_equation    | 52:                     |
|                         |                         | poisson_disk_sampling   |
| 3: cornell_box          | 28: mandelbrot_zoom     | 53: print_offset        |
| 4: diff_sph             | 29: marching_squares    | 54: rasterizer          |
| 5: euler                | 30: mass_spring_3d_ggui | 55: regression          |
| 6: eulerfluid2d         | 31: mass_spring_game    | 56: sdf_renderer        |
| 7: explicit_activation  | 32:                     | 57: simple_derivative   |
|                         | mass_spring_game_ggui   |                         |
| 8: export_mesh          | 33: mciso_advanced      | 58: simple_texture      |
| 9: export_ply           | 34: mgpcg               | 59: simple_uv           |
| 10: export_videos       | 35: mgpcg_advanced      | 60: snow_phaseField     |
| 11: fem128              | 36: minimal             | 61: stable_fluid        |
| 12: fem128_ggui         | 37: minimization        | 62: stable_fluid_ggui   |
| 13: fem99               | 38: mpm128              | 63: stable_fluid_graph  |
| 14: fractal             | 39: mpm128_ggui         | 64: taichi_bitmasked    |
| 15: fractal3d_ggui      | 40: mpm3d               | 65: taichi_dynamic      |
| 16: fullscreen          | 41: mpm3d_ggui          | 66: taichi_logo         |
| 17: game_of_life        | 42: mpm88               | 67: taichi_ngp          |
| 18: gui_image_io        | 43: mpm88_graph         | 68: taichi_sparse       |
| 19: gui_widgets         | 44: mpm99               | 69: texture_graph       |
| 20: implicit_fem        | 45:                     | 70: tutorial            |
|                         | mpm_lagrangian_forces   |                         |
| 21:                     | 46: nbody               | 71:                     |
| implicit_mass_spring    |                         | two_stream_instability  |
| 22:                     | 47: odop_solar          | 72: vortex_rings        |
| initial_value_problem   |                         |                         |
| 23: jacobian            | 48: oit_renderer        | 73: waterwave           |
| 24:                     | 49: patterns            |                         |
| karman_vortex_street    |                         |                         |
+-----------------------------------------------------------------------------+
42
Running example minimal ...
[Taichi] Starting on arch=x64
42.0
>>> Running time: 0.25s

Consider attaching this log when maintainers ask about system information.
>>> Running time: 6.11s

taichi-dev / taichi

RuntimeError LLVM backend supports up to 512 snode trees #8355