Loading multiple math font is slow

Description

If setting more than one fonts, then the time for loading a single font becomes longer and longer.

Add info or delete as appropriate:

[x] Relevant for XeTeX
[x] Relevant for LuaTeX (I haven't test with LuaTeX but the code is the same)
[x] Issue tracker has been searched for similar issues?
[ ] Links to discussion if appropriate

Further details

\documentclass{article}
\usepackage{l3benchmark,unicode-math}

\ExplSyntaxOn
\clist_map_inline:nn
  {
    Thin, UltraLight, ExtraLight, Light, Book, Regular,
    Medium, SemiBold, Bold, ExtraBold, Heavy, Ultra
  }
  { \benchmark_once:n { \setmathfont { FiraMath-#1.otf } [ version = #1 ] } }
\ExplSyntaxOff

\begin{document}
NULL
\end{document}

On my computer, a typical result is as the following:

...
0.12 seconds (5.2e5 ops)
0.138 seconds (6e5 ops)
0.124 seconds (5.43e5 ops)
0.126 seconds (5.51e5 ops)
0.127 seconds (5.52e5 ops)
0.129 seconds (5.57e5 ops)
0.129 seconds (5.64e5 ops)
0.13 seconds (5.67e5 ops)
0.131 seconds (5.69e5 ops)
0.146 seconds (6.41e5 ops)
0.136 seconds (6.02e5 ops)
0.145 seconds (6.24e5 ops)
...

You may find that the time used and operations are all increasing.

After some investigation, I find the problem is in \@@_mathmap_noparse:nnn, where

\cs_new:Nn \__um_mathmap_noparse:nnn
  {
    % DEBUG
    % \tl_log:c { g__um_switchto_#1_tl }
    \tl_gput_right:cx { g__um_switchto_#1_tl }
      {
        \__um_set_mathcode:nnnn {#2} {\mathalpha} {\l__um_symfont_label_tl} {#3}
      }
  }

Notice that if #1 is literal here, \g__um_switchto_#1_tl will not be cleared. As we loading more fonts, this tl becomes extremely large, so makes the expansion and assignment slower and slower. For other situations e.g. #1=up, there is no such problem.

latex3 / unicode-math