Major re-structure of build-embeddings tool; Use multiple sv/topic with the same description now

[shifucun]

Major changes:

• Re-structure build embeddings script to use a few modularized functions from tools/nl/embeddings/utils.py
• Make base and custom embeddings build scripts almost identical now. Further clean up is needed to merge them.
• Use multiple sv/topic with the same description now. The diff can be inspected from _preindex.csv files
• Updated all the embedding indexes based on the changes
• svdiff report for base: https://storage.mtls.cloud.google.com/datcom-embedding-diffs/boxu_base_uae_mem_2024_06_20_00_09_46.html

Minor changes:

• Rename/Removed several non critical embedding indexes
• Updated documentation and scripts

TODO:

• clean up all custom_dc path assumptions in nl_server, web_server(admin) and build_embedding_tool.
• deprecate custom dc embedding build script following above.
• add back tests for tools/nl/embeddings/utils.py.

[shifucun]

Epic PR indeed!

Could this be broken into multiple smaller PRs by any chance?

This is one of the smaller PR after break up :) At this point, the effective code change is the two build_embeddings script which is very hard to do point fix as all the fundamental functions changes. The preindex file diff is a guard here.