Open mrbende opened 2 months ago
so...you want to reduce the dimensionality of your embeddings and visualize them in 2D/3D?
something like this would do:
prompt: "i have a large dataset which i've embedded and i'd like to reduce the dimensionality of these embeddings and visualize them using 3d-force-graph in a pure next js app. do as much of this in js as possible even the PCA reduction. do it all in a single react component"
gpt-4o:
Alright, let's implement everything in a single React component within a Next.js app. This component will fetch the embeddings, perform PCA to reduce their dimensions, and visualize them using `3d-force-graph`.
If you haven't already, create a Next.js app:
npx create-next-app@latest my-3d-graph-app
cd my-3d-graph-app
npm install
Install the necessary dependencies:
npm install ml-pca 3d-force-graph react-three-fiber three
Create a file called pages/index.js
and add the following code:
import { useState, useEffect, useRef } from 'react';
import Head from 'next/head';
import PCA from 'ml-pca';
import ForceGraph3D from '3d-force-graph';
const performPCA = (data) => {
const pca = new PCA(data);
const reducedData = pca.predict(data, { nComponents: 3 }).to2DArray();
return reducedData.map(([x, y, z]) => ({ x, y, z }));
};
export default function Home() {
const [data, setData] = useState({ nodes: [], links: [] });
const containerRef = useRef();
useEffect(() => {
const fetchEmbeddings = async () => {
const response = await fetch('/embeddings.json');
const embeddings = await response.json();
const reducedEmbeddings = performPCA(embeddings);
const nodes = reducedEmbeddings.map((emb, id) => ({
id: id.toString(),
x: emb.x,
y: emb.y,
z: emb.z,
}));
const links = []; // Define your links if you have any
setData({ nodes, links });
};
fetchEmbeddings();
}, []);
useEffect(() => {
if (data.nodes.length > 0) {
const graph = ForceGraph3D()(containerRef.current)
.graphData(data)
.nodeAutoColorBy('id');
return () => {
graph._destructor();
};
}
}, [data]);
return (
<div>
<Head>
<title>3D Graph</title>
<link rel="icon" href="/favicon.ico" />
</Head>
<div ref={containerRef} style={{ width: '100%', height: '100vh' }} />
</div>
);
}
embeddings.json
Place the embeddings.json
file in the public
directory of your Next.js app.
performPCA
function reduces the dimensionality of the embeddings to 3 dimensions using PCA.embeddings.json
) in the public
directory.ForceGraph3D
library is used to render the graph inside a container div
using a ref (containerRef
).Run your Next.js app with:
npm run dev
Open http://localhost:3000
in your browser, and you should see your 3D force graph visualizing the embeddings.
Let me know if you need any more help!
I came across this repository looking for a force-directed network engine I could render in a next js app... I have a large network I'm looking to render, and would like to define a custom function (or utilize some existing tool) that uses force directed behavior in combination with spatial projection of UMAP from vector embeddings of node ids...
Is there an established method, or integration with this react repo, for achieving this? Thanks!