Closed Kurokabe closed 8 months ago
@Kurokabe ah ok, so the issue is with prompting and kv cache turned on
you are actually the first one to test mesh prompting! how well is it working without kv cache? can you share what you are seeing?
yea i can fix this later today, in addition to making the quantizer more memory efficient
Thanks for looking into this! 😄
Some results on the train set with a gpt trained up to a loss of 3.293. The first row is the ground truth The second row shows the first 100 faces of the ground truth given to the gpt The last row is the gpt output
@Kurokabe not bad! thank you Farid! will let you know once i get this fixed tonight 🚀
Some results on the train set with a gpt trained up to a loss of 3.293. The first row is the ground truth The second row shows the first 100 faces of the ground truth given to the gpt The last row is the gpt output
Looks very good! Did you train on the 15k dataset and are you prompting through codes?
The loss seem high but I think the paper had metric of 1.4, not sure if that was their transformer loss.
Some results on the train set with a gpt trained up to a loss of 3.293. The first row is the ground truth The second row shows the first 100 faces of the ground truth given to the gpt The last row is the gpt output
Looks very good! Did you train on the 15k dataset and are you prompting through codes?
That's exact, when prompting with the start of the mesh, it's able to create meaningful content, but when tasked to generate from nothing (gpt.generate(temperature=0)
) it creates a messy mesh
Some results on the train set with a gpt trained up to a loss of 3.293. The first row is the ground truth The second row shows the first 100 faces of the ground truth given to the gpt The last row is the gpt output
Looks very good! Did you train on the 15k dataset and are you prompting through codes?
That's exact, when prompting with the start of the mesh, it's able to create meaningful content, but when tasked to generate from nothing (
gpt.generate(temperature=0)
) it creates a messy mesh
I agree, it's much better to give it a little push in the right direction. I've managed to create multi able different objects using the text generation but it needs such a low loss rate. I haven't tried prompting it but I'm guessing makes it so it can generate meshes with relative high training loss.
I wonder how much the cross attention with text helps it at the start, it might be interesting testing the impact of the prompt tokens when using text as well. If the text + prompt give much better results vs just text, it might be worth for @lucidrains to revisit to see if there's any way to increase the impact by the text during the mesh generation.
@Kurokabe hey Farid, after a few hours of debugging (hierarchical transformers are confusing), i think i finally figured out the issue
do you want to try 0.5.7 and see if it fixes your original script?
It works perfectly now, thank you for fixing it so fast 🙏
@Kurokabe you don't happen to work at one of the startups that has been reaching out to me through email about meshgpt, are you?
@Kurokabe you don't happen to work at one of the startups that has been reaching out to me through email about meshgpt, are you?
What's the name of the company? I didn't hear something about that anyway
@Kurokabe ohh nvm, i'm receiving some emails from startups in the 3d space. thought you may had been part of one of them
@Kurokabe you are in academia?
@Kurokabe you are in academia?
Kind of, I work at a company atm, but I'll soon start a PhD
@lucidrains
Hey,
I tested prompt tokens + text and by only using 1 token it managed to kick-start the generation. So it would seem that the cross-attention of the text have a very weak impact on the first tokens, any ideas around that? If it's possible to increase the impact of the cross-attention to have a stronger relationship with the mesh, then image-to-3d might be possible due a image is a just a bigger vector (kind of).
Text + tokens, using 0 to 5 tokens
@Kurokabe you don't happen to work at one of the startups that has been reaching out to me through email about meshgpt, are you?
@lucidrains I do 🚀
I wanted to experiment how the MeshTransformer is able to complete a mesh by giving the initial codes, but there is a problem where I think the prompt codes are not correctly given down the line. Here is a small debug code:
It gives the following error:
If I disable however the kv cache with
generated = gpt.generate(prompt=codes, cache_kv=False)
, it works (albeit being slow).With the cache, in x_transformers > Attention > forward,
ck.shape=[202,16,6,64]
andk.shape=[2, 16, 1, 64]
causing the shape mismatch error (same shapes for cv and v after)