How to improve the accuracy of generating 3D models from 2D drawings

[wangbochao1]

Your project is excellent and impressive; Our team is currently researching this direction, generating 3D models through 2D drawings. Due to industrial needs, we hope to achieve an accuracy rate of 100%. May I ask if you have any ideas to ensure an accuracy rate of 100%. Can it be achieved through RAG technology

[devBjn]

@neka-nat It's an amazing project. After testing many times, I recognize that sometimes it will return errors on the 1st try. However, it will run successfully on the 1st try with another image 2D. So you can explain this issue to me. Thank you so much

[wangbochao1]

@neka-nat It's an amazing project. After testing many times, I recognize that sometimes it will return errors on the 1st try. However, it will run successfully on the 1st try with another image 2D. So you can explain this issue to me. Thank you so much

@devBjn Please send out your two pictures and take a look. I'll see if we can optimize the effect by adjusting the prompt .

[neka-nat]

Thank you for your question! I believe it’s quite challenging to achieve 100% accuracy, but there are a few methods that could improve precision further.

• The first method is fine-tuning. This requires training data, but in the case of the 2D-to-3D problem, I believe the training data can be relatively easy to prepare. Specifically, by preparing CAD codes, we can automatically generate 2D drawings. This allows us to create pairs of CAD codes and corresponding drawings.

• The second method is to enhance the evaluation function using LLMs. Currently, we have feedback in place to check the rendered CAD image against the input drawing, but these images are from different perspectives. By making them from the same perspective, there’s a possibility of detecting which dimensions are incorrect using image processing or similar techniques. If we can explicitly indicate in the prompt where the error lies, more accurate drawings could be generated.

• The third method is to improve accuracy by changing the prompts for each component. For example, by preparing prompts or templates specifically designed for gears, flanges, bolts, and other parts in advance, we can switch to the appropriate prompts depending on the type of drawing given. The LLM can classify which component the drawing corresponds to, allowing for switching to the correct prompt for each case.