search

LAION-AI / laion-3d

Collect large 3d dataset and build models
262 stars 23 forks source link

Annotation maker with Blender and CLIP Interrogator or BLIP #2

Open lalalune opened 1 year ago

lalalune commented 1 year ago

The datasets we have are decent model-wise, very low quality annotation wise.

I have experimented with running them through BLIP with really good success. With a 3D model on a black background, it pretty much always gets it right.

I've made a headless Blender script to capture the other data that we need: https://github.com/nv-tlabs/GET3D/pull/17/files#diff-c95401620d13c6e6aed785545efca7bda6dbdc91c68ed5bcec4cb9947f1f7bbf

This could be extracted to a new repo (this one?) and we can add automatic annotation, so it can just be a singular pipeline.

lalalune commented 1 year ago

https://github.com/lalalune/ImprovedShapenetRenderer

I've split the ShapeNet Renderer PR into a standalone repo.

@rom1504 what is SOTA for CLIP interrogation? Have been using https://github.com/salesforce/BLIP with great success on Shapenet, since we are rendering on a black background it's fairly accurate and low noise

rom1504 commented 1 year ago

I think try out the specific blip pipeline we used for laion coco. Links are in blogpost