czg1225 / SlimSAM

SlimSAM: 0.1% Data Makes Segment Anything Slim
Apache License 2.0
248 stars 14 forks source link

Add Transformers.js update + demo #6

Closed xenova closed 5 months ago

xenova commented 5 months ago

Hi there! 🤗 This PR adds a bullet point for SlimSAM + transformers.js support (which was added today), allowing these models to be run directly in a browser, using a few lines of JS code. Demo video:

https://github.com/czg1225/SlimSAM/assets/26504141/1ad1840d-0304-49a1-a0fe-6f07070073c7

Demo link + source code: https://huggingface.co/spaces/Xenova/segment-anything-web Release notes: https://github.com/xenova/transformers.js/releases/tag/2.14.0


Transformers.js uses a very similar API as the python library, as can be seen with the following example code:

Example: Perform mask generation w/ Xenova/slimsam-77-uniform.

import { SamModel, AutoProcessor, RawImage } from '@xenova/transformers';

const model = await SamModel.from_pretrained('Xenova/slimsam-77-uniform');
const processor = await AutoProcessor.from_pretrained('Xenova/slimsam-77-uniform');

const img_url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/corgi.jpg';
const raw_image = await RawImage.read(img_url);
const input_points = [[[340, 250]]] // 2D localization of a window

const inputs = await processor(raw_image, input_points);
const outputs = await model(inputs);

const masks = await processor.post_process_masks(outputs.pred_masks, inputs.original_sizes, inputs.reshaped_input_sizes);
console.log(masks);
// [
//   Tensor {
//     dims: [ 1, 3, 410, 614 ],
//     type: 'bool',
//     data: Uint8Array(755220) [ ... ],
//     size: 755220
//   }
// ]

Visualization of masks:

animation

czg1225 commented 5 months ago

We have add the demo link in our readme, thanks!

xenova commented 5 months ago

Amazing! Will close this PR then 😇