[Feat] add frame resampler to enable batch processing for video data - Githubissues

Luodian / Otter

🦦 Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.

https://otter-ntu.github.io/

MIT License

3.55k stars 242 forks source link

[Feat] add frame resampler to enable batch processing for video data #155

Closed Luodian closed 1 year ago

Luodian commented 1 year ago

Before you open a pull-request, please check if a similar issue already exists or has been closed before.

When you open a pull-request, please be sure to include the following

[x] A descriptive title: [xxx] XXXX
[x] A detailed description
[x] Assign an issue type tag (label):
- dataset (mimic-it download, usage, etc.),
- demo (online demo),
- doc (readme, wiki, paper, video, etc.),
- evaluation (evaluation result, performance of Otter, etc.),
- model (model configuration, components, etc.),
- train (training configuration, process, code, etc.)

Thank you for your contributions!

    def resample_frames(self, image_ids, resample_frames):
        indices = np.linspace(0, len(image_ids) - 1, resample_frames, dtype=int)
        image_ids = [image_ids[i] for i in indices]
        assert len(image_ids) == resample_frames
        return image_ids