Added FES integration including whisper

vitrivr / vitrivr-engine

vitrivr's next-generation retrieval engine. It is capable of extracting and retrieving a wider range of multimedia objects such as audio, video, images or 3d models.

https://vitrivr.org

MIT License

6 stars 3 forks source link

Added FES integration including whisper #57

Closed faberf closed 6 months ago

faberf commented 7 months ago

automated speech recognition
image captioning (with or without prompt)
OCR
clip
image classification

there are still some issues with the fixeddurationsegmenter:

if the duration is longer than the video, nothing happens
the last segment is missing

lucaro commented 7 months ago

there are still some issues with the fixeddurationsegmenter:
* if the duration is longer than the video, nothing happens

* the last segment is missing

What do you mean by 'nothing happens'? Is no segmentation performed, or are no segments emitted? If it is the latter, would this be the same issue as the second point you mentioned? Do you plan to address any of these things in this pull request since you are explicitly mentioning them here?

faberf commented 7 months ago

They are probably the same issue. I plan to address this early next week, I just wanted to already begin the review process, considering ralph asked me.

faberf commented 7 months ago

What is the status here?

faberf commented 7 months ago

Just pushed the fixes. @sauterl let me know if there is anything i missed

EDIT: I just realised I forgot to write the documentation of how to use the specific analyzers.

faberf commented 6 months ago

Also, as a note: it may be useful to transfer ownership of the feature-extraction-server repo to the vitrivr organization

lucaro commented 6 months ago

Also, as a note: it may be useful to transfer ownership of the feature-extraction-server repo to the vitrivr organization

Yes, please do!