search

beeldengeluid / dane-visual-feature-extraction-worker

Uses the VizXP feature extraction to produce features/vector spaces
MIT License
0 stars 0 forks source link

34 drop audio processing #37

Closed Veldhoen closed 4 months ago

Veldhoen commented 6 months ago

I think the functionality is in place to fix #34, and the tests all pass now On the go, changed spelling for all mentions of spectrogram

Todo: integration test, run containerized end-to-end

Veldhoen commented 5 months ago

Added some more tests and improvements here and there. Also, added Dockerfile for GPU support. For me, building repeatedly fails on poetry connection issues, which I can circumvent by leaving the poetry.lock file out of the image.

I ran this version end to end in a container, tgat is: without mounting anything but the config, downloading the model and input from S3 and uploading the output to S3. It all seems to work fine!

mwigham commented 5 months ago

@Veldhoen what do you want for the review? Look at the code, run locally (non-GPU), run on a server (both ways), ...?

Veldhoen commented 5 months ago

It would be great if you could run it locally to see if you encounter any issues. And also to run the tests locally, as not all of them can be run by Github actions (due to the model not being public). I left audio processing in there as a legacy option, but don't spend too much time on that. I'm also curious to see if the GPU running actually works, but I think our only option currently is to deploy it in Openshift - which is my next step. If you can find the time, a scan of the code (changes) is always appreciated ofcours. Especially for the tests as I feel I still have a lot to learn about proper testing.