Think about how to speed up semantic similarity

TeMU-BSC / iberifier

2 stars 0 forks source link

Think about how to speed up semantic similarity #29

Open BlancaCalvo opened 2 years ago

BlancaCalvo commented 2 years ago

Parallelize
Pre-filter telegram with keywords (maybe menéame too)
Pre-filter by length
Creating windows to have concatenated conversations

BlancaCalvo commented 2 years ago

@asier-gutierrez propone https://www.philschmid.de/static-quantization-optimum

cuquiwi commented 2 years ago

Seems too complicated to accelerate just a little. I will parallelize. The current Memory in the VM allows us to use 2 threads without swapping memory in theory.

cuquiwi commented 2 years ago

Finally parallelizing the encoding of the messages is a bad idea (even not possible). And the encoding already uses multiprocessing. So I will try Asier's method