search

bertvandepoel / snelSLiM

A linguistic set of tools in Go and web interface in PHP to do quick Stable Lexical Marker Analysis
GNU Affero General Public License v3.0
3 stars 0 forks source link

Look into multithreading #18

Closed bertvandepoel closed 4 years ago

bertvandepoel commented 5 years ago

A large corpus with lots of files (for example 2G 1198 files) takes a very long time to calculate all the scores for in the analysis. Potentially, parts of the n*m calculation of G tests and the like could be split up into threads based on a split keyword frequency list.

bertvandepoel commented 5 years ago

Talked to people on Freenode IRC #go-nuts: 01:43 < dho> Bert_2: you'll be fine slicing up an array and spinning up runtime.GOMAXPROCS goroutines 01:44 < dho> use runtime.GOMAXPROCS goroutines. you've got a CPU-bound problem, and that should be the magic number you need. 01:45 < dho> Bert_2: just like your regular function except you pass them a slice instead of the array 01:45 < dho> so they all have a distinct view into it 01:46 < dho> go func(s []T) { / your logic / }(data[lo:hi]) // update lo and hi