ashander / ftprime

Forward-time simulation of the msprime data structure (for development)
2 stars 1 forks source link

Switch to append_columns method for adding records #41

Open ashander opened 6 years ago

ashander commented 6 years ago

per discussion elsewhere, it's much faster but only if you avoid lists. as @molpopgen summarized:

passing lists to append_columns is a performance killer, too. One solution for ftprime will be to use numpy arrays (with the correct dtypes, which I can write down in the README for my project when I get a moment...) and act like you're in C:


x = np.zeroes([INITIAL_SIZE], dtype = the_right_type)
x_size = 0 #dummy variable

fill X with new data, and x_size += 1 each time


> When appropriate, send the slice `x[:x_size]` to the correct version of `append_columns`.

> When filling, if `x_size` will become `len(x)` after adding data, create a new array that's the concatenation of `x` and a new empty array of size `INITIAL_SIZE`.