iqbal-lab-org / make_prg

Code to create a PRG from a Multiple Sequence Alignment file
Other
21 stars 7 forks source link

Copy-on-write might not be efficient in python #47

Open leoisl opened 1 year ago

leoisl commented 1 year ago

See https://llvllatrix.wordpress.com/2016/02/19/python-vs-copy-on-write . TLDR: every python object has reference counting. If we are sharing read-only python objects hoping this will be efficient due to copy-on-write optimisation, it might not be efficient because even reading a python object increases its reference counting, i.e. we modify the reference count, thus we modify the page, and thus effectively copy it in the child process. However, I do remember testing data sharing and copy-on-write optimisation in python, and this did not show up on my tests I think... But I am not sure I read the whole large data object on the child process. Something to benchmark later