we should be able to parallelize by dividing the file into chunks and reading each chunk in a separate thread or process
multi-threading will be harder to implement than multi-processing b/c of the python GIL and thus (might?) require a compiled extension to python but multi-processing will be slower and probably require copying of each chunk into the larger array
@d-laub and I have had some very productive chats about this and are working on a strategy 💪
we should be able to parallelize by dividing the file into chunks and reading each chunk in a separate thread or process
multi-threading will be harder to implement than multi-processing b/c of the python GIL and thus (might?) require a compiled extension to python but multi-processing will be slower and probably require copying of each chunk into the larger array
@d-laub and I have had some very productive chats about this and are working on a strategy 💪