various optimizations - Githubissues

graph-genome / component_segmentation

Read in ODGI Bin output and identify co-linear components

Apache License 2.0

3 stars 4 forks source link

various optimizations #47

Closed dimatr closed 4 years ago

dimatr commented 4 years ago

The biggest change is the introduction of the sparse containers. This feature can be switched on by setting JSON_VERSION = 15 in the __init__.py. Otherwise the output should be identical to the current v14

LinkColumn.participants and Component.occupants hold only the relevant path_ids
Component.matrix holds only the non-empty info in the form [ [path_id, [shift_array, bin_array]] ]

There's also additional processing speed-up and lower memory consumption.

josiahseaman commented 4 years ago

Thank you Dmytro. I wasn't expecting you to build this in as a switch. I think it will be fine to migrate the data and schematize to v15. That's a small enough change I believe I can do it today.

dimatr commented 4 years ago

A switch is a temp helper 1) to check the correctness of the speed/memory optimizations and 2) for other devs to be able to build the v15 output files. Once the v15 is implemented and stabilized I will remove the branching v14/v15.

subwaystation commented 4 years ago

I will review when the deNBI cloud is up again.

josiahseaman commented 4 years ago

This is our new de facto branch. It's great work. @dimatr would you please handle the merge?