Augmenting coiteration to deal with unordered levels (i.e. hash maps). Implement conjunctive merge logic when input to co-iteration is a hash map. The current implementation assumes input levels are ordered. We can deal with unordered levels (during a conjunctive merge) relatively straightforward.
The API that calls coiteration should and will eventually perform the following logic: If it is sortable, sort the level. Otw if it is not sortable, then it must be has_locate == True property.
If the unordered level is part of a disjunctive merge, then the current thought is that we will convert it to a compressed level that can be sortable. This will be handled in a future PR.
See below for details
Proposed Implementation
In coiteration, if the unordered level is part of a conjunctive merge, we can iterate through the level that is ordered and use locate to check if the index exists in the hash level. If it returns NULL, then we don't append to the output. If it does not return NULL, then we append the output with this index.
Currently, we want logic that implements this depending on if one of the level formats passed to Coiterate return has_locate == True and is not ordered.
adam2392
commented
1 year ago
Summary
Augmenting coiteration to deal with unordered levels (i.e. hash maps). Implement conjunctive merge logic when input to co-iteration is a hash map. The current implementation assumes input levels are ordered. We can deal with unordered levels (during a conjunctive merge) relatively straightforward.
The API that calls coiteration should and will eventually perform the following logic: If it is sortable, sort the level. Otw if it is not sortable, then it must be has_locate == True property.
If the unordered level is part of a disjunctive merge, then the current thought is that we will convert it to a compressed level that can be sortable. This will be handled in a future PR.
See below for details
Proposed Implementation
In coiteration, if the unordered level is part of a conjunctive merge, we can iterate through the level that is ordered and use locate to check if the index exists in the hash level. If it returns NULL, then we don't append to the output. If it does not return NULL, then we append the output with this index.
Currently, we want logic that implements this depending on if one of the level formats passed to Coiterate return has_locate == True and is not ordered.