Currently zip and comap logic is convoluted. It also relies on join operation that is not well supported by Ray. So we need to use pure map and group map operations to realize this operation.
Actually zip and comap should always stay together, so in the long run we may need a breaking change to merge these two functions. If we can merge we may also get rid of the need of dataframe metadata, which is extremely hard to maintain.
But as the first step, we will clean up the logic and remove joins first.
Currently
zipandcomaplogic is convoluted. It also relies on join operation that is not well supported by Ray. So we need to use pure map and group map operations to realize this operation.Actually
zipandcomapshould always stay together, so in the long run we may need a breaking change to merge these two functions. If we can merge we may also get rid of the need of dataframe metadata, which is extremely hard to maintain.But as the first step, we will clean up the logic and remove joins first.