Closed seabbs closed 1 year ago
This does seem to work, but only seems to be a ~10% improvement in that step.
This does seem to work, but only seems to be a ~10% improvement in that step.
For sure write_bin is the real cost of that step. It looks like it can be vectorised if given the same input so perhaps that should be a focus (sort input and use as vectorised - then combine)?
Have checked, lapply + reassemble is ~100x slower than vectorized version.
The problem is that the ellipsis argument approach doesn't give an easily vectorizable collection as an argument. Can give that functionality up and get back substantial performance.
What about if you take everything in using list(...)
and then sort into type, use write_bin on each vectorised and then concatenate? the precise order of the args shouldn't matter? You might need a failsafe we don't know the type section that uses the current approach for safety as well.
That would assume that most common args are sortable etc. but I think they probably are? Or we could restrict hash_salt to just a few argument types that we have tested?
As discussed in #2 this PR switches out
Reduce for
unlist`. I am not clear if this makes sense but everything appears to work on the face of it.