Optimization for Cascalog 1.10.x

nathanmarz / cascalog

Data processing on Hadoop without the hassle.

Other

1.38k stars 178 forks source link

Optimization for Cascalog 1.10.x #227

Closed ipostelnik closed 10 years ago

ipostelnik commented 10 years ago

I know everyone should be moving to Cascalog 2, but we still have a sizable codebase running on Cascalog 1.x, and it will take some time and effort to move.

This pull request has 2 optimizations:

replace non-nil check with a cascading filter, which appears to be a lot faster than equivalent clojure filter op
create a vector of join types rather than (effectively) a linked list. This helps with performance of multi-way joins

sritchie commented 10 years ago

Looks like this has some merge conflicts?

ipostelnik commented 10 years ago

I couldn't select the right base for the pull request. I made my branch of 1.20.2 release tag.

sritchie commented 10 years ago

Hey, I don't think that tags show up in the list, but if you type in "1.10.2" (not 1.20.2) github should give you the option.

ipostelnik commented 10 years ago

I can't pick tag as target of pull request. Seems like I need a real branch.

-Igor

On Tue, Jan 14, 2014 at 4:53 PM, Sam Ritchie notifications@github.comwrote:

Hey, I don't think that tags show up in the list, but if you type in "1.10.2" (not 1.20.2) github should give you the option.

— Reply to this email directly or view it on GitHubhttps://github.com/nathanmarz/cascalog/pull/227#issuecomment-32316720 .