search

Netflix / PigPen

Map-Reduce for Clojure
Apache License 2.0
565 stars 55 forks source link

Is this project being maintained? #182

Open divs1210 opened 6 years ago

divs1210 commented 6 years ago

Hi! We have been using Pig at work for sometime, and testing etc. is a pain. PigPen looks really great, but it seems to be using an older version of Pig, and there haven't been major updates in a while. Is this still under active development? Thanks!

mbossenbroek commented 6 years ago

At this point we're still supporting it, but it is not under active development. I generally view it as active/stable. This is mostly because the people using it haven't asked for anything new in a very long time. :)

PigPen predates Spark, and was an early option for doing big data in more of a coding, less scripting fashion, at a higher level of abstraction than raw Hadoop or Cascading was providing. Now that Spark is taking over that space, combined with a lot of active investment in their community, we've seen a lot of people moving in that direction. Also considering the rapid decline of Pig (as it should be), and it's kind of a no brainer.

If you're really into Clojure and have an existing Clojure ecosystem to build on, it might be worth considering, but in that case I would recommend using the Cascading backend. If you do go that route, I would be happy to assist transferring the project to another active maintainer.

divs1210 commented 6 years ago

Hi @mbossenbroek ! We already have a bunch of Pig scripts and UDFs that we would like to be able to reuse. The Cascading approach should work, but will have to discuss with my team. PigPen sure feels more natural than Cascalog, btw! Thanks for the nice library!