search

apache / incubator-heron

Apache Heron (Incubating) is a realtime, distributed, fault-tolerant stream processing engine from Twitter
https://heron.apache.org/
Apache License 2.0
3.65k stars 597 forks source link

Add Spout for new OSS Distributed Log #645

Open lewiskan opened 8 years ago

lewiskan commented 8 years ago

per @kramasamy, help-wanted enhancement. Add Heron Spout for newly OSSed Distributed Log. http://distributedlog.io/html/ https://github.com/twitter/distributedlog

maosongfu commented 8 years ago

Confirmed with Twitter Messaging team. They are happy to contribute the DistributedLog spout.

sijie commented 8 years ago

I saw the contrib dir is removed. what is the principle to manage the spouts?

kramasamy commented 8 years ago

@sijie - we removed it during cleaning purposes. But the idea of contrib will speed up development. We are looking at the following directory structure for contrib

contrib/<component>/<software-version>/src/<language>/....

For example, a spout for distributed log will be 

contrib/spout/dlog-0.1.1/src/java/com/twitter/dlog/<code>

Another example, of a scheduler is

contrib/scheduler/yarn-1.0.1/src/java/com/twitter/scheduler/yarn/...

Hope this helps.

khurrumnasimm commented 7 years ago

Is there anyone active working on this? If no one, I am interested in contributing.

kramasamy commented 7 years ago

@khurrumnasimm - @sijie is in charge of this. @sijie wondering if you are ok with @khurrumnasimm can contribute?

sijie commented 7 years ago

@kramasamy I am okay with that. @khurrumnasimm feel free to work on it.

wangli1426 commented 7 years ago

May I ask what does OSS mean?

On 6 December 2016 at 04:08, Sijie Guo notifications@github.com wrote:

@kramasamy https://github.com/kramasamy I am okay with that. @khurrumnasimm https://github.com/khurrumnasimm feel free to work on it.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/twitter/heron/issues/645#issuecomment-264962401, or mute the thread https://github.com/notifications/unsubscribe-auth/AFBD9I9lrYiT1PvPF3K1b_XDj2mZr03eks5rFG9FgaJpZM4IdUqm .

sijie commented 7 years ago

@wangli1426 it stands for 'open source software'.

sijie commented 7 years ago

@khurrumnasimm feel free to send out the pull request once you have a change ready for review. I would help review it.

huijunwu commented 7 years ago

Another example, of a scheduler is

contrib/scheduler/yarn-1.0.1/src/java/com/twitter/scheduler/yarn/...

I saw yarn in heron/heron/schedulers/src/java/com/twitter/heron/scheduler/yarn/. what scheduler should be in contrib dir? and what scheduler shoud not?