search

apache / incubator-heron

Apache Heron (Incubating) is a realtime, distributed, fault-tolerant stream processing engine from Twitter
https://heron.apache.org/
Apache License 2.0
3.65k stars 597 forks source link

Upgrade pex in Heron #881

Open billonahill opened 8 years ago

billonahill commented 8 years ago

Ideally we would fetch a pex binary at build time, but for now the code is committed and aging. This ticket is to upgrade pex to the latest release. In my tests this has worked fine, but we need to get this change merged into pex first: https://github.com/pantsbuild/pex/pull/277

huijunwu commented 4 years ago

Curious the motivation to choose PEX. why not use the Bazel py_binary, which could make the building process easier?

joshfischer1108 commented 4 years ago

Following up from the conversation on Slack to keep others aware. I believe that at the time the core development in Heron was happening pi_binary rules did not exist. It would probably be easier to migrate to the pi_binary rules moving forward.

nicknezis commented 4 years ago

It seems there are two locations that are relevant to the PEX discussion. Are there any I'm missing? What changes would be needed to use the output of py_binary build task? Would this be easier or harder for analytics developers? I'm currently looking to use {Py}Gradle and the LinkedIn PEX plugin to package my analytics. It would not be ideal if we forced analytic developers to only use Bazel.

  1. Heron Executor: Runs a pyton binary with the pex file as an argument (per component) https://github.com/apache/incubator-heron/blob/e1dda9acfded22ac1de86d73981118488d7a3b19/heron/executor/src/python/heron_executor.py#L724
  2. Heron CLI checks for pex extension: https://github.com/apache/incubator-heron/blob/e1dda9acfded22ac1de86d73981118488d7a3b19/heron/tools/cli/src/python/submit.py#L439
  3. Heron CLI: Uses pex_loader library to load the Topology class at submit time. https://github.com/apache/incubator-heron/blob/e1dda9acfded22ac1de86d73981118488d7a3b19/heron/tools/cli/src/python/execute.py#L124
joshfischer1108 commented 4 years ago

Could you describe what an analytic developer is?

nicknezis commented 4 years ago

A developer writing a Heron topology to run in a managed instance of Heron.