[Task]: Improve the example for PythonExternalTransform - Githubissues

apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.

https://beam.apache.org/

Apache License 2.0

7.89k stars 4.27k forks source link

[Task]: Improve the example for PythonExternalTransform #33095

Open liferoad opened 1 week ago

liferoad commented 1 week ago

What needs to happen?

https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/multi_language_inference works with Beam 2.46.0 but

it is broken with Beam 2.60.0 due to the new Python package changes (huggingface hub complains).
Also the way to setup the maven project for this example does not work well. https://github.com/liferoad/beam-multi_language_inference is a working example with the generated pom file.
To support ML, it will be better to show how to use custom container to avoid staging too many Python packages

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

[ ] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam YAML
[X] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Infrastructure
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner