apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.
https://beam.apache.org/
Apache License 2.0
7.8k stars 4.22k forks source link

[Feature Request]: Support int32 for properties of Cross lang transformations written in Java #31673

Open Naireen opened 3 months ago

Naireen commented 3 months ago

What would you like to happen?

All cross lang transformations that are implemented in java that have a python wrapper that use the old ExternalTransform API such as KafkaIO are forced to have int64 properties since the translation layer converts a python int to int64. So if an int32 property is added to the java transform, unless the python wrapper specifies an np.int32 (which is clunky since it needlessly adds an np dependency), the python pipeline will error.

Issue comes from this file: https://github.com/apache/beam/blob/0f2e1963987f1fbb3329016d8c862639ed4fbe43/sdks/python/apache_beam/typehints/schemas.py#L4

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components