search

apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.
https://beam.apache.org/
Apache License 2.0
7.77k stars 4.21k forks source link

Unknown Coder URN beam:coder:pickled_python:v1 in SnowflakeIO for Python SDK #21632

Open damccorm opened 2 years ago

damccorm commented 2 years ago

SnowflakeIO for Python SDK is not working. Each launch of the pipelines ends with the following error message. RuntimeError:


org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalArgumentException: Unknown Coder URN beam:coder:pickled_python:v1. Known URNs: [beam:coder:avro:generic:v1,
beam:coder:bytes:v1, beam:coder:bool:v1, beam:coder:string_utf8:v1, beam:coder:kv:v1, beam:coder:varint:v1,
beam:coder:interval_window:v1, beam:coder:iterable:v1, beam:coder:timer:v1, beam:coder:length_prefix:v1,
beam:coder:global_window:v1, beam:coder:windowed_value:v1, beam:coder:param_windowed_value:v1, beam:coder:double:v1,
beam:coder:row:v1, beam:coder:sharded_key:v1, beam:coder:custom_window:v1]

 

Imported from Jira BEAM-14075. Original Jira may contain additional context. Reported by: sfc-gh-kbregula.

mmas commented 1 year ago

You'll probably need to set the type or custom apache_beam.coders with .with_output_types(...) in the step right before the SnowflakeIO transformation.