search

spring-attic / scriptable-transform

Apache License 2.0
0 stars 8 forks source link

Unable to deploy stream containing scriptable-transform-processor #32

Closed sabbyanandan closed 5 years ago

sabbyanandan commented 5 years ago

From @AddaxT:

After creating any stream definition containing scriptable-transform-processor-kafka:2.1.0.RELEASE using python as the language deployment is failing at YAML validation (it seems is failing when generating deployment descriptor for K8s). 

We are using SCDF server 2.0.0.RELEASE, Skipper 2.0.0.RELEASE and GKE 1.11.7-gke.12. Please see the image below. Same script worked fine on SCDF 1.7.3 local deployer.

image

sabbyanandan commented 5 years ago

Hi, @AddaxT: The scriptable-transform processor is limited in capabilities; besides that, you won't be able to unit/IT test any of your business logic.

There are few options for Python. 1) Use any of the available python processors

2) Use the gRPC processor, so you can negotiate with a side-car running native Python workload via gRPC.

3) Alternatively, since you're deploying onto Kubernetes, you could build a python-native application as a Docker image, and register that in SCDF to build a streaming data pipeline with it. We have experimented that in SCDF via spring-cloud/spring-cloud-dataflow#3110. A preliminary version of the docs/samples on this PR commit here. We are actively iterating on it, so please expect changes and be prepared for ongoing updates to the PR.

sabbyanandan commented 5 years ago

Closing this due to no activity.