Open viacheslav-fomin-main opened 3 years ago
The service trait is disabled in Knative enabled environment, and the knative-service trait is enabled instead.
Hi Team, I'm working alongside @viacheslav-fomin-main. I would like to provide some context for this issue to help you reproduce if needed.
Our .proto
file looks something like this (we package it in com.example:example-proto:1.0.0-SNAPSHOT
):
syntax = "proto3";
package com.example;
option java_package = "com.example";
option java_multiple_files = true;
message ExampleRequest {
string request_message = 1;
int32 request_id = 2;
}
message ExampleResponse {
string response_message = 1;
int32 response_id = 2;
}
service ExampleService {
rpc Run(ExampleRequest) returns (ExampleResponse);
}
}
Our SampleGrpcCamelRoute.java
route looks like this:
package com.example.service.grpc;
import com.google.protobuf.ByteString;
import com.example.ExampleRequest;
import com.example.ExampleResponse;
import org.apache.camel.Message;
import org.apache.camel.builder.RouteBuilder;
import java.nio.charset.StandardCharsets;
public class SampleGrpcCamelRoute extends RouteBuilder {
@Override
public void configure() throws Exception {
fromF("grpc://localhost:9000/com.example.ExampleService?synchronous=true")
.process(exchange -> {
final Message message = exchange.getMessage();
final ExampleRequest request = message.getBody(ExampleRequest.class);
final String output = "Hello World!";
final ExampleResponse response = ExampleResponse.newBuilder()
.setMessage(output)
.build();
message.setBody(response);
});
}
}
We have verified this route works locally using java unit tests and Apache Camel ContextTestSupport org.apache.camel.ContextTestSupport
.
kamel
CLIWe run the integration using the following command:
kamel run --dev \
-d mvn:com.example:example-proto:1.0.0-SNAPSHOT \
-d mvn:org.apache.camel:camel-grpc:3.9.0 \
-d mvn:org.apache.camel:camel-componentdsl:3.9.0 \
SampleGrpcCamelRoute.java
Using kubectl port-forward
on the integration pod we are able to verify the grpc route is working using the same tests we had run locally.
We tried a plethora of camel-k trait
s including:
--trait container.service-port=9000 \
--trait container.expose=true \
--trait knative.enabled=true \
--trait knative-service.enabled=true \
--trait knative-service.autoscaling-class=kpa.autoscaling.knative.dev \
--trait knative-service.min-scale=1 \
--trait knative-service.max-scale=1 \
Thank you for the support!
It seems Knative relies on the port name to determine the protocol: https://github.com/knative/serving/pull/2539.
Based on the code sample from https://knative.dev/docs/serving/samples/grpc-ping-go/, it seems the port name must be set to h2c
.
Could you try to set the port name using the container trait, e.g.:
$ kamel run -t container.port-name=h2c
Also, mind that the container port may need be accessed using the Knative service mesh. I'm not sure accessing the port through the Kubernetes proxy actually works.
I've changed the name of this issue in an attempt to reflect the use case more closely. Feel free to amend it.
@astefanutti it still does not work but I can not even see the knative service or just service trait applied in the camel k operator logs. Maybe something additional must be installed for it to work?
I tried both:
-t container.port-name=h2c \
-t container.port=9000 \
-t container.expose=true \
-t knative.enabled=true \
-t knative-service.auto=true \
-t knative-service.autoscaling-class=kpa.autoscaling.knative.dev \
-t knative-service.min-scale=1 \
-t knative-service.max-scale=1 \
and
-t container.service-port-name=h2c \
-t container.service-port=9000 \
-t container.expose=true \
-t knative.enabled=true \
-t knative-service.auto=true \
-t knative-service.autoscaling-class=kpa.autoscaling.knative.dev \
-t knative-service.min-scale=1 \
-t knative-service.max-scale=1 \
with no success.
Could you please add -t knative-service.enabled=true
, and provide the output of kubectl get it sample-grpc-camel-route
?
Command:
kamel run \
-t container.service-port-name=h2c \
-t container.service-port=9000 \
-t knative.enabled=true \
-t knative-service.enabled=true \
-t knative-service.auto=true \
-t knative-service.autoscaling-class=kpa.autoscaling.knative.dev \
-t knative-service.min-scale=1 \
-t knative-service.max-scale=1 \
-d mvn:com.example:example-proto:1.0.0-SNAPSHOT \
-d mvn:org.apache.camel:camel-grpc:3.9.0 \
-d mvn:org.apache.camel:camel-componentdsl:3.9.0 \
SampleGrpcCamelRoute.java
Output
$ kubectl get it sample-grpc-camel-route
NAME PHASE KIT REPLICAS
sample-grpc-camel-route Running kit-c1q88iiga9qjbbv623lg 1
Sorry I meant kubectl get it sample-grpc-camel-route -o yaml
, to output the Integration definition.
apiVersion: camel.apache.org/v1
kind: Integration
metadata:
creationTimestamp: "2021-04-12T17:32:58Z"
generation: 1
name: sample-grpc-camel-route
namespace: camel-k
resourceVersion: "154392144"
selfLink: /apis/camel.apache.org/v1/namespaces/camel-k/integrations/sample-grpc-camel-route
uid: ca791fb8-2902-4af0-b508-ad9eb0a72249
spec:
dependencies:
- mvn:com.example:example-proto:1.0.0-SNAPSHOT
- mvn:org.apache.camel:camel-grpc:3.9.0
- mvn:org.apache.camel:camel-componentdsl:3.9.0
sources:
- content: |-
package com.example.service.grpc;
import com.google.protobuf.ByteString;
import com.example.ExampleRequest;
import com.example.ExampleResponse;
import org.apache.camel.Message;
import org.apache.camel.builder.RouteBuilder;
import java.nio.charset.StandardCharsets;
public class SampleGrpcCamelRoute extends RouteBuilder {
@Override
public void configure() throws Exception {
fromF("grpc://localhost:9000/com.example.ExampleService?synchronous=true")
.process(exchange -> {
final Message message = exchange.getMessage();
final ExampleRequest request = message.getBody(ExampleRequest.class);
final String output = "Hello World!";
final ExampleResponse response = ExampleResponse.newBuilder()
.setMessage(output)
.build();
message.setBody(response);
});
}
}
name: SampleGrpcCamelRoute.java
traits:
container:
configuration:
servicePort: 9000
servicePortName: h2c
knative:
configuration:
enabled: true
knative-service:
configuration:
auto: true
class: kpa.autoscaling.knative.dev
enabled: true
maxScale: 1
minScale: 1
status:
conditions:
- firstTruthyTime: "2021-04-12T17:32:58Z"
lastTransitionTime: "2021-04-12T17:32:58Z"
lastUpdateTime: "2021-04-12T17:32:58Z"
message: camel-k
reason: IntegrationPlatformAvailable
status: "True"
type: IntegrationPlatformAvailable
- firstTruthyTime: "2021-04-12T17:33:12Z"
lastTransitionTime: "2021-04-12T17:33:12Z"
lastUpdateTime: "2021-04-12T17:33:12Z"
message: kit-c1q88iiga9qjbbv623lg
reason: IntegrationKitAvailable
status: "True"
type: IntegrationKitAvailable
- lastTransitionTime: "2021-04-12T17:33:12Z"
lastUpdateTime: "2021-04-12T17:33:12Z"
message: different controller strategy used (deployment)
reason: CronJobNotAvailableReason
status: "False"
type: CronJobAvailable
- firstTruthyTime: "2021-04-12T17:33:12Z"
lastTransitionTime: "2021-04-12T17:33:12Z"
lastUpdateTime: "2021-04-12T17:33:12Z"
message: deployment name is sample-grpc-camel-route
reason: DeploymentAvailable
status: "True"
type: DeploymentAvailable
- firstTruthyTime: "2021-04-12T17:33:15Z"
lastTransitionTime: "2021-04-12T17:33:15Z"
lastUpdateTime: "2021-04-12T17:33:15Z"
reason: ReplicaSetReady
status: "True"
type: Ready
dependencies:
- mvn:com.example:example-proto:1.0.0-SNAPSHOT
- mvn:org.apache.camel.k/camel-k-loader-java
- mvn:org.apache.camel.k/camel-k-runtime
- mvn:org.apache.camel:camel-componentdsl:3.9.0
- mvn:org.apache.camel:camel-grpc:3.9.0
digest: voLtPQb2I5N8-AC5xlk4fAJWTa8uBsZd-GEsQ16Enk7s
image: quay.io/verygoodsecurity/camel-k-kit-c1q88iiga9qjbbv623lg@sha256:473ee4ebb3aefe32ee22fd771a645f96471baf592e5510a7977e9e27841d8e2f
kit: kit-c1q88iiga9qjbbv623lg
lastInitTimestamp: "2021-04-12T17:32:58Z"
phase: Running
platform: camel-k
profile: knative
replicas: 1
runtimeProvider: quarkus
runtimeVersion: 1.6.0
selector: camel.apache.org/integration=sample-grpc-camel-route
version: 1.3.1
The deployment strategy for the Integration uses a standard Deployment, instead of a KnativeService resource. I assumed that explicitly enabling the knative-service trait would force it, but it does not.
Could you please retry by adding the -t deployer.kind=knative-service
option to the kamel run
command? It should create a KnativeService resource that you can output with kubectl get ksvc sample-grpc-camel-route -o yaml
.
It doesn't produce a KnativeService when I add -t deployer.kind=knative-service
OR -t deployer.enabled=true -t deployer.kind=knative-service
.
Instead I get the following:
$ kubectl get ksvc sample-grpc-camel-route -o yaml
Error from server (NotFound): services.serving.knative.dev "sample-grpc-camel-route" not found
This is the generated integration:
$ kubectl get it sample-grpc-camel-route -o yaml
apiVersion: camel.apache.org/v1
kind: Integration
metadata:
creationTimestamp: "2021-04-12T18:20:19Z"
generation: 1
name: sample-grpc-camel-route
namespace: camel-k
resourceVersion: "154519472"
selfLink: /apis/camel.apache.org/v1/namespaces/camel-k/integrations/sample-grpc-camel-route
uid: 5ab7ffd6-438c-46a2-8deb-aede4703dd12
spec:
dependencies:
- mvn:com.example:example-proto:1.0.0-SNAPSHOT
- mvn:org.apache.camel:camel-grpc:3.9.0
- mvn:org.apache.camel:camel-componentdsl:3.9.0
sources:
- content: |-
package com.example.service.grpc;
import com.google.protobuf.ByteString;
import com.example.ExampleRequest;
import com.example.ExampleResponse;
import org.apache.camel.Message;
import org.apache.camel.builder.RouteBuilder;
import java.nio.charset.StandardCharsets;
public class SampleGrpcCamelRoute extends RouteBuilder {
@Override
public void configure() throws Exception {
fromF("grpc://localhost:9000/com.example.ExampleService?synchronous=true")
.process(exchange -> {
final Message message = exchange.getMessage();
final ExampleRequest request = message.getBody(ExampleRequest.class);
final String output = "Hello World!";
final ExampleResponse response = ExampleResponse.newBuilder()
.setMessage(output)
.build();
message.setBody(response);
});
}
}
name: SampleGrpcCamelRoute.java
traits:
container:
configuration:
servicePort: 9000
servicePortName: h2c
deployer:
configuration:
enabled: true
kind: knative-service
knative:
configuration:
enabled: true
knative-service:
configuration:
auto: true
class: kpa.autoscaling.knative.dev
enabled: true
maxScale: 1
minScale: 1
status:
conditions:
- firstTruthyTime: "2021-04-12T18:20:19Z"
lastTransitionTime: "2021-04-12T18:20:19Z"
lastUpdateTime: "2021-04-12T18:20:19Z"
message: camel-k
reason: IntegrationPlatformAvailable
status: "True"
type: IntegrationPlatformAvailable
- lastTransitionTime: "2021-04-12T18:20:19Z"
lastUpdateTime: "2021-04-12T18:20:19Z"
message: 'controller strategy: knative-service'
reason: DeploymentAvailable
status: "False"
type: DeploymentAvailable
- firstTruthyTime: "2021-04-12T18:20:19Z"
lastTransitionTime: "2021-04-12T18:20:19Z"
lastUpdateTime: "2021-04-12T18:20:19Z"
message: kit-c1q88iiga9qjbbv623lg
reason: IntegrationKitAvailable
status: "True"
type: IntegrationKitAvailable
- lastTransitionTime: "2021-04-12T18:20:19Z"
lastUpdateTime: "2021-04-12T18:20:19Z"
message: different controller strategy used (knative-service)
reason: CronJobNotAvailableReason
status: "False"
type: CronJobAvailable
- lastTransitionTime: "2021-04-12T18:20:19Z"
lastUpdateTime: "2021-04-12T18:20:19Z"
status: Unknown
type: Ready
dependencies:
- mvn:com.example:example-proto:1.0.0-SNAPSHOT
- mvn:org.apache.camel.k/camel-k-loader-java
- mvn:org.apache.camel.k/camel-k-runtime
- mvn:org.apache.camel:camel-componentdsl:3.9.0
- mvn:org.apache.camel:camel-grpc:3.9.0
digest: vYn6n2SDK9WEMqnBgXWq04-W7A2t6FK6qpo34ra0ko8Y
image: quay.io/verygoodsecurity/camel-k-kit-c1q88iiga9qjbbv623lg@sha256:473ee4ebb3aefe32ee22fd771a645f96471baf592e5510a7977e9e27841d8e2f
kit: kit-c1q88iiga9qjbbv623lg
lastInitTimestamp: "2021-04-12T18:20:19Z"
phase: Running
platform: camel-k
profile: knative
runtimeProvider: quarkus
runtimeVersion: 1.6.0
selector: camel.apache.org/integration=sample-grpc-camel-route
version: 1.3.1
And when I run using --dev
, it gets stuck on:
Condition "Ready" is "Unknown" for Integration sample-grpc-camel-route
I've tried locally, and adding the -t knative-service.enabled=true
managed to force the KnativeService creation on my side. So there is something else...
One issue your example raises is that the fromF
DSL is not supported for HTTP service detection. I've created #2210. In the meantime, could you please retry by changing fromF
with from
in your route, and provide the output of kubectl get ksvc
?
I've also created #2209, to keep track of the general discussion of gRPC support with Knative in Camel K.
Still no output with kubectl get ksvc
when changing from fromF
to from
. The integration is also still hanging on Condition "Ready" is "Unknown" for Integration sample-grpc-camel-route
when I run it using --dev
.
Might this be an issue with knative not parsing the integration as intended?
I cannot find the reason why Camel K does not create the KnativeService resource for the Integration, while it is when I try locally. Mind that I've tested on master
branch, but I don't see a reason why that would not work with v1.3.1.
I could suggest to try first removing Knative from the equation, with kamel run --profile kubernetes
or kamel run --profile openshift
. That'll help testing gRPC with Camel Quarkus running in a standard Deployment. Then get back to Knative once the former works as expected.
Hi @astefanutti, following your request to go from fromF
to from
, I also get the following error - it appeared when I removed the trait deployer.kind
. However it still runs successfully when running my unit tests locally.
[1] 2021-04-13 15:34:51,955 INFO [org.apa.cam.k.Runtime] (main) Apache Camel K Runtime 1.6.0
[1] 2021-04-13 15:34:51,961 INFO [org.apa.cam.qua.cor.CamelBootstrapRecorder] (main) bootstrap runtime: org.apache.camel.quarkus.main.CamelMainRuntime
[1] 2021-04-13 15:34:52,019 INFO [org.apa.cam.k.lis.SourcesConfigurer] (main) Loading routes from: SourceDefinition{name='SampleGrpcCamelRoute', language='java', location='file:/etc/camel/sources/i-source-000/SampleGrpcCamelRoute.java', }
[1] 2021-04-13 15:34:53,070 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Apache Camel 3.7.0 (camel-1) is starting
[1] 2021-04-13 15:34:53,072 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) StreamCaching is not in use. If using streams then it's recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
[1] 2021-04-13 15:34:53,080 ERROR [org.apa.cam.imp.eng.AbstractCamelContext] (main) Error starting CamelContext (camel-1) due to exception thrown: Unable to find generated class for service ExampleService: java.lang.IllegalStateException: Unable to find generated class for service ExampleService
[1] at org.apache.camel.quarkus.grpc.runtime.QuarkusBindableServiceFactory.lambda$createBindableService$1(QuarkusBindableServiceFactory.java:58)
[1] at java.base/java.util.Optional.orElseThrow(Optional.java:408)
[1] at org.apache.camel.quarkus.grpc.runtime.QuarkusBindableServiceFactory.createBindableService(QuarkusBindableServiceFactory.java:57)
[1] at org.apache.camel.quarkus.grpc.runtime.CamelGrpcRecorder$QuarkusGrpcConsumer.doStart(CamelGrpcRecorder.java:82)
[1] at org.apache.camel.support.service.BaseService.start(BaseService.java:115)
[1] at org.apache.camel.impl.engine.AbstractCamelContext.startService(AbstractCamelContext.java:3108)
[1] at org.apache.camel.impl.engine.InternalRouteStartupManager.doStartOrResumeRouteConsumers(InternalRouteStartupManager.java:362)
[1] at org.apache.camel.impl.engine.InternalRouteStartupManager.doStartRouteConsumers(InternalRouteStartupManager.java:282)
[1] at org.apache.camel.impl.engine.InternalRouteStartupManager.safelyStartRouteServices(InternalRouteStartupManager.java:181)
[1] at org.apache.camel.impl.engine.InternalRouteStartupManager.doStartOrResumeRoutes(InternalRouteStartupManager.java:115)
[1] at org.apache.camel.impl.engine.AbstractCamelContext.doStartCamel(AbstractCamelContext.java:2889)
[1] at org.apache.camel.impl.engine.AbstractCamelContext.doStartContext(AbstractCamelContext.java:2702)
[1] at org.apache.camel.impl.engine.AbstractCamelContext.doStart(AbstractCamelContext.java:2665)
[1] at org.apache.camel.support.service.BaseService.start(BaseService.java:115)
[1] at org.apache.camel.impl.engine.AbstractCamelContext.start(AbstractCamelContext.java:2431)
[1] at org.apache.camel.quarkus.main.CamelMain.doStart(CamelMain.java:75)
[1] at org.apache.camel.support.service.BaseService.start(BaseService.java:115)
[1] at org.apache.camel.quarkus.main.CamelMain.startEngine(CamelMain.java:120)
[1] at org.apache.camel.quarkus.main.CamelMainRuntime.start(CamelMainRuntime.java:49)
[1] at org.apache.camel.quarkus.core.CamelBootstrapRecorder.start(CamelBootstrapRecorder.java:45)
[1] at io.quarkus.deployment.steps.CamelBootstrapProcessor$boot-173480958.deploy_0(CamelBootstrapProcessor$boot-173480958.zig:101)
[1] at io.quarkus.deployment.steps.CamelBootstrapProcessor$boot-173480958.deploy(CamelBootstrapProcessor$boot-173480958.zig:40)
[1] at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:675)
[1] at io.quarkus.runtime.Application.start(Application.java:90)
[1] at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:97)
[1] at io.quarkus.runtime.Quarkus.run(Quarkus.java:62)
[1] at io.quarkus.runtime.Quarkus.run(Quarkus.java:38)
[1] at io.quarkus.runtime.Quarkus.run(Quarkus.java:104)
[1] at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:29)
What configuration did you use for the printed logs? is it with the Knative profile disabled?
No it's enabled, same configs as before:
kamel run --dev \
-t container.service-port-name=h2c \
-t container.service-port=9000 \
-t knative.enabled=true \
-t knative-service.enabled=true \
-t knative-service.auto=true \
-t knative-service.autoscaling-class=kpa.autoscaling.knative.dev \
-t knative-service.min-scale=1 \
-t knative-service.max-scale=1 \
-d mvn:com.example:example-proto:1.0.0-SNAPSHOT \
-d mvn:org.apache.camel:camel-grpc:3.9.0 \
-d mvn:org.apache.camel:camel-componentdsl:3.9.0 \
SampleGrpcCamelRoute.java
Hum sorry I'm lost :) it seems we are progressing. The stack trace seems to indicate that the Integration deploys, but the gRPC component is not able to find the stub class. Could you provide the output of kubectl get ksvc
?
I believe this is a step backwards. The scope of the new issue is before trying to make it a service.
When using fromF()
, my route integration was deploying and working fine and I could test it using port-forwarding. As per your request, when I switch to using from()
my route integration no longer works (the pod keeps crashing) and I get the above errors in the logs about not finding the class stub.
When using fromF(), my route integration was deploying and working fine and I could test it using port-forwarding
Now I'm confused even more 🙃. I understood that this was not working initially.
I think it'd be easier that we put together an example, demonstrating the use of gRPC endpoint with Knative, so we can troubleshoot on a share basis, that will ultimately become an official example.
It could be created into https://github.com/apache/camel-k/tree/master/examples. That'd be awesome that you submit a PR there and we'll help to make it working?
@astefanutti this is a rough draft of what I'm trying to get to work as a service (using knative is a plus): https://github.com/apache/camel-k/pull/2220. If we are able to get it to deploy as a service using camel-k without knative then that's good enough.
Hi @astefanutti . Thank you for all your help so far.
As @moehajj said, we are trying to work without knative and even using --profile=kubernetes
is not working. After some analysis of camel-k code, we found this:
if !meta.ExposesHTTPServices {
e.Integration.Status.SetCondition(
v1.IntegrationConditionServiceAvailable,
corev1.ConditionFalse,
v1.IntegrationConditionServiceNotAvailableReason,
"no http service required",
)
return false, nil
}
Having a check that a service must be HTTP is pretty restrictive given that a Camel integration could support any protocol. I understand why this restriction would exist in Knative, but I don't see the value in this being a restriction in Camel-K.
There should be a way to completely bypass this check or force creation of a kubernetes service even if this check fails. Should I create a new issue for this?
EDIT:
@viacheslav-fomin-main figured it out.
adding -t service.auto=false
is what was needed to force creation.
Yes as @fshaikh-vgs shared, the following traits spin up a nodePort service.
--profile kubernetes \
-t service.enabled=true \
-t service.auto=false \
-t container.service-port=9000 \
-t container.port=9000 \
The service has the following specs:
spec:
clusterIP: 10.102.112.181
externalTrafficPolicy: Cluster
ports:
- name: http
nodePort: 30872
port: 9000
protocol: TCP
targetPort: http
selector:
camel.apache.org/integration: sample-grpc-camel-route
sessionAffinity: None
type: NodePort
However I believe I cannot access my grpc service (which is running inside the pod at port 9000) because targetPort
is set to http
instead of 9000.
When I deploy a service manually instead of using camel-k, I am able to access the grpc service fine. I do this using the following:
cat <<EOF | tee sample-grpc-service.yaml
apiVersion: v1
kind: Service
metadata:
name: sample-grpc
labels:
camel.apache.org/integration: sample-grpc-camel-route
spec:
ports:
- port: 9000
protocol: TCP
selector:
camel.apache.org/integration: sample-grpc-camel-route
EOF
kubectl apply -f sample-grpc-service.yaml
Is it possible to set the nodePort in the service generated by camel-k? I cannot find any trait for this.
As far as I can remember, the rationale was that we can auto configure stuffs for the http protocol 'easily' but for other protocols it may be more complex so we decided to leave such task for the user (but I agree that we need some additional documentation on the subject).
Notes:
This issue has been automatically marked as stale due to 90 days of inactivity. It will be closed if no further activity occurs within 15 days. If you think that’s incorrect or the issue should never stale, please simply write any comment. Thanks for your contributions!
Hi!
I am trying to run a gRPC service using camel and expose a port so I can use it in other camel applications. Everything runs in camel k with Knative. But for some reason I can not create a service. Basically I tried all the possible combinations from traits documentation but nothing seems to work. Can you please help me here? Maybe I am missing something.
Here is the operator log which does not seem to have the service trait logged in it or an errors.
Thank you!