ymc101
commented
4 months ago hi @leokondrashov do you have an idea what is causing this issue?
Open ymc101 opened 4 months ago
ymc101
commented
4 months ago hi @leokondrashov do you have an idea what is causing this issue?
leokondrashov
commented
4 months ago I have little idea, why it can happen. Please provide description of the failing revision: kubectl describe revision fibonacci-python and a pod, while it exists. Better to describe deployment and pod while it times out for proper understanding of what's hindering the deployment.
ymc101
commented
4 months ago This is the description:
yapm0011@node-0:~/vswarm/benchmarks/fibonacci$ kubectl describe revision fibonacci-python
Name: fibonacci-python-00001
Namespace: default
Labels: serving.knative.dev/configuration=fibonacci-python
serving.knative.dev/configurationGeneration=1
serving.knative.dev/configurationUID=9b15e7f2-4791-4015-a8b7-fbc3a84b0ab5
serving.knative.dev/routingState=active
serving.knative.dev/service=fibonacci-python
serving.knative.dev/serviceUID=8b0ccd47-0233-4d31-a4d6-30607f251f61
Annotations: serving.knative.dev/creator: kubernetes-admin
serving.knative.dev/routes: fibonacci-python
serving.knative.dev/routingStateModified: 2024-03-09T08:33:05Z
API Version: serving.knative.dev/v1
Kind: Revision
Metadata:
Creation Timestamp: 2024-03-09T08:33:05Z
Generation: 1
Owner References:
API Version: serving.knative.dev/v1
Block Owner Deletion: true
Controller: true
Kind: Configuration
Name: fibonacci-python
UID: 9b15e7f2-4791-4015-a8b7-fbc3a84b0ab5
Resource Version: 15516
UID: 6148fb13-695d-44f4-be8f-31c45bdd3350
Spec:
Container Concurrency: 0
Containers:
Args:
--addr=0.0.0.0:50000
--function-endpoint-url=0.0.0.0
--function-endpoint-port=50051
--function-name=fibonacci-python
--value=10
--generator=linear
--lowerBound=1
--upperBound=10
Image: docker.io/vhiveease/relay:latest
Name: user-container-0
Ports:
Container Port: 50000
Name: h2c
Protocol: TCP
Readiness Probe:
Success Threshold: 1
Tcp Socket:
Port: 0
Resources:
Args:
--addr=0.0.0.0
--port=50051
Image: docker.io/vhiveease/fibonacci-python:latest
Name: user-container-1
Resources:
Enable Service Links: false
Timeout Seconds: 300
Status:
Actual Replicas: 0
Conditions:
Last Transition Time: 2024-03-09T08:43:36Z
Message: The target is not receiving traffic.
Reason: NoTraffic
Severity: Info
Status: False
Type: Active
Last Transition Time: 2024-03-09T08:33:06Z
Reason: Deploying
Status: Unknown
Type: ContainerHealthy
Last Transition Time: 2024-03-09T08:43:36Z
Message: Initial scale was never achieved
Reason: ProgressDeadlineExceeded
Status: False
Type: Ready
Last Transition Time: 2024-03-09T08:43:36Z
Message: Initial scale was never achieved
Reason: ProgressDeadlineExceeded
Status: False
Type: ResourcesAvailable
Container Statuses:
Image Digest: index.docker.io/vhiveease/relay@sha256:605a16487dbff86f2e04875df6f338913ea02342aa664f4d14d3d8d71b8c697b
Name: user-container-0
Image Digest: index.docker.io/vhiveease/fibonacci-python@sha256:938172f0d1de67a4cb159b4a0b8606be03bc5b1a2c954dc5f5c31b66327965c9
Name: user-container-1
Observed Generation: 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning InternalError 10m (x2 over 20m) revision-controller failed to update deployment "fibonacci-python-00001-deployment": Operation cannot be fulfilled on deployments.apps "fibonacci-python-00001-deployment": the object has been modified; please apply your changes to the latest version and try againWhat are the steps to reproduce? How exactly are you trying to deploy the Fibonacci benchmark?
ymc101
commented
3 months ago After setting up the vHive nodes, and running the deployer client (source /etc/profile && pushd ./tools/deployer && go build && popd && ./tools/deployer/deployer -funcPath ~/vhive/configs/knative_workloads) and the invoker client (pushd ./tools/invoker && go mod tidy && go build && popd && ./tools/invoker/invoker) once, i did make pull in the Fibonacci directory and then kn service apply -f ./yamls/knative/kn-fibonacci-python.yaml as I recall. Am I missing some steps in between?
leokondrashov
commented
3 months ago @dhschall Hi, can you help over here? The steps look according to the docs. This error doesn't seem to be connected to the absence of the stub container needed for Firecracker in vHive.
dhschall
commented
3 months ago Hi, sorry for the delayed response. This is the latest run from the continuous integration. The function seems ok.
Can you try to follow those steps to see if its working with the normal k8 cluster:
# Deploy the function
kubectl apply -f benchmarks/fibonacci/yamls/knative//kn-fibonacci-python.yaml
# Get the logs from the Fibonacci container
kubectl logs -n default -c user-container-0 -l serving.knative.dev/service=fibonacci-python
# If that works try to invoke with the test client
go build ./test-client.go
./test-client --addr fibonacci-python.default.127.0.0.1.sslip.io:80 --name 'Example text for CI'
According to your message the second step fail. In that case please send the logs.
Thanks.
ymc101
commented
2 months ago @dhschall Hi, I tried the steps listed and got some error. Below are the terminal logs:
yapm0011@node-0:~/vswarm$ kubectl apply -f benchmarks/fibonacci/yamls/knative//kn-fibonacci-python.yaml
Warning: Kubernetes default value is insecure, Knative may default this to secure in a future release: spec.template.spec.containers[0].securityContext.allowPrivilegeEscalation, spec.template.spec.containers[0].securityContext.capabilities, spec.template.spec.containers[0].securityContext.runAsNonRoot, spec.template.spec.containers[0].securityContext.seccompProfile, spec.template.spec.containers[1].securityContext.allowPrivilegeEscalation, spec.template.spec.containers[1].securityContext.capabilities, spec.template.spec.containers[1].securityContext.runAsNonRoot, spec.template.spec.containers[1].securityContext.seccompProfile
service.serving.knative.dev/fibonacci-python created
yapm0011@node-0:~/vswarm$ kubectl logs -n default -c user-container-0 -l serving.knative.dev/service=fibonacci-python
time="2024-04-16T08:03:38Z" level=info msg="Connect to 0.0.0.0:50051"
time="2024-04-16T08:03:48Z" level=info msg="Started relay server at 0.0.0.0:50000\n"
time="2024-04-16T08:03:48Z" level=info msg="Downstream function: fibonacci-python at addr 0.0.0.0:50051\n"
time="2024-04-16T08:03:48Z" level=info msg="Input generator: linear, bound: [1:10]\n"
yapm0011@node-0:~/vswarm$ go build ./test-client.go
go: cannot find main module, but found .git/config in /users/yapm0011/vswarm
to create a module there, run:
go mod initOh sorry my bad. You have build the test client in the corresponding directory
cd tools/test-client/
go build ./test-client.go
./test-client --addr fibonacci-python.default.127.0.0.1.sslip.io:80 --name 'Example text for CI'
Hi, I am trying to test the deployment and invocation of the fibonacci benchmark on a knative vHive cluster by following the [readme].(https://github.com/vhive-serverless/vSwarm/tree/main/benchmarks/fibonacci) When starting the function, it timed out and says there is no ready Revision. Is there a way to solve this issue?
Below are some terminal logs from the master node: