search

Azure / draft-classic

A tool for developers to create cloud-native applications on Kubernetes.
https://draft.sh
MIT License
3.92k stars 395 forks source link

Draft Up port forwarding error can not connect to example-spring-boot #804

Closed quantum-fusion closed 6 years ago

quantum-fusion commented 6 years ago

This is like #777 , however this is with a real world test case project.

Note, I do not use draft create for this project, because the chart files, and Dockerfile have been manually created.

The problem is that when running draft up, the process of building the Docker image with draft fails.

However, The docker image is successfully built via command line using docker.

The results are showing a random number that is generated and a path being generated that isn't where the JAR file is located in the target build.

The target build is located $HOME/springboot-swagger-example-master-cassandra/target/spring-boot-web-0.0.1-SNAPSHOT.jar

NOT where draft up thinks it is being located with the MAGICNUMBER.

/var/lib/docker/tmp/docker-builderMAGICNUMBER/spring-boot-web-0.0.1-SNAPSHOT.jar

see trace results.

draft up Draft Up Started: 'spring-boot-web-0.0.1': 01CEY4GD4VHMD3MGS114QT5QX8 spring-boot-web-0.0.1: Building Docker Image: FAIL ❌ (6.0006s) Inspect the logs with draft logs 01CEY4GD4VHMD3MGS114QT5QX8 MacBook-Pro-2:springboot_swagger_example-master-cassandra $ draft logs 01CEY4GD4VHMD3MGS114QT5QX8 Step 1/6 : FROM java:8 ---> d23bdf5b1b1b Step 2/6 : MAINTAINER info@technologyventureslimited.com ---> Using cache ---> d23490ec5233 Step 3/6 : EXPOSE 8080 9042 ---> Using cache ---> eba04ddf9385 Step 4/6 : ADD ./spring-boot-web-0.0.1-SNAPSHOT.jar . 2018/06/01 12:57:35 error while building: ADD failed: stat /var/lib/docker/tmp/docker-builder930196639/spring-boot-web-0.0.1-SNAPSHOT.jar: no such file or directory MacBook-Pro-2:springboot_swagger_example-master-cassandra $ ls /var/lib/docker ls: /var/lib/docker: No such file or directory

MacBook-Pro-2:springboot_swagger_example-master-cassandra $ sudo mkdir /var/lib/docker/tmp/docker-builder930196639 MacBook-Pro-2:springboot_swagger_example-master-cassandra $ cp ./target/spring-boot-web-0.0.1-SNAPSHOT.jar /var/lib/docker/tmp/docker-builder930196639 cp: /var/lib/docker/tmp/docker-builder930196639/spring-boot-web-0.0.1-SNAPSHOT.jar: Permission denied MacBook-Pro-2:springboot_swagger_example-master-cassandra hottelet$ sudo cp ./target/spring-boot-web-0.0.1-SNAPSHOT.jar /var/lib/docker/tmp/docker-builder930196639

MacBook-Pro-2:springboot_swagger_example-master-cassandra $ draft up Draft Up Started: 'spring-boot-web-0.0.1': 01CEY4RHRGNZJK4090K9XJ7RKM spring-boot-web-0.0.1: Building Docker Image: FAIL ❌ (5.0011s) Inspect the logs with draft logs 01CEY4RHRGNZJK4090K9XJ7RKM MacBook-Pro-2:springboot_swagger_example-master-cassandra$ draft logs 01CEY4RHRGNZJK4090K9XJ7RKM Step 1/6 : FROM java:8 ---> d23bdf5b1b1b Step 2/6 : MAINTAINER info@technologyventureslimited.com ---> Using cache ---> d23490ec5233 Step 3/6 : EXPOSE 8080 9042 ---> Using cache ---> eba04ddf9385 Step 4/6 : ADD ./spring-boot-web-0.0.1-SNAPSHOT.jar . 2018/06/01 13:02:01 error while building: ADD failed: stat /var/lib/docker/tmp/docker-builder698728306/spring-boot-web-0.0.1-SNAPSHOT.jar: no such file or directory

bacongobbler commented 6 years ago

Yep that was the same error I was seeing earlier today when I was testing out the app. Sorry :(

quantum-fusion commented 6 years ago

@squillace @bacongobbler I know it is the weekend, and I spent some time this morning on a Saturday morning. I am sorry you are having maven repository issues, so I have gone ahead, to find you a simpler use case and workaround, and I been able to replicate the issue with the example-spring-boot test case. Also, in addition, I have added a REST API call to say Hello spring boot, so that we can be more like this example (https://spring.io/guides/gs/spring-boot/), yet with adding draft up, and charts for use with Kubernetes Helm as a use case.

I have attached the example-spring-boot.tar.zip test case with these improvements.

You will see if you root cause the issue, that there are some port mapping connectivity issues internally, which are very hard to root cause, and have nothing to do with spring boot.

Steps to get going and reproduce:

% mvn clean install

% draft up Draft Up Started: 'example-spring-boot': 01CF07Z8JMGEKAEXT0TZNS29N5 example-spring-boot: Building Docker Image: SUCCESS ⚓ (31.0147s) example-spring-boot: Pushing Docker Image: SUCCESS ⚓ (22.2483s) example-spring-boot: Releasing Application: SUCCESS ⚓ (2.3176s) Inspect the logs with draft logs 01CF07Z8JMGEKAEXT0TZNS29N5

example-spring-boot.tar.zip MacBook-Pro-2:example-spring-boot $ draft connect Connect to java:4567 on localhost:50486

java: load Java programming language agent, see java.lang.instrument

java: show splash screen with specified image java: See http://www.oracle.com/technetwork/java/javase/documentation/index.html for more details. ERROR: logging before flag.Parse: E0602 08:38:57.202695 9861 portforward.go:331] an error occurred forwarding 50486 -> 4567: error forwarding port 4567 to pod 9662fb011793c4193ca6e580d42ad0545d4d62464ac8dd459bce674a3240e177, uid : exit status 1: 2018/06/02 12:38:58 socat[10112.139868946745088] E connect(5, AF=2 127.0.0.1:4567, 16): Connection refused ERROR: logging before flag.Parse: E0602 08:38:57.223479 9861 portforward.go:331] an error occurred forwarding 50486 -> 4567: error forwarding port 4567 to pod 9662fb011793c4193ca6e580d42ad0545d4d62464ac8dd459bce674a3240e177, uid : exit status 1: 2018/06/02 12:38:58 socat[10113.140338421610240] E connect(5, AF=2 127.0.0.1:4567, 16): Connection refused ERROR: logging before flag.Parse: E0602 08:38:57.260134 9861 portforward.go:331] an error occurred forwarding 50486 -> 4567: error forwarding port 4567 to pod 9662fb011793c4193ca6e580d42ad0545d4d62464ac8dd459bce674a3240e177, uid : exit status 1: 2018/06/02 12:38:58 socat[10114.139768725653248] E connect(5, AF=2 127.0.0.1:4567, 16): Connection refused

@squillace @bacongobbler Please let me know if you can roll in these updates and improvements, because it makes the test case just like the example-java example. The main differences on connectivity and port, are unclear, and we may need you to pull in your java experts on Monday.

quantum-fusion commented 6 years ago

kubectl get pods NAME READY STATUS RESTARTS AGE cassandra-0 1/1 Running 0 15m cassandra-1 0/1 Pending 0 15m example-spring-boot-java-113667454-mzp62 0/1 CrashLoopBackOff 7 14m

quantum-fusion commented 6 years ago

get deployments --all-namespaces NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE default example-spring-boot-java 1 1 1 0 19m kube-system kube-dns 1 1 1 1 297d kube-system tiller-deploy 1 1 1 1 30d

quantum-fusion commented 6 years ago

kubectl describe pod example-spring-boot-java-113667454-mzp62 Name: example-spring-boot-java-113667454-mzp62 Namespace: default Node: minikube/192.168.64.3 Start Time: Sat, 02 Jun 2018 08:37:14 -0400 Labels: app=example-spring-boot-java draft=example-spring-boot pod-template-hash=113667454 Annotations: buildID=01CF07Z8JMGEKAEXT0TZNS29N5 kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"example-spring-boot-java-113667454","uid":"ad9db161-6661-11e8-86... Status: Running IP: 172.17.0.8 Controllers: ReplicaSet/example-spring-boot-java-113667454 Containers: java: Container ID: docker://c4e4a5b685a872372b7da9de50bf4e3423fa0ce3861c6237c0051a608158a566 Image: docker.io/joethecoder2/example-spring-boot:69c2295bc4de33d6bc28ceca998198c9618040da Image ID: docker-pullable://joethecoder2/example-spring-boot@sha256:8037c70347a69a4365fecfd4b9515b456186994ab2fc93ece8acf41302904b8f Port: 4567/TCP State: Terminated Reason: Error Exit Code: 123 Started: Sat, 02 Jun 2018 08:53:19 -0400 Finished: Sat, 02 Jun 2018 08:53:20 -0400 Last State: Terminated Reason: Error Exit Code: 123 Started: Sat, 02 Jun 2018 08:48:12 -0400 Finished: Sat, 02 Jun 2018 08:48:13 -0400 Ready: False Restart Count: 8 Limits: cpu: 100m memory: 128Mi Requests: cpu: 100m memory: 128Mi Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-wgdn6 (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: default-token-wgdn6: Type: Secret (a volume populated by a Secret) SecretName: default-token-wgdn6 Optional: false QoS Class: Guaranteed Node-Selectors: Tolerations: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message

16m 16m 1 default-scheduler Normal Scheduled Successfully assigned example-spring-boot-java-113667454-mzp62 to minikube 16m 16m 1 kubelet, minikube Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "default-token-wgdn6" 16m 16m 1 kubelet, minikube spec.containers{java} Normal Pulling pulling image "docker.io/joethecoder2/example-spring-boot:69c2295bc4de33d6bc28ceca998198c9618040da" 16m 16m 1 kubelet, minikube spec.containers{java} Normal Pulled Successfully pulled image "docker.io/joethecoder2/example-spring-boot:69c2295bc4de33d6bc28ceca998198c9618040da" 16m 6s 9 kubelet, minikube spec.containers{java} Normal Created Created container 16m 6s 9 kubelet, minikube spec.containers{java} Normal Started Started container 16m 6s 8 kubelet, minikube spec.containers{java} Normal Pulled Container image "docker.io/joethecoder2/example-spring-boot:69c2295bc4de33d6bc28ceca998198c9618040da" already present on machine 16m 4s 73 kubelet, minikube spec.containers{java} Warning BackOff Back-off restarting failed container 16m 4s 73 kubelet, minikube Warning FailedSync Error syncing pod

quantum-fusion commented 6 years ago

@squillace @bacongobbler Get the example-spring-boot.tar.zip

Steps to get going and reproduce:

% mvn clean install

% draft up Draft Up Started: 'example-spring-boot': 01CF07Z8JMGEKAEXT0TZNS29N5 example-spring-boot: Building Docker Image: SUCCESS ⚓ (31.0147s) example-spring-boot: Pushing Docker Image: SUCCESS ⚓ (22.2483s) example-spring-boot: Releasing Application: SUCCESS ⚓ (2.3176s) Inspect the logs with draft logs 01CF07Z8JMGEKAEXT0TZNS29N5

MacBook-Pro-2:example-spring-boot $ draft connect

quantum-fusion commented 6 years ago

@squillace @bacongobbler I have isolated this connection problem down to the way helm manages packages and have found that the basic example using minikube on MacOS may be a bug in Helm.

https://stackoverflow.com/questions/50826457/helm-chart-will-not-expose-ip-address-to-ping-localhostport

bacongobbler commented 6 years ago

Hey @quantum-fusion, I found that with your example spring boot application it would not boot, though I think I can see what happened to cause it.

After pulling down your zipfile, I ran draft up with no modifications to the application. This was the result:

><> draft up
Draft Up Started: 'example-spring-boot': 01CFXKGMWX838P20MJ97H782J2
example-spring-boot: Building Docker Image: SUCCESS ⚓  (1.0003s)
example-spring-boot: Pushing Docker Image: SUCCESS ⚓  (4.2386s)
example-spring-boot: Releasing Application: SUCCESS ⚓  (2.1456s)
Inspect the logs with `draft logs 01CFXKGMWX838P20MJ97H782J2`

So it looks like Draft succeeded to deploy your application. In reality, it actually failed to start which is why draft connect won't work; draft connect can only connect to running applications.

If I look at the pod's status:

><> kubectl get pods
NAME                                        READY     STATUS             RESTARTS   AGE
example-spring-boot-java-755c6f9496-hgdnr   0/1       CrashLoopBackOff   3          1m

And then the pod logs:

><> k logs example-spring-boot-java-755c6f9496-hgdnr | head -n 1
Error: -jar requires jar file specification

I'm assuming this is based on the example spring boot application, which is broken out of the box and is being removed in #799.

Would you mind trying with the example-java application instead?

On our end, I think what we can do is get draft logs to display the currently running pod's logs rather than just the build logs. That way you would've had the information required to determine that the spring-boot application didn't actually start up properly.

I'm going to close this ticket as resolved since it's an application issue, and we can follow up in #821. Thanks so much for trying out Draft!

quantum-fusion commented 6 years ago

I did this test on GitHub.com/quantum-fusion/springboot_swagger_example-master-cassandra, after changing the default port to 8080 for the helm charts.

What is noticeable, is that the port mappings internally disallow all traffic.

Also the REST API tests, may reveal a problem I am having with the REST API, but see the responses are blocked.

curl 127.0.0.1:64858/ curl: (52) Empty reply from server MacBook-Pro:~ quantum-fusion$ curl 127.0.0.1:64858/restaurant/ curl: (52) Empty reply from server MacBook-Pro:~ quantum-fusion$ curl 127.0.0.1:64858/restaurant/helloworld curl: (52) Empty reply from server MacBook-Pro:~ quantum-fusion$ curl 127.0.0.1:64858/restaurant/greeting curl: (52) Empty reply from server MacBook-Pro:~ quantum-fusion$ curl 127.0.0.1:64858/restaurant/greetings curl: (52) Empty reply from server

draft up Draft Up Started: 'example-javahello': 01CFXXJVZBRM74H061A2P77CVG example-javahello: Building Docker Image: SUCCESS ⚓ (7.0030s) example-javahello: Pushing Docker Image: SUCCESS ⚓ (1.8318s) example-javahello: Releasing Application: SUCCESS ⚓ (2.4442s) Inspect the logs with draft logs 01CFXXJVZBRM74H061A2P77CVG MacBook-Pro:~/Desktop/Projects/springboot_swaggerexample-master-cassandra quantum-fusion$ draft connect Connect to java:8080 on localhost:64324 java: java: . ____ java: /\ / __' () _ \ \ \ \ java: ( ( )__ | ' | '| | ' \/ _` | \ \ \ \ java: \/ _)| |)| | | | | || (| | ) ) ) ) java: ' |__| .|| ||| |\, | / / / /

java: :: Spring Boot :: (v1.5.9.RELEASE) java: java: 2018-06-14 01:16:54.939 INFO 1 --- [ main] c.s.springboot.SpringBootWebApplication : Starting SpringBootWebApplication v0.0.1-SNAPSHOT on example-javahello-java-5bfb9bccc-8ckl5 with PID 1 (/spring-boot-web-0.0.1-SNAPSHOT.jar started by root in /) java: 2018-06-14 01:16:55.046 INFO 1 --- [ main] c.s.springboot.SpringBootWebApplication : No active profile set, falling back to default profiles: default ERROR: logging before flag.Parse: E0613 21:17:06.318688 62774 portforward.go:331] an error occurred forwarding 64324 -> 8080: error forwarding port 8080 to pod 8b68c26c77ddf683b8e390d7aa0454fd47d002c28c091352bf294bdd9ec9cf07, uid : exit status 1: 2018/06/14 01:17:06 socat[27792] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused ERROR: logging before flag.Parse: E0613 21:17:06.342450 62774 portforward.go:331] an error occurred forwarding 64324 -> 8080: error forwarding port 8080 to pod 8b68c26c77ddf683b8e390d7aa0454fd47d002c28c091352bf294bdd9ec9cf07, uid : exit status 1: 2018/06/14 01:17:06 socat[27793] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused ERROR: logging before flag.Parse: E0613 21:17:06.370990 62774 portforward.go:331] an error occurred forwarding 64324 -> 8080: error forwarding port 8080 to pod 8b68c26c77ddf683b8e390d7aa0454fd47d002c28c091352bf294bdd9ec9cf07, uid : exit status 1: 2018/06/14 01:17:06 socat[27794] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused

draft connect Connect to java:8080 on localhost:64858 ERROR: logging before flag.Parse: E0613 21:20:45.753693 62903 portforward.go:331] an error occurred forwarding 64858 -> 8080: error forwarding port 8080 to pod 6d4efe80a11284849e071a131f6028329d062d40e4a13a9ca5970f696da3953e, uid : exit status 1: 2018/06/14 01:20:45 socat[29459] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused ERROR: logging before flag.Parse: E0613 21:20:53.368085 62903 portforward.go:331] an error occurred forwarding 64858 -> 8080: error forwarding port 8080 to pod 6d4efe80a11284849e071a131f6028329d062d40e4a13a9ca5970f696da3953e, uid : exit status 1: 2018/06/14 01:20:53 socat[29493] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused ERROR: logging before flag.Parse: E0613 21:21:00.016045 62903 portforward.go:331] an error occurred forwarding 64858 -> 8080: error forwarding port 8080 to pod 6d4efe80a11284849e071a131f6028329d062d40e4a13a9ca5970f696da3953e, uid : exit status 1: 2018/06/14 01:21:00 socat[29549] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused ERROR: logging before flag.Parse: E0613 21:21:04.782427 62903 portforward.go:331] an error occurred forwarding 64858 -> 8080: error forwarding port 8080 to pod 6d4efe80a11284849e071a131f6028329d062d40e4a13a9ca5970f696da3953e, uid : exit status 1: 2018/06/14 01:21:04 socat[29587] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused ERROR: logging before flag.Parse: E0613 21:21:06.102098 62903 portforward.go:331] an error occurred forwarding 64858 -> 8080: error forwarding port 8080 to pod 6d4efe80a11284849e071a131f6028329d062d40e4a13a9ca5970f696da3953e, uid : exit status 1: 2018/06/14 01:21:06 socat[29588] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused

quantum-fusion commented 6 years ago

@bacongobbler The problem was the binding in the Dockerfile to the location for the .jar file.

So I did a mvn clean install, and then copied the .jar file to the current directory with the Dockerfile.

FROM java:8 MAINTAINER joe@email.com EXPOSE 4567 ADD ./example-spring-boot-0.0.1-SNAPSHOT.jar . ENTRYPOINT ["java", "-jar", "example-spring-boot-0.0.1-SNAPSHOT.jar"] CMD java -jar example-spring-boot-0.0.1-SNAPSHOT.jar $@

See results that follow, as I thought, the connection errors internally with port routing are refused.

draft connect Connect to java:4567 on localhost:51707 java: java: . _ _ java: /\ / '_ () \ \ \ \ java: ( ( )\ | ' | '| | ' \/ ` | \ \ \ \ java: \/ _)| |)| | | | | || (| | ) ) ) ) java: ' |__| .|| ||| |\, | / / / /

java: :: Spring Boot :: (v1.5.9.RELEASE) java: java: 2018-06-14 01:42:41.946 INFO 1 --- [ main] com.example.demo.DemoApplication : Starting DemoApplication v0.0.1-SNAPSHOT on example-spring-boothello-java-768bc9bb47-tzpj4 with PID 1 (/example-spring-boot-0.0.1-SNAPSHOT.jar started by root in /) java: 2018-06-14 01:42:42.157 INFO 1 --- [ main] com.example.demo.DemoApplication : No active profile set, falling back to default profiles: default ERROR: logging before flag.Parse: E0613 21:42:43.617929 63480 portforward.go:331] an error occurred forwarding 51707 -> 4567: error forwarding port 4567 to pod 99a3d3e586c54a53e5a621f34794fa38584c7d2375f36d8e4a7725ae225d6b9a, uid : exit status 1: 2018/06/14 01:42:43 socat[6973] E connect(5, AF=2 127.0.0.1:4567, 16): Connection refused ERROR: logging before flag.Parse: E0613 21:42:43.629152 63480 portforward.go:331] an error occurred forwarding 51707 -> 4567: error forwarding port 4567 to pod 99a3d3e586c54a53e5a621f34794fa38584c7d2375f36d8e4a7725ae225d6b9a, uid : exit status 1: 2018/06/14 01:42:43 socat[6974] E connect(5, AF=2 127.0.0.1:4567, 16): Connection refused ERROR: logging before flag.Parse: E0613 21:42:43.641287 63480 portforward.go:331] an error occurred forwarding 51707 -> 4567: error forwarding port 4567 to pod 99a3d3e586c54a53e5a621f34794fa38584c7d2375f36d8e4a7725ae225d6b9a, uid : exit status 1: 2018/06/14 01:42:43 socat[6975] E connect(5, AF=2 127.0.0.1:4567, 16): Connection refused java: 2018-06-14 01:42:44.043 INFO 1 --- [ main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@6d5380c2: startup date [Thu Jun 14 01:42:43 UTC 2018]; root of context hierarchy

quantum-fusion commented 6 years ago

See example-spring-boot with Dockerfile fix, still has issue with binding to ./target/myfile.jar

example-spring-boot.tar.zip

quantum-fusion commented 6 years ago

@bacongobbler As I suspected there are still connection problems with spring-boot due to port mapping errors.

quantum-fusion commented 6 years ago

@bacongobbler @squillace I was able to attach the example-spring-boot project, and you can replicate this in your sandbox, to fix the Dockerfile bindings to the .JAR file in ./target/*.jar , once you are done with that binding fix, the real problem of fixing draft up, should work as shown below.

The problem was that draft up would not connect to the service, and Helm install also would not connect to the service, both bugs, and not support requests.

I was able to get springboot running using Kubernetes, and created a script to connect to the springboot service.

https://github.com/quantum-fusion/springboot_swagger_example-master-cassandra/blob/master/draft.connect

Window1:

kubectl run metoo --image=joethecoder2/spring-boot-web:1.0.0

kubectl get pods | grep metoo

Window2:

kubectl port-forward metoo-5565455fd6-2t9kw 12345:8080

Window3:

curl http://localhost:12345/restaurant/helloworld

quantum-fusion commented 6 years ago

I was able to make a draft.up script that builds and uploads the Docker image. https://github.com/quantum-fusion/springboot_swagger_example-master-cassandra/blob/master/draft.up