search

spring-cloud / spring-cloud-dataflow

A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes
https://dataflow.spring.io
Apache License 2.0
1.11k stars 583 forks source link

Docker Hub Images springcloud/spring-cloud-dataflow-server 2.8.x are broken #4621

Closed klopfdreh closed 3 years ago

klopfdreh commented 3 years ago

Hi,

I just wanted to update to the springcloud/spring-cloud-dataflow-server 2.8.1, but saw that the images on docker hub are broken.

See Layers of working 2.7.2 (all are filled):

https://hub.docker.com/layers/springcloud/spring-cloud-dataflow-server/2.7.2/images/sha256-cba62d74826f348ea153937448b5c2cabff5d45694f8ca6fc8fdfb8823d20d6b?context=explore

See Layers of broken 2.8.1 (all are empty): https://hub.docker.com/layers/springcloud/spring-cloud-dataflow-server/2.8.1/images/sha256-c534037a42fd162aa4b9308be8cf660ffc3b872b8983d91de1397dd5b30c4b53?context=explore

Thanks in advance.

sabbyanandan commented 3 years ago

Hi, @klopfdreh. Starting from 2.8.x, we switched to pack and the cloud-native Java buildpack and assembly mechanism. Previously, we used to generate the images on our own using Dockerfiles. This new method and the generated container image are used in OSS and our commercial offering, and we don't see any issues with them.

Curious as to why it is not bootstrapping on your end as expected. Perhaps you can share a bit more detail on how you're installing SCDF and any specific steps involved in the provisioning, which we can review.

All that said, @jvalkeal will review and get back to you on the layer discrepancy.

klopfdreh commented 3 years ago

We are using a multi stage docker build which uses a hardend base image and copy out the scdf jar from the official docker image with COPY —from=image:tag.

I created this issue because there was no hint on docker hub. Maybe it is worth to update and tell users what you explained in this issue.

Maybe we can proceed with a wget from out of our docker file and load the server jar from maven central repository.

jvalkeal commented 3 years ago

It looks like if you need to build your own image then best would be indeed to take it from central. Remember that we did same dance for skipper and composed-task-runner images.

Just out of interest, what exactly you need if our new paketo based images would not suit your needs as is?

klopfdreh commented 3 years ago

Maybe it is my fault - can you provide a link to the new image? Maybe it is just a change at the image name.

jvalkeal commented 3 years ago

Well 2.8.1 you linked is paketo based. It's build from a fatjar like this https://github.com/spring-cloud/spring-cloud-dataflow/blob/99ad15ac3e84985de1ffefd53a91343bbc141084/.github/workflows/ci.yml#L66 i.e. for snaphots. It surely helped us to manage images and we can now build outside of a maven build and it's easier to support different jvm's, etc.

If you need to harden something what is it? If you just need to build with your own base images, that's fine but then you're on your own. We do quite a bit of testing and are not on latest jdk as there was a ssl issue per #4495. These are kinda things what lands on those who build their own images.

klopfdreh commented 3 years ago

Correct me if I am wrong but from what I can see in your ci.yaml you push the image to the tag 2.8.1-jdk11 not 2.8.1 anymore:

https://hub.docker.com/layers/springcloud/spring-cloud-dataflow-server/2.8.1-jdk11/images/sha256-c534037a42fd162aa4b9308be8cf660ffc3b872b8983d91de1397dd5b30c4b53?context=explore

See here:

https://github.com/spring-cloud/spring-cloud-dataflow/blob/99ad15ac3e84985de1ffefd53a91343bbc141084/.github/workflows/ci.yml#L103

but even if I have a look at this image at docker hub the layers are empty. Maybe something is wrong with the new way of the ci integration or don’t you provide the image to docker hub anymore?

jvalkeal commented 3 years ago

I think you're confusing what that dockerhub layer info means. I don't know paketo internals but those just shows layer initialise commands and I think there's none with paketo and it shows layers have a size.

This is what it looks from dive image

Anyway you would not find that executable jar from there as buildpacks will explode things and runs things from tree.

klopfdreh commented 3 years ago

Now I see - thanks! I guess we are going to get the fat jar from maven rep in this case. May you can provide a link?

klopfdreh commented 3 years ago

I guess I found it: https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-dataflow-server

I think this issue can be closed, now.

Thanks a lot for all the help and the hints!

klopfdreh commented 3 years ago

Hi,

just wanted to let you know that the download of the fat jar and building our own docker file worked:

  1. Create a docker file based on a valid jdk image
  2. Download the jar with wget -Y on https://repo1.maven.org/maven2/org/springframework/cloud/spring-cloud-dataflow-server/2.8.1/spring-cloud-dataflow-server-2.8.1.jar
  3. Add application.yaml / bootstrap.yaml into a relative path ./config/ to the server jar (in Kubernetes mounted with a config map as files)
  4. Check the SHA1 sum with echo "ad271b01c4e3d72f2e7a402f3d29c314347c2da5 ./spring-cloud-dataflow-server-2.8.1" | sha1sum -c' and fail if not valid
  5. Set a valid entry point in the docker file to start the server