[bitnami/spring-cloud-dataflow] SCDF upgrade to 2.11.1 corrupts skipper tables in DB and streams not accessible via UI

[dhpizza]

Name and Version

bitnami/spring-cloud-dataflow

What architecture are you using?

amd64

What steps will reproduce the bug?

1. Have spring-cloud-dataflow bitnami Chart deployed with images:

appVersion: 2.9.3
dependencies:
- name: spring-cloud-dataflow
  version: 16.0.2

2. Upgrade to

   appVersion: 2.11.1
   dependencies:
   - name: spring-cloud-dataflow
   version: 25.0.0

3. Access spring-cloud-dataflow UI

4. Go to streams and ERROR pop- up will show up logref:SkipperException message:No release manager available for 'null'

What is the expected behavior?

The list of previously registered and deployed streams should be visible. The upgrade should not change the existing data or make it un-accessible.

What do you see instead?

Additional information

I found that after the upgrade the skipper_release table in the in the skipper postgreSQL DB has values of pkg_json_string changed from 7-digit integers like 7076817 to { "metadata":{"apiVersion":"skipper.spring.io/v1", ...

[javsalgar]

Hi,

It seems that you are performing the upgreade from 16.x.x -> 25.x.x. This means performing the upgrade between 9 major releases. In this sense, it is expected that breaks occur. My advice would be to perform upgrades from 16.x.x to 17.x.x, and then from 17..x.x -> 18.x.x, until you find where the breaking occurs.

[dhpizza]

Hi @javsalgar

I did what you suggested and found the breaking change to be in 24.0.0 to 25.0.0

Bitnami Chart version	spring-cloud-dataflow app version	spring-cloud-skipper app version	task issues	stream issues	comments
24.0.0	docker.io/bitnami/spring-cloud-dataflow:2.11.0-debian-11-r0	docker.io/bitnami/spring-cloud-skipper:2.9.3-debian-11-r131	no issues 👍 🥳	no issues 👍 🥳	no issues 👍 🥳
25.0.0	docker.io/bitnami/spring-cloud-dataflow:2.11.1-debian-11-r0	docker.io/bitnami/spring-cloud-skipper:2.11.1-debian-11-r0	no issues	❌ "message": "No release manager available for 'null'", "logref": "SkipperException",	😿

Does that help to pinpoint the issue?

Thanks

[carrodher]

From 24.x.x to 25.x.x the only change was the Kafka subchart version, see https://github.com/bitnami/charts/tree/main/bitnami/spring-cloud-dataflow#to-2500. Not sure if that could impact your deployment, but at first glance, I can't see any relation.

[dhpizza]

From 24.x.x to 25.x.x the only change was the Kafka subchart version, see https://github.com/bitnami/charts/tree/main/bitnami/spring-cloud-dataflow#to-2500. Not sure if that could impact your deployment, but at first glance, I can't see any relation.

As you can see from my table above, there was a change in the skipper version.

[dgomezleon]

Hi @dhpizza ,

It seems the change you mentioned was introduced here, so I guess you will be able to reproduce the issue by upgrading from 24.0.0 to 24.0.1.

There, we changed the source of spring-cloud-skipper, since it was moved to a monorepo (link).

Spring Cloud Dataflow has moved to a monorepo.
This repo is archived and now exists as the spring-cloud-skipper module(s) in the Dataflow monorepo.
Please file any issues/PRs in the Dataflow repo.

[dhpizza]

Hi @dgomezleon you right the issue, starts with upgrading from 24.0.0 to 24.0.1.

The issue is with the skipper version

Bitnami Chart version	scdf dataflow version	scdf skipper version
24.0.0	docker.io/bitnami/spring-cloud-dataflow:2.11.0-debian-11-r0	docker.io/bitnami/spring-cloud-skipper:2.9.3-debian-11-r131
24.0.1	docker.io/bitnami/spring-cloud-dataflow:2.11.0-debian-11-r7	docker.io/bitnami/spring-cloud-skipper:2.11.0-debian-11-r2

To reproduce

• Start with fresh and empty postgreSQL
• deploy chart version 24.0.0
• deploy example stream via dataflow UI
  • upgrade chart to 24.0.1

Result:

• GET /streams/definitions?page=0&size=20&sort=name,ASC returns on errors: [{message: "500 : [no body]", logref: "InternalServerError", _links: {self: {href: "/"}}}]
• skipper logs PSQL Error

2023-12-12 08:29:51.186  WARN 1 --- [pool-3-thread-1] o.h.engine.jdbc.spi.SqlExceptionHelper   : SQL Error: 0, SQLState: 22003
2023-12-12 08:29:51.186 ERROR 1 --- [pool-3-thread-1] o.h.engine.jdbc.spi.SqlExceptionHelper   : Bad value for type long : {"metadata":{"apiVersion":"skipper.spring.io/v1","origin":null,"repositoryId":1,"repositoryName":"local","kind":"SpringCloudDeployerApplication","name":"dheereman-stream-test","displayName":null,"version":"1.0.0","packageSourceUrl":null,"packageHomeUrl":null,"tags":null,"maintainer":"dataflow","description":"http | file","sha256":null,"iconUrl":null},"templates":[],"dependencies":[{"metadata":{"apiVersion":"skipper.spring.io/v1","origin":null,"repositoryId":null,"repositoryName":null,"kind":"SpringCloudDeployerApplication","name":"file","displayName":null,"version":"1.0.0","packageSourceUrl":null,"packageHomeUrl":null,"tags":null,"maintainer":"dataflow","description":null,"sha256":null,"iconUrl":null},"templates":[{"name":"file.yml","data":"apiVersion: skipper.spring.io/v1\nkind: SpringCloudDeployerApplication\nmetadata:\n  {{#metadata.entrySet}}\n  \"{{{key}}}\": \"{{{value}}}\"\n  {{/metadata.entrySet}}\nspec:\n  resource: \"{{{spec.resource}}}\"\n  resourceMetadata: \"{{{spec.resource}}}:jar:metadata:{{{spec.version}}}\"\n  version: \"{{{spec.version}}}\"\n  applicationProperties:\n    {{#spec.applicationProperties.entrySet}}\n    \"{{{key}}}\": \"{{{value}}}\"\n    {{/spec.applicationProperties.entrySet}}\n  deploymentProperties:\n    {{#spec.deploymentProperties.entrySet}}\n    \"{{{key}}}\": \"{{{value}}}\"\n    {{/spec.deploymentProperties.entrySet}}\n"}],"dependencies":[],"configValues":{"raw":"\"metadata\":\n  \"name\": \"file\"\n\"spec\":\n  \"resource\": \"docker:springcloudstream/file-sink-kafka\"\n  \"applicationProperties\":\n    \"management.endpoints.web.exposure.include\": \"health,info,bindings\"\n    \"spring.cloud.dataflow.stream.app.label\": \"file\"\n    \"management.metrics.tags.application.type\": \"${spring.cloud.dataflow.stream.app.type:unknown}\"\n    \"management.metrics.tags.stream.name\": \"${spring.cloud.dataflow.stream.name:unknown}\"\n    \"management.metrics.tags.application\": \"${spring.cloud.dataflow.stream.name:unknown}-${spring.cloud.dataflow.stream.app.label:unknown}-${spring.cloud.dataflow.stream.app.type:unknown}\"\n    \"spring.cloud.dataflow.stream.name\": \"dheereman-stream-test\"\n    \"management.metrics.tags.instance.index\": \"${vcap.application.instance_index:${spring.cloud.stream.instanceIndex:0}}\"\n    \"wavefront.application.service\": \"${spring.cloud.dataflow.stream.app.label:unknown}-${spring.cloud.dataflow.stream.app.type:unknown}-${vcap.application.instance_index:${spring.cloud.stream.instanceIndex:0}}\"\n    \"spring.cloud.stream.bindings.input.group\": \"dheereman-stream-test\"\n    \"management.metrics.tags.application.guid\": \"${spring.cloud.application.guid:unknown}\"\n    \"management.metrics.tags.application.name\": \"${vcap.application.application_name:${spring.cloud.dataflow.stream.app.label:unknown}}\"\n    \"spring.cloud.dataflow.stream.app.type\": \"sink\"\n    \"spring.cloud.stream.bindings.input.destination\": \"dheereman-stream-test.http\"\n    \"wavefront.application.name\": \"${spring.cloud.dataflow.stream.name:unknown}\"\n  \"version\": \"3.2.1\"\n  \"deploymentProperties\":\n    \"spring.cloud.deployer.group\": \"dheereman-stream-test\"\n    \"spring.cloud.deployer.bootVersion\": \"2\"\n"},"fileHolders":[]},{"metadata":{"apiVersion":"skipper.spring.io/v1","origin":null,"repositoryId":null,"repositoryName":null,"kind":"SpringCloudDeployerApplication","name":"http","displayName":null,"version":"1.0.0","packageSourceUrl":null,"packageHomeUrl":null,"tags":null,"maintainer":"dataflow","description":null,"sha256":null,"iconUrl":null},"templates":[{"name":"http.yml","data":"apiVersion: skipper.spring.io/v1\nkind: SpringCloudDeployerApplication\nmetadata:\n  {{#metadata.entrySet}}\n  \"{{{key}}}\": \"{{{value}}}\"\n  {{/metadata.entrySet}}\nspec:\n  resource: \"{{{spec.resource}}}\"\n  resourceMetadata: \"{{{spec.resource}}}:jar:metadata:{{{spec.version}}}\"\n  version: \"{{{spec.version}}}\"\n  applicationProperties:\n    {{#spec.applicationProperties.entrySet}}\n    \"{{{key}}}\": \"{{{value}}}\"\n    {{/spec.applicationProperties.entrySet}}\n  deploymentProperties:\n    {{#spec.deploymentProperties.entrySet}}\n    \"{{{key}}}\": \"{{{value}}}\"\n    {{/spec.deploymentProperties.entrySet}}\n"}],"dependencies":[],"configValues":{"raw":"\"metadata\":\n  \"name\": \"http\"\n\"spec\":\n  \"resource\": \"docker:springcloudstream/http-source-kafka\"\n  \"applicationProperties\":\n    \"management.endpoints.web.exposure.include\": \"health,info,bindings\"\n    \"spring.cloud.dataflow.stream.app.label\": \"http\"\n    \"management.metrics.tags.application.type\": \"${spring.cloud.dataflow.stream.app.type:unknown}\"\n    \"management.metrics.tags.stream.name\": \"${spring.cloud.dataflow.stream.name:unknown}\"\n    \"management.metrics.tags.application\": \"${spring.cloud.dataflow.stream.name:unknown}-${spring.cloud.dataflow.stream.app.label:unknown}-${spring.cloud.dataflow.stream.app.type:unknown}\"\n    \"spring.cloud.dataflow.stream.name\": \"dheereman-stream-test\"\n    \"management.metrics.tags.instance.index\": \"${vcap.application.instance_index:${spring.cloud.stream.instanceIndex:0}}\"\n    \"wavefront.application.service\": \"${spring.cloud.dataflow.stream.app.label:unknown}-${spring.cloud.dataflow.stream.app.type:unknown}-${vcap.application.instance_index:${spring.cloud.stream.instanceIndex:0}}\"\n    \"spring.cloud.stream.bindings.output.producer.requiredGroups\": \"dheereman-stream-test\"\n    \"management.metrics.tags.application.guid\": \"${spring.cloud.application.guid:unknown}\"\n    \"spring.cloud.stream.bindings.output.destination\": \"dheereman-stream-test.http\"\n    \"management.metrics.tags.application.name\": \"${vcap.application.application_name:${spring.cloud.dataflow.stream.app.label:unknown}}\"\n    \"spring.cloud.dataflow.stream.app.type\": \"source\"\n    \"wavefront.application.name\": \"${spring.cloud.dataflow.stream.name:unknown}\"\n  \"version\": \"3.2.1\"\n  \"deploymentProperties\":\n    \"spring.cloud.deployer.group\": \"dheereman-stream-test\"\n    \"spring.cloud.deployer.bootVersion\": \"2\"\n"},"fileHolders":[]}],"configValues":null,"fileHolders":[]}
...

because the skipper_status table in the skipper DB had this

suddenly holds wrong values in the platform_status column

[dgomezleon]

Hi @dhpizza,

My gut is that this could be related to that discrepancy between versions existing in previous releases. Note that Chart version 24.0.1 is the first one where spring-cloud-skipper and spring-cloud-dataflow use the same component version (2.11.0). Not sure if an upgrade using a Chart with all the components from version 2.9.3 to 2.11.0 would work. Otherwise, it could be related to an upstream change in the schema expected by the component.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[dhpizza]

Hi @dhpizza,

My gut is that this could be related to that discrepancy between versions existing in previous releases. Note that Chart version 24.0.1 is the first one where spring-cloud-skipper and spring-cloud-dataflow use the same component version (2.11.0). Not sure if an upgrade using a Chart with all the components from version 2.9.3 to 2.11.0 would work. Otherwise, it could be related to an upstream change in the schema expected by the component.

Hi @dgomezleon should that not be part of the quality assurance when releasing a new version of the Chart? Any advice on how to workaround this is highly appreciated 😃

[dhpizza]

opened new issue https://github.com/bitnami/charts/issues/21839

[carrodher]

I just reopened this one and removed the new one so we don't lose all the existing context

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[dhpizza]

Hi @carrodher again auto-closed without resolution 😿

[nithril]

We had the same issue with the springcloud/spring-cloud-dataflow-server:2.11.2 image. Not sure it relates to bitnami.

We chose to shortest path, delete / recreate everything.

[tomgeraghty3]

I've raised an Issue for this (not realising this one exists) here: https://github.com/spring-cloud/spring-cloud-dataflow/issues/5715 but I think I know the issue ... one of the skippers tables (skipper_manifest.data) was missing from the migration. I've raised a PR to fix it in a fix-forward fashion: https://github.com/spring-cloud/spring-cloud-dataflow/pull/5714 if someone could review plz

[dhpizza]

Hi @tomgeraghty3

IIUC I could now use the latest 2.11.X image of skipper and it would have that fix included?

Unfortunately not the case, the issue persists when running server image docker.io/bitnami/spring-cloud-skipper:2.11.2-debian-12-r11

[dhpizza]

I can confirm that an upgrade to

docker.io/bitnami/spring-cloud-dataflow:2.11.3-debian-12-r3 docker.io/bitnami/spring-cloud-skipper:2.11.3-debian-12-r4

i.e. bitnami/spring-cloud-data-flow chart version 29.0.9

fixes the skipper issue. Unfortunately, now the server has errors

java.lang.Exception: Apparent connection leak detected

when querying tasks/definitions which I might raise in a separate issue.