search

cloudfoundry / cflinuxfs4-release

Cloud Foundry stack based on Ubuntu 22.04 LTS
Apache License 2.0
0 stars 2 forks source link

Incompatible change in v1.x (Ruby and Python removed) #2

Closed jochenehret closed 1 year ago

jochenehret commented 1 year ago

Hi everyone,

the cflinuxfs4 releases v1.x have Ruby and Python removed. The release has been automatically pushed to the cf-deployment "develop" branch and now we see a number of buildpack detection tests failing:

https://concourse.wg-ard.ci.cloudfoundry.org/teams/main/pipelines/cf-deployment/jobs/experimental-cats-cflinuxfs4/builds/136

Summarizing 5 Failures:
  [FAIL] [detect] Buildpacks php when using cflinuxfs4 stack [It] makes the app reachable via its bound route
  /go/src/github.com/cloudfoundry/cf-acceptance-tests/detect/buildpacks.go:149
  [FAIL] [detect] Buildpacks ruby when using cflinuxfs4 stack [It] makes the app reachable via its bound route
  /go/src/github.com/cloudfoundry/cf-acceptance-tests/detect/buildpacks.go:44
  [FAIL] [detect] Buildpacks java when using cflinuxfs4 stack [It] makes the app reachable via its bound route
  /go/src/github.com/cloudfoundry/cf-acceptance-tests/detect/buildpacks.go:84
  [FAIL] [detect] Buildpacks staticfile when using cflinuxfs4 stack [It] makes the app reachable via its bound route
  /go/src/github.com/cloudfoundry/cf-acceptance-tests/detect/buildpacks.go:186
  [FAIL] [detect] Buildpacks binary when using cflinuxfs4 stack [It] makes the app reachable via its bound route
  /go/src/github.com/cloudfoundry/cf-acceptance-tests/detect/buildpacks.go:206

This is the failing CATs suite: https://github.com/cloudfoundry/cf-acceptance-tests/blob/main/detect/buildpacks.go

Tests fail because some buildpacks (and also some test apps) assume that a "ruby" or "erb" executable is available. This could also affect running CF apps.

Why exactly did you decide to remove Ruby and Python? Is there a chance to revert this change to enable a smooth transition from cflinuxfs3 to cflinuxfs4?

Thanks and Best Regards,

Jochen.

ryanmoran commented 1 year ago

We will be fixing these issues in the buildpacks and advise not to upgrade to the v1 line of cflinuxfs4 until all buildpacks are patched. The following issues can be tracked for progress:

stephanme commented 1 year ago

This comes very late, ~5 weeks before cflinuxfs3/Bionic goes out of maintenance.

Users with cflinuxfs4 apps that have a runtime dependency on the stack provided ruby/python will see their apps crashing after the next CF update (that ships the cflinuxf4 version w/o ruby/python).

h0nIg commented 1 year ago

not just very late, you combine a security deadline (fs4) with a buildpack upgrade. Most of our users pin the buildpack version via #hashtag instead of relying on the latest buildpack provided with CF-deployment. This is required due to stable delivery of the same artefact with the ability to easily downgrade in case of issues.

This is without grace period and pre-notification and will break many applications due to short time frames!

Just as an example, the most critical aspect is not the mentioned runtime dependency but the build time dependency: https://github.com/cloudfoundry/staticfile-buildpack/blob/master/CHANGELOG#L4 IMHO breaking changes only done via major version change, but this was ignored as well...

With this in place, you force applications with ruby templating to have major actions on short notice and there is even no way to downgrade or roll back their buildpack, once fs3 is gone due to a CVE! Despite that they are busy with cflinuxfs4 checks, this should not happen and would be a disaster, caused by "theoretical future reductions of CVE" and non-available bundles of ruby inside the individual buildpacks.

https://github.com/cloudfoundry/cflinuxfs4/pull/3

Remove Ruby from the stack to reduce the CVE surface area of the image.

Overall i would like to request you to think about theoretical CVE's in future vs. breaking changes. I accept the reason, because reduction of CVE is a good approach! But please with a reasonable timeframe of 3-6 months ETA to production (meaning 2-4 months as of today) or at least without bundle of fs3 deadline. Thank you!

@ryanmoran @TisVictress @sophiewigmore @robdimsdale @krismarc

krismarc commented 1 year ago

This is serious. The product is 'sold' as best developer experience. So they don't have to care about things like this except the code. What should I tell to my users now? Please start shipping your own runtime/interpreter? Please also start to maintain it. We are not going to do that anymore after so many years of doing so.

We just talk about buildpacks so far and possibility to continue using them. What about all other reasons of using removed software? Eg. Liberty buildpacks requires only Ruby to be executed. However, we added a custom script which traps java exceptions and if it's a case then using python we generate a dump and send it to S3. This will become broken until we ship python ourselves. Buildpack maintainer would not add python as he do not really need it fo the buildpack itself.

There might be many usecases where people use python in addition to their main purpose of the buildpack.

That's quite a breaking change. As mentioned above. People will start seeing crashed apps and there might be no quick fix for potential @developer.

@kevin-ortega

ryanmoran commented 1 year ago

Hello folks,

We apologize for the late-breaking change. The blast-radius of this specific change ended up being much bigger than we anticipated. For our failure to communicate this effectively at an earlier point in time, we are sorry.

On the topic of the change and its need to address our CVE posture, let me first talk about the current state of the world. Today, the cflinuxfs3 stack contains a version of Ruby (v2.5) that has been unmaintained upstream for nearly 2 years. The stack also contains a version of Python (v2.7) that has been unmaintained upstream for over 3 years. Both runtimes likely contain any number of critical-level security vulnerabilities for which there is no patch or mitigation. Anybody running these stacks in their production cluster should be extremely wary of the possibility of their exploitation.

For cflinuxfs4, based on Ubuntu Jammy, the Ruby version, 3.0, is expected to go out of support approximately 1 year from now (3/31/2024). Python version 3.10, which is the version shipped in the python3 package as part of Jammy, has a slightly better timeline as it remains in support for the next 3.5 years (Oct. 2026), but still falls out of support before Jammy and hence cflinuxfs4 would in April 2027.

With that security posture playing out right now in cflinuxfs3 and there being no good time to ever introduce a breaking change in the stack (especially, when it is exceedingly easy to unknowingly couple to it through your own application code), we felt an urgent need to make the breaking change with as much lead time as we could before cflinuxfs4 became the only supported stack and a change of this magnitude would become all the more painful. Timelines and workloads being what they are, this change happened very late in our process to move everything we could onto cflinuxfs4.

At this point, we cannot roll back the clock and provide early communication that might ease the transition. We can only provide guidance and support to move forward. For the specific issues outlined, we believe there are reasonable workarounds that should not require immediate re-writes of any application.

Specifically, if you are still in need of Ruby or Python in your application and were relying upon the presence of those provided in the stack, you can trigger either the Ruby or Python buildpacks to install a Ruby or Python for you by just including it ahead of your usual buildpack in a multi-buildpack chain. For example, if I use the Staticfile buildpack, and still want Ruby for some reason, I can ask CF to build my app with both Ruby and the Staticfile buildpack with the following command:

cf push <app> --buildpack ruby_buildpack --buildpack staticfile_buildpack

For the breaking change in the Staticfile buildpack, we apologize for our failure to bump the major version. FWIW, the feature of using ERB templating was never expected to be an external-facing feature and has been marked and logged as deprecated for more than 5 years. Separate from incorrect versioning of this feature removal, we feel that the removal of the feature came with exceedingly early feedback of its pending removal. If you still want to enable this workflow, you can follow the steps provided above to get a Ruby installed at runtime. You can then also include a .profile script with contents similar to the following:

erb $HOME/nginx/conf/nginx.conf.erb $HOME/nginx/conf/nginx.conf

This should enable the same outcome without relying upon the internal implementation details of a buildpack for its side effect. You should also consider migrating to the NGINX buildpack as has been stated in the deprecation notice.

Again, apologies for the communication issues and our failure to provide better guidance before now. We believe everyone should have at least some path forward at this point and we are happy to hear any other issues folks encounter with this change or in the migration to cflinuxfs4 more broadly.

Sincerely, The CF Buildpacks Team

jochenehret commented 1 year ago

php-buildpack v4.6.2 is not working yet on cflinuxfs4: https://github.com/cloudfoundry/php-buildpack/issues/854

krismarc commented 1 year ago

Can anyone point me to the information about which cf-deployment release will include this stack as a default? I'd probably be able to find it in the PRs and the source code. However, it would be nice to have a statement here as well.

jochenehret commented 1 year ago

Can anyone point me to the information about which cf-deployment release will include this stack as a default? I'd probably be able to find it in the PRs and the source code. However, it would be nice to have a statement here as well.

This issue summarizes the cflinuxfs4 roadmap: https://github.com/cloudfoundry/cf-deployment/issues/989

The next cf-deployment release will not yet make cflinuxfs4 the default stack. We are first trying to fix all issues related to the cflinuxfs4-release 1.x.x upgrade.

beyhan commented 1 year ago

Hi folks,

@ryanmoran thank you for the understanding and the detailed explanation. I also agree that going that direction is the right one. In the CF community we don't have a well defined deprecation process and maybe that could help regarding the messaging around this change, but this is a topic for the TOC. In this change I'm concerned about the many apps we already have on the cflinuxfs4 stack, where some of them could break with the next CF update containing the Ruby and Python removal. This is an effect from the short timelines between cflinuxfs3 deprecation and the availability of cflinuxfs4 but this is another topic :-). The proposed mitigations need to be executed by end users of CF and our experience is that this takes time. Additionally, we have to announce such breaking changes three months in advance. I think that it will help us and in general make end users of CF happier if we can find a way how the CF operators could mitigate this for end users and offer a deprecation period to them. Do you see any options in that direction?

Regards, Beyhan

beyhan commented 1 year ago

Hi folks,

There is a proposed RFC for this change. This was discussed in the last TOC meeting on 2023-04-04. The meeting notes are available here. We will be happy to get your feedback on this.

Regards, Beyhan

robdimsdale commented 1 year ago

The RFC mentioned above has been approved and we have a clear path forward (providing two BOSH releases - one with ruby + python and one without).

I'll close this issue now.