Good development practice is to test everything in non-production and only promote working configurations into production. In highly regulated environments is it not unusual to perfom unit, integration, funtional, performance and user-acceptance testing before considering something to be ready for production. It is also not unusual to have requirements for dedicated production and non-production hardware which in turn dictates separate CloudFoundry foundations for each.
Within CloudFoundry deployments each buildpack must have a unique name and there is no support for having multiple versions of the same buildpack.
When a new buildpack is released from upstream, CF platform operators need to introduce that buildpack into their CF foundation, but directly replacing an existing buildpack is not a totally risk-free operation. As buildpacks evolve older component versions are removed and newer ones are added, TLS Cipher defaults are changed as different vulnerabilities are discovered and from time to time a faulty component is introduced. Replacing an existing buildpack can mean that an app which has formerly been deployed in non-production for acceptance-testing with an older buildpack will fail to start by the time it is deployed to production with a newer one.
Application developers need assurances that the buildpack that they are using when they begin their integration testing will still be available by the time their application is pushed into production.
To overcome this issue it is not uncommon for platform operators to have developed their own solution for introducing new buildpacks more gracefully. An example of this would be deploying using buildpack names which include a version number (e.g. java_buildpack-4.40, java_buildpack-4.41) and providing guarantees around how long a given buildpack will be present on the platform (e.g. ensure at least current+previous versions and additionally preserve any buildpack that is less than one month old)
When pushing applications to CF a developer may either specify the buildpack(s) by name, or allow buildpack autodetection to decide. If specifying by name then the developer will need to update their mainfests each time a new buildpack version is added - This is tedious, frustrating and easily forgotten. Using autodetection offers a significant advantage because the buildpack priority order within CF can be used to steer people towards newer buildpacks whilst still providing older ones for backwards compatability. Unfortunately autodetection comes with two disadvantates:
It doesn't always choose the correct buildpack
Downloading every buildpack to the diego cell each time an app is staged creates stress on backend storage and gigabytes of network traffic, particularly with multiple versions of each buildpack.
What?
This proposal is for allowing regex/wildcards within buildpack names in order to help the buildpack detection be more precise.
e.g.
buildpacks:
- /java_buildpack-.*/
Where regex is specified (denoted with the leading and trailing forward-slash characters), all non-matching buildpacks can be ignored in the detect phase. Behaviour where regex is not used would be unaltered.
Making this change would enable developers to steer the buildpack detection process and ensure that their app was detected as the correct type, whilst still allowing for the use version-named buildpacks on the platform. This change would also reduce network traffic as diego cells would not need to download all buildpacks each time an app is staged.
Although this isn't a perfect solutuion for buildpack versioning, it will not require large changes within CF to implement, nor disruption to the developer experience.
Putting this proposal up for discussion to see if this appeals to the community, or if anyone has a better approach to managing buildpack versioning.
Proposal Name
Guide buildpack detection using regex
Why?
Good development practice is to test everything in non-production and only promote working configurations into production. In highly regulated environments is it not unusual to perfom unit, integration, funtional, performance and user-acceptance testing before considering something to be ready for production. It is also not unusual to have requirements for dedicated production and non-production hardware which in turn dictates separate CloudFoundry foundations for each.
Within CloudFoundry deployments each buildpack must have a unique name and there is no support for having multiple versions of the same buildpack.
When a new buildpack is released from upstream, CF platform operators need to introduce that buildpack into their CF foundation, but directly replacing an existing buildpack is not a totally risk-free operation. As buildpacks evolve older component versions are removed and newer ones are added, TLS Cipher defaults are changed as different vulnerabilities are discovered and from time to time a faulty component is introduced. Replacing an existing buildpack can mean that an app which has formerly been deployed in non-production for acceptance-testing with an older buildpack will fail to start by the time it is deployed to production with a newer one.
Application developers need assurances that the buildpack that they are using when they begin their integration testing will still be available by the time their application is pushed into production.
To overcome this issue it is not uncommon for platform operators to have developed their own solution for introducing new buildpacks more gracefully. An example of this would be deploying using buildpack names which include a version number (e.g. java_buildpack-4.40, java_buildpack-4.41) and providing guarantees around how long a given buildpack will be present on the platform (e.g. ensure at least current+previous versions and additionally preserve any buildpack that is less than one month old)
When pushing applications to CF a developer may either specify the buildpack(s) by name, or allow buildpack autodetection to decide. If specifying by name then the developer will need to update their mainfests each time a new buildpack version is added - This is tedious, frustrating and easily forgotten. Using autodetection offers a significant advantage because the buildpack priority order within CF can be used to steer people towards newer buildpacks whilst still providing older ones for backwards compatability. Unfortunately autodetection comes with two disadvantates:
What?
This proposal is for allowing regex/wildcards within buildpack names in order to help the buildpack detection be more precise.
e.g.
Where regex is specified (denoted with the leading and trailing forward-slash characters), all non-matching buildpacks can be ignored in the detect phase. Behaviour where regex is not used would be unaltered.
Making this change would enable developers to steer the buildpack detection process and ensure that their app was detected as the correct type, whilst still allowing for the use version-named buildpacks on the platform. This change would also reduce network traffic as diego cells would not need to download all buildpacks each time an app is staged.
Although this isn't a perfect solutuion for buildpack versioning, it will not require large changes within CF to implement, nor disruption to the developer experience.
Putting this proposal up for discussion to see if this appeals to the community, or if anyone has a better approach to managing buildpack versioning.
Proposal URL
n/a (Google docs blocked by employer)
Point of Contact
jimconner on CF-Slack