[service] [request]: Need for testing environments in lieu of free dynos

DrewImm commented 2 years ago

Required Terms

[X] I agree to follow this project's Code of Conduct
[X] I have read and accept the Salesforce Program Agreement

What service(s) is this request for?

Dynos, Postgres, Redis

Tell us about what you're trying to solve. What challenges are you facing?

With the upcoming deprecation of free dynos; this will create a gap in the ability to perform quality assurance, integration testing, and user acceptance testing. Currently, we use free dynos prior to production deployments, often times prior to our clients begin generating revenue; as is common on Heroku. Deprecation of free dynos means we will be unable to run testing with environment parity.

Andser99 commented 2 years ago

This is already partially available through Heroku local deployment. https://devcenter.heroku.com/articles/heroku-local However, the need to setup your own addons/db/whatever might be the reason you won't use it. Heroku could provide a fully local environment with addon setups in this case.

ombr commented 2 years ago

Hi @Andser99,

Heroku local is far away from a review app in terms of dev/prod parity. It is not using docker but the local environment, postgresql and redis version will not mirror what heroku is using.

We tried in the past to have a local environment mirror Heroku and we ended up doing this. https://github.com/SharinPix/docker-heroku/blob/heroku-18/Dockerfile and then this: https://github.com/SharinPix/docker-heroku/blob/main/Dockerfile But it is far from ideal. It would be great if Heroku add something like this out of the box.

We add an issue in the past that we could reproduce only using the heroku docker image. (Upgrading the docker image helped us

For the record here is the follow-up we had from heroku (https://help.heroku.com/778719):

Hi Luc

Hope you are well.

Since we last spoke, I've been able to track down the cause of this issue:
The updated stack image amongst other updates included an upgrade of libpq from v11 to v12, pulled in via the upstream PostgreSQL APT repository [1].
Our pre-release review process identified this major version change, so we had reviewed the PostgreSQL 12 release notes [2] to check for any libpq changes that might affect customers.
However it appears that there was a breaking change missing from the release notes, regarding libpq's validation of connection parameters now being more strict [3]
This affects the sharinpix app since its database.yml configuration file uses connect_timeout: <%= ENV['DB_CONNECT_TIMEOUT'] || 5 %># s -- which is interpreted as '5# s' rather than 5 since there is no white-space before the comment (meaning it's not parsed as a comment)
Therefore the updated libpq now raises an error, when previously it silently ignored the invalid value for connect_timeout
On top of this, the ruby-pg gem appears to have a bug where it doesn't surface the real error message (invalid integer value "5# s" for keyword "connect_timeout") but instead reports a generic PG::UnableToSend: no connection to the server - which then > makes debugging harder
As such we'd recommend updating the value for connect_timeout to add a space character before the comment, so that the app > works fine with libpq v12 when we eventually update to it in the future (we'll be pinning to v11 for the short/medium term and carefully assessing impact before attempting the rollout again). Making this change will also ensure that your chosen value for connect_timeout is used, since at the moment libpq v11 is ignoring it entirely.

We have also:
Reported the missing libpq breaking change mention on the release notes to the PostgreSQL project
Filed an issue against ruby-pg about the real error message being hidden [4]
Added more detail to the rollback message on the Heroku changelog entry [5] (in case that helps for your communication to customers)
I just want to say again - we're extremely sorry for the disruption this caused you and your customers.

If you have any other questions please let us know.

[1] https://github.com/heroku/stack-images/blob/f081455c0ee8f23989b25755aec2ac01c7c8f3f8/heroku-18/bin/heroku-18.sh#L17 [2] https://www.postgresql.org/docs/12/release-12.html [3] https://github.com/postgres/postgres/commit/e7a2217978d9cbb2149bfcb4ef1e45716cfcbefb [4] https://github.com/ged/ruby-pg/issues/302 [5] https://devcenter.heroku.com/changelog-items/1688

Thanks.

Luc

DrewImm commented 2 years ago

To @Andser99's point; there are some additional drawbacks of using local env:

No parity in hardware and OS
No parity in buildpacks/slug process
No parity in network infrastructure, external routing, NAT

afawcett commented 2 years ago

Interested in this threads views on our announcement today for eco and mini price plans - I know these are still not free - but might be acceptable low cost ways to test software that could be built into your client sow along with other pre-production project resources and time? https://help.heroku.com/KP5RQQVO/low-cost-plans-faq

Additionally we have also posted for discussion on this site Heroku Playgrounds - though these are aimed at developer education than pre-production testing - so i mention it more as it maybe something of related interest.

capeterson commented 2 years ago

https://blog.heroku.com/eco-and-mini-plans-ga is our official answer on new plans that are effective for individual developers and lightweight testing use cases.

heroku / roadmap