[pull:db] SSL validation error on subscriptions with custom SSL certs

[anavarre]

While it's not ideal and we very much want to check the SSL certificate always before we pull down a DB into Cloud IDE, there are cases when it's not possible. As an example, ACP has a known limitation https://docs.acquia.com/cloud-platform/known-issues/#self-service-ssl-certificates-overwrite-acquia-s-default-certificate

When requesting the Acquia default domain, the subscriber’s self-service SSL certificate loads instead of the Acquia SSL certificate that covers the Acquia default domains. This behavior causes an SSL error in the browser. Install and activate two or more custom certificates on any affected environment to remove this error on the Acquia default domain.

This is having the AH_DOWNLOAD link fail with the below cURL error:

cURL error 51: SSL: no alternative certificate subject name matches target host name `<stage>.devcloud.acquia-sites.com`

This issue is about exploring a way to cURL with the -k or --insecure option to pass this roadblock without blocking our ACP customers from using Cloud IDE successfully altogether.

[danepowell]

Can you confirm if this is still an issue? I don't see why pull:db would call the production site URL. It downloads the DB via Cloud API, the customer SSL cert should have nothing to do with it.

[grasmash]

It's not hitting a prod URL. The Acquia URL returned by the Cloud API had an invalid cert.

[SophieG]

This is also affecting non ACP customers as ACSF ones cURL error 51: SSL: no alternative certificate subject name matches target host name '<sub>01dev.enterprise-g1.acquia-sites.com'

[grasmash]

I think we really need the full verbose command output here.

[danepowell]

Also an affected subscription name or UUID so we can reproduce (via support ticket or TAM if you'd like to communicate it privately)

[grasmash]

I do not think ACLI should fall back to pulling down a DB without checking the certificate. Seems risky. Perhaps we could just make the error more helpful.

@CashWilliams thoughts?

[SophieG]

For the ACSF customer we found a work around, as the non-valid certificate issue was due to the fact that they were using the old method to install certificates (certificates installed on the balancers by Acquia vs customers installing themselves certificates from their Cloud UI). This method was causing an "override" of the defaults Acquia certificate on the balancers, hence the SSL error when accessing <sub>01dev.enterprise-g1.acquia-sites.com.

[danepowell]

I'm finally able to reproduce this and explain why it's happening. If you call the getEnvironmentsDatabaseDownloadBackup endpoint, it generates and returns a tokenized 10-minute download URL on the customer domain. The backup doesn't stream directly from Cloud API.

I'm also going to change the title to reflect the problem, rather than an (insecure) solution.

[grasmash]

So what's the solution? Can we simply change the url?

[grasmash]

Wait... does this happen in 1.9.0? It looks like we use the cloud.acquia.com base url from the client: https://github.com/acquia/cli/blob/2d7f7828a0034e69f7bec167a4d889561db88312/src/Command/Pull/PullCommandBase.php#L278

[danepowell]

It looks like we use the cloud.acquia.com base url from the client

@grasmash welcome to the party 😄 That's the question I was asking from the beginning. See my last post: yes, we call cloud.acquia.com to get the backup URL. The problem is that Cloud / Cloud API does not directly stream the backup file as you'd expect; rather, it returns a 302 redirect to a tokenized download URL on the customer domain, which downloads the file from gluster.

So this does happen in 1.9.0 and there's nothing Acquia CLI can do about it other than to ignore the SSL cert.

As to solutions, I think our only choices are:

• Encourage customers to fix the SSL cert (see Sophie's post above)
• Add an --ignore-ssl-certs parameter
• Coordinate with the Cloud or Cloud API teams to get them to stream downloads from the domain with working SSL cert, or directly from Cloud API

[paulwhenry]

Note that the fix per Sophie's post above, is not achievable for all customers. I have an ACP customer (z1166506) who's experiencing this same issue, but they've installed their SSL cert themselves (via the SSL self-service process). So there is no workaround. Leaving the --ignore-ssl-certs parameter as the only (current) suggested option.

[grasmash]

I'd like to get sign off from SecEng before adding such an option.

[paulwhenry]

To be clear, I'm not requesting that we implement the --ignore-ssl-certs parameter (as I'm aware of the security concerns). I'm asking what options customers have.

[grasmash]

I think the three categories of solutions are:

1. fix the certificate
2. Ignore the certificate
3. Make a change to cloud such that customer domains are not used

ACLI is only able to control #2.

[bschilt]

I just ran into this issue. Created an IDE on a customer's subscription. Walking though the "getting started" steps and "copy from the cloud platform" button ultimately fails when attempting to pull the DB.

[nmangold]

I am also experiencing this issue with an ACSF account. The problem seems to be well understood at this point, and I'm not sure if any of the information will be helpful, but just in case, https://insight.acquia.com/support/tickets/1174414. I assume the problem for me is also caused by the overridden certs.

[dhaggman]

Is there a workaround for when the certificate is installed via Acquia Cloud UI?

https://docs.acquia.com/cloud-platform/known-issues/#self-service-ssl-certificates-overwrite-acquia-s-default-certificate mentions:

Install and activate two or more custom certificates on any affected environment to remove this error on the Acquia default domain.

But only one certificate per environment can be active at any given time.

[grasmash]

@anavarre @danepowell @itafroma Is there an open Cloud API ticket to address this issue?

The issue being "Cloud API should provide a database download URL that uses an Acquia domain."

[grasmash]

I'm starting to lean towards adding a --insecure option to the pull:* commands with a big fat warning. @CashWilliams can you add your opinion here?

[nmangold]

Confirming that removing the custom SSL cert and eliminating the ELB solved the issue for me.

[grasmash]

I chatted with the API team, and it seems like this is unlikely to be resolved. The solution on cloud, in the documented work arounds, is to simply change the protocol to be http when downloading a database back up in this scenario.

[paperscenery]

Thanks to all of you for documenting and looking into this issue. For anyone who encounters cURL error 51: SSL: no alternative certificate subject name matches target host name as we did using Lando to pull sites from Acquia Cloud for development, we found the following three workarounds to be successful. Each has its drawbacks, so pick your poison.

a) Temporarily disable any custom SSL certificates via the Acquia Cloud UI on an environment before running lando pull / or lando acli pull:db — obviously this is only workable for pulls from a non-production environment

b) Make the pull from a dev or test environment that doesn’t have a custom SSL installed in the first place — depending on your workflow this might be the easiest.

c) Manually download the database backup and run lando db-import — slower and you have to place your db dumb in a directory INSIDE of the Lando root directory (and add it to git-ignore)

[isaiahnixon-acquia]

I just ran into this issue when using the Acquia Lando recipe to pull down a database from ACSF. @paperscenery's c) option seems to be my only recourse in this situation. Just wanted to put this back on the radar.