Devops: Get a dedicated SSL cert for Prod

[MeghanStothers]

From Nick Corcoran @ CITZ reached me in teams

Hi Meghan. I just saw the announcement of the Digital Telework Agreement (Great Job) and see that you're the product owner. I would ask that you obtain a dedicated TLS (SSL) certificate for your production site, and not use the platform wildcard certificate. The wildcard cert is fine for non-prod, but all prod sites need a dedicated certificate. These can be obtained through the following process: https://developer.gov.bc.ca/BC-Government-OpenShift-DevOps-Security-Considerations#tls-certificates

request: get a dedicated TLS (SSL) certificate for production site, and not use the platform wildcard certificate.

@warrenchristian1telus : Can you advise about the urgency of this request? I'm thinking it could wait until next week, but I'd welcome your recommendation.

[Abuchana]

I have email Igor to assist

[warrenchristian1telus]

@MeghanStothers I would recommend requesting the following sub-domains prior to requesting certificates for the same:

• digital-journeys.gov.bc.ca
• analytics.digital-journeys.gov.bc.ca
• api.digital-journeys.gov.bc.ca
• camunda.digital-journeys.gov.bc.ca
• files.digital-journeys.gov.bc.ca
• forms-flow.digital-journeys.gov.bc.ca

I'm wondering if this may be better off as an Epic, so that we can create sub-issues for requesting the domains, related SSL certificates, and the subsequent implementation & testing.

[HasanSifat]

Waiting on 1-2 istore requests to solve this issue

[MeghanStothers]

Check Peter Velinov

[MeghanStothers]

Request submitted by email to IT Service desk/Peter Velinov with a request to follow up with Stella as I am away until August 29.

[MeghanStothers]

Can someone assist and send off the iStore request to an approver? Thank you. I am not able to attach the email, so I'll send it to Alison and Stella by email.

[Abuchana]

Peter Velinov and Kunal have connected with Chris Brown at TES to assist in getting these SSL certs.

[Abuchana]

Peter Velinov is going to confirm with Chris Brown (TES) whether he is going to create the SSL certs for us or if Peter needs to submit an iStore to OCIO to have them created. Whatever avenue we take, once the certs are received, Kunal will install them for us.

[Abuchana]

Chris Brown (TES) is creating these for us and once completed, Warren and/or Kunal will implement them for us.

[Abuchana]

@kkapoor-fresh is there any progress on this one? Last I saw Chris was creating the certs for us...

[kkapoor-fresh]

@Abuchana I haven't heard back from Chris on this yet. I have sent a followup email to him right away

[fazil-ey]

@kkapoor-fresh @warrenchristian1telus Hey, do you have any advice ont his ticket? Is this urgent?

[warrenchristian1telus]

@fazil-ey @kkapoor-fresh I don't think this is urgent as we already have SSL - just not the custom cert. I do have the new certificate available when someone has time to implement. Just let me know who/where I need to send it to.

[fazil-ey]

Waiting for @kkapoor-fresh to action this

[fazil-ey]

@kkapoor-fresh is there anyway we can action this anytime soon?

[kkapoor-fresh]

@fazil-ey, @warrenchristian1telus mentioned that this might not be an item that is urgent. Let me know if it is and I can take a look at it next week.

[warrenchristian1telus]

I don't see a problem with that @kkapoor-fresh. I know you're busy with PECSF. I have the certs for this, if you don't have them yet, so just let me know if/when you need them.

[warrenchristian1telus]

We've had some issues with SSL implementation. The current state of the new certificates is as follows:

Webapp - Waiting on new certificate Webapi - Waiting on new certificate Camunda - Reverted due to application errors Forms flow - Reverted due to application errors Files - Reverted due to application errors Anlytics - In production

@kkapoor-fresh Will continue to proceed with installations and troubleshooting. Please update this ticket to reflect changes as they become available.

[kkapoor-fresh]

Following up with Chris for the new certs.

[MeghanStothers]

@kkapoor-fresh Checking in on this one while warren is off - can we help in any way? Is the Chris you're referring to with the DJ team or with TES? Thanks for moving this along

[kkapoor-fresh]

@MeghanStothers Chris is with the Telus team. I am waiting for a response from him. Will keep the group updated.

[fazil-ey]

@kkapoor-fresh We are launching on March 17, 2023. Let us know if you have any updates on this from Chris

[kkapoor-fresh]

Just got new certs from Chris. I will test them out and see if they work fine.

[kkapoor-fresh]

@Abuchana @warrenchristian1telus I tested the new certs. The cert for the web api seems to be working fine in Prod. Will monitor it through Monday The cert for web app is still not correct. I have sent a note to Chris to check that one.That one might need to be resent.

[MeghanStothers]

Everything should work as expected - to send Igor the latest update on this one

[Abuchana]

Warren connected with Sumesh (AOT) for some assistance, waiting on response

[Stella-Archer]

Iman has tested every thing working fine - error free

[warrenchristian1telus]

We are waiting on two certificates. We need a new one for documents (swagger endpoint) and there is an issue with the main certificate (digital-journeys.apps.silver.devops.gov.bc.ca) needs to be fixed or replaced, as it's from a different SA (Signing Authority) and seems to have a different key from the rest.

[MeghanStothers]

from Warren: we have one of two tickets. One is outstanding for docs. Warren and Iman to connect and make a plan.

[iman-jamali-fw]

@warrenchristian1telus forms-flow-document URL was shortened on DEV to comply with the SSL certificate length limitation. I tested PDF generation and it worked well.

Also done for TEST at this URL.

PROD is still waiting for that change.

[Stella-Archer]

Warren trying to track down right key (there is a mis match :(

[warrenchristian1telus]

We have received renewed SSL certificates for all domains other than documents.

The earliest expiry dates are Fri, 08 Sep 2023 - so we should implement all before then to avoid errors / interruptions.

To resolve the issue with a missing key for documents domain, I have requested a new certificate be created for digital-journeys-docs.apps.silver.devops.gov.bc.ca. I'm hoping this will be available to us prior to Sept. 8th so we can implement all of them at once.

[MeghanStothers]

Testing of specific services will be required to make the component is working and verify the date- we will do the testing this week.

[warrenchristian1telus]

Hi @MeghanStothers @Abuchana @Stella-Archer @iman-jamali-fw - I had a chat with @kkapoor-fresh earlier, and we thought it would be safer to implement this tonight, just in case we have any surprises with the SSL certificates that expire on Friday. It may be nice to have additional time to get external support if something doesn't work as expected.

If we can get a quick approval for this, and hopefully Iman's help, we can schedule for after 4pm this evening. Otherwise, we can proceed tomorrow afternoon as previously discussed.

[Abuchana]

@warrenchristian1telus that makes sense to me and will go with whatever the devs think is best. I have to go out after work today but would be back home at 5:30 if you need me for anything.

[iman-jamali-fw]

@warrenchristian1telus I'm good for both this afternoon or tomorrow.

[MeghanStothers]

Thanks! Go ahead tonight if you're set. I'll be here to support until about 5:30 or 6, although I will not have any PSA devs to support/test. I can certainly ask Ayush to test after- hours if you let me know what tests we should be performing.

[iman-jamali-fw]

@MeghanStothers One sample submission from each live form would be great to ensure different components (website, API, Camunda, PDF generation server) are working OK with the new SSL certs. I'll do the initial test and can submit one Telework test submission.

[warrenchristian1telus]

Thanks @iman-jamali-fw. @kkapoor-fresh can we schedule this to start at 4pm? I'm hoping we'll be able to test somewhere between 4:30-5pm.

[kkapoor-fresh]

@warrenchristian1telus Yes, 4 PM looks OK.

[iman-jamali-fw]

@warrenchristian1telus I'm good with 4 pm too!

[MeghanStothers]

@Abuchana All set for 4 today. @iman-jamali-fw I can help test the Telework (pdf) as well as form access from the website, I'll need other devs to check Camunda, API etc.

[warrenchristian1telus]

I would also like to mention that we not only have to confirm that it still works, but that the certificate dates have successfully updated to 2024.

Private Zenhub Image

[warrenchristian1telus]

@MeghanStothers @Abuchana @iman-jamali-fw @kkapoor-fresh

All SSL certificates have been updated (except documents). Please verify and let us know if any of you spot any issues.

[iman-jamali-fw]

@warrenchristian1telus Thank you. I'll get to the testing.

[MeghanStothers]

Awesome! I submitted a telework agreement at 5:02 today and sent to alison.buchanan@gov.bc.ca in case she can intercept. Findings -

• I tested in the live prod environment
• I was able to begin the digital submission from the Careers & MyHR website
• My submission was successful in terms of email messages as expected, as well as the pdf was attached to the email and displays as expected.
• I can test with someone else with a gov address to make sure the supervisor's role delivers as expected

• @iman-jamali-fw - can you remind me if you have a gov address. If so, I can test with you :)

[warrenchristian1telus]

I've verified all the domains now provide the new certs which expire in Sept. 2024. You may need to clear your cache to see the changes. Everything is looking good on my end.

[iman-jamali-fw]

I checked all the components and they're working great with the new SSL certs.

Also sent one Telework with myself as employee and manager, received both emails with the PDF attached. This means all components (website, API, Camunda and PDF generation) works OK with the new certs.

Tested on PROD in inPrivate/inCongnito window making sure the new certs are picked up on the website.

[MeghanStothers]

Excellent - unless there are further tests, this one is a go! I have asked @chriscaldwell-psa to keep an eye on the DJ mailbox just in case we get any issues in the morn. Thanks all!

[warrenchristian1telus]

Great news - thanks @iman-jamali-fw!

If nobody else can find any issues, I believe this ticket can be closed.