search

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
282 stars 201 forks source link

IET needs PO approval for Sidekiq UI promotion to production #46253

Open ewilson-adhoc opened 2 years ago

ewilson-adhoc commented 2 years ago

What is the issue

During our Go/No Go presentation this Tuesday (8/23) we were instructed to update the initiative release plan (https://github.com/department-of-veterans-affairs/va.gov-team/issues/40061) and release plan (https://github.com/department-of-veterans-affairs/va.gov-team/issues/46139) and then contact our PO for final approval. @td-usds signed off on this before he went on vacation, but I wanted to confirm with @mchelen before we proceed. We’re ready to start the deploy of the standalone Sidekiq UI service into prod.

Ideally we would like to progress with this before Thomas got back from vacation.

Who is it effecting or who will be effected

This is blocking IET from releasing Sidekiq UI standalone to prod, which is one of our quarterly objectives -- IET would like to accomplish this work before our contract ends so that it will not fall upon Console Services to complete this task.

What have you done to resolve this so far

IET presented at the go/no go meeting: the template that we used to describe our VSP platform initiative was incorrect, we were instructed to remedy this issue and then talk to our PO for approval. We filled out the correct templates: our PO is on vacation, we were instructed to use this escalation process.

Dependencies

Mike Chelan is IETs stand-in PO.

After Monday 08/29 Thomas will return from vacation and is the PO who should be contacted for this decision -- we have a sync scheduled with him when he comes back and can discuss the matter with him then, but would prefer to avoid postponing this until he comes back.

What's at risk

We have already gotten approval from Thomas for this release, but I would like to follow the procedure requested of us during the go/no go.

Recommendation(s)

1) IET is given approval for release, or is given steps that need to be taken in order to get approval asynchronously.

2) IET waits for the next go/go no to present their new documents, this meeting is already very full with other teams trying to wrap up spike work.

Additional Information

initiative release plan release plan Go/No go Notes Talking points for go/no go

andreahewitt-odd commented 2 years ago

@ewilson-adhoc @FHiatt What metrics/monitoring/alerting are you tracking after go live to measure impact and where is that being tracked?

andreahewitt-odd commented 2 years ago

@ewilson-adhoc when is your sync with Thomas? I hate releasing stuff on Fridays so if it is Monday when he gets back, can you just wait until then to go live?

ewilson-adhoc commented 2 years ago

@andreahewitt-odd we have datadog alerts set up to monitor that the endpoint in responsive, and if the service goes down an alert is sent to our slack channel. In terms of measuring impact, our plan was to monitor the number of users who have requested access for the standalone service versus (which would indicate new users to the tool).

I managed to completely miss that today is Friday -- an agree on avoiding non-critical Friday releases: we have a scheduled sync first thing on Monday with him, so we can probably go ahead and close out this issue. Thank you for taking a look at this.

andreahewitt-odd commented 2 years ago

Just going to keep this open until monday when it's resolved or needs to be escalated

td-usds commented 2 years ago

If the alerting for the prod (not any other envs) deploy can be directed at the #oncall channel and PagerDuty, then this may proceed.