🚀 Pre-release master -> staging_JellyBeans3

matusdrobuliak66 commented 1 year ago

What kind of pre-release?

master branch

Sprint Name

JellyBeans

Pre-release version

3

Commit SHA

f1fefe2f3b7ce0a5e42cd9814244926147673f50

Did the commit CI suceeded?

[x] The commit CI succeeded.

Motivation

Weekly release to Staging

What Changed

https://github.com/ITISFoundation/osparc-simcore/pull/4100 by @GitHK
https://github.com/ITISFoundation/osparc-simcore/pull/4117 by @matusdrobuliak66
https://github.com/ITISFoundation/osparc-simcore/pull/4131 by @matusdrobuliak66
https://github.com/ITISFoundation/osparc-simcore/pull/4133 by @sanderegg
https://github.com/ITISFoundation/osparc-simcore/pull/4055 by @GitHK
https://github.com/ITISFoundation/osparc-simcore/pull/3974 by @GitHK
https://github.com/ITISFoundation/osparc-simcore/pull/4139 by @GitHK
https://github.com/ITISFoundation/osparc-simcore/pull/4136 by @matusdrobuliak66
https://github.com/ITISFoundation/osparc-simcore/pull/4105 by @matusdrobuliak66
https://github.com/ITISFoundation/osparc-simcore/pull/4141 by @pcrespov
https://github.com/ITISFoundation/osparc-simcore/pull/4064 by @pcrespov
https://github.com/ITISFoundation/osparc-simcore/pull/4138 by @sanderegg
https://github.com/ITISFoundation/osparc-simcore/pull/4137 by @sanderegg
https://github.com/ITISFoundation/osparc-simcore/pull/4146 by @matusdrobuliak66
https://github.com/ITISFoundation/osparc-simcore/pull/4147 by @matusdrobuliak66
https://github.com/ITISFoundation/osparc-simcore/pull/4134 by @GitHK
https://github.com/ITISFoundation/osparc-simcore/pull/4145 by @matusdrobuliak66
https://github.com/ITISFoundation/osparc-simcore/pull/4154 by @dependabot
https://github.com/ITISFoundation/osparc-simcore/pull/4160 by @GitHK
https://github.com/ITISFoundation/osparc-simcore/pull/4162 by @sanderegg
https://github.com/ITISFoundation/osparc-simcore/pull/4163 by @GitHK
https://github.com/ITISFoundation/osparc-simcore/pull/4165 by @sanderegg
https://github.com/ITISFoundation/osparc-simcore/pull/4170 by @GitHK

Devops check ⚠️ devops

[x] Add changes (if any) and release tags in https://github.com/ITISFoundation/osparc-ops-environments/tags
[x] Add changes (if any) and release tags in osparc-ops-deployment-configuration (staging.osparc.io & osparc-staging.speag.com)
[x] warning LOG_FORMAT_LOCAL_DEV_ENABLED=0 needs to be added to the config file in all deployments (ops-deployment-configuration)

e2e testing check 🧪

green

Summary 📝

[x] make release-staging name=JellyBeans version=3 git_sha=f1fefe2f3b7ce0a5e42cd9814244926147673f50
[x] Draft pre-release

[x] Announce

{"start": "2023-04-27T12:00:00.000Z", "end": "2023-04-27T12:30:00.000Z", "reason": "Release JellyBeans3 "}

Releasing

[x] Release (release draft)
[x] Check Release CI
[x] Check deployed
- [x] aws deploy
- [x] dalco deploy
[x] Delete announcement
[x] Check e2e runs

[x] Announce

https://github.com/ITISFoundation/osparc-simcore/releases/tag/staging_JellyBeans3

matusdrobuliak66 commented 1 year ago

For now, we will wait with the pre-release until we understand more why these alerts started to pop up in Master: "Could not contact dynamic-sidecar to save service" AND NOT container_name:/.*graylog_graylog.*/

@GitHK is looking into it

https://github.com/ITISFoundation/osparc-simcore/pull/4160

GitHK commented 1 year ago

I would like to block this release since currently I am investigating an issue which is causing false positives on master when services are shut down. For some unknown reason they try to be removed once more after they are removed.

GitHK commented 1 year ago

Identified the source of the issues:

Release strategy:

to master check after a few days that https://github.com/ITISFoundation/osparc-simcore/pull/4163 no longer produces tracebacks in director-v2
to staging check that autoscaling is still working after the changes introduced by https://github.com/ITISFoundation/osparc-simcore/pull/4163

matusdrobuliak66 commented 1 year ago

I am changing the Commit SHA to the: https://github.com/ITISFoundation/osparc-simcore/commit/b2223f1387d8597d9603183c1e8d528467883db7 which includes the fix

GitHK commented 1 year ago

@matusdrobuliak66 I'd like you to wait for the below PR to be in master. Will provide a very clean director-v2 log situation.

https://github.com/ITISFoundation/osparc-simcore/pull/4170

matusdrobuliak66 commented 1 year ago

@matusdrobuliak66 I'd like you to wait for the below PR to be in master. Will provide a very clean director-v2 log situation.

recycle Drop log priority #4170

Ok I updated the Commit SHA to the: https://github.com/ITISFoundation/osparc-simcore/commit/f1fefe2f3b7ce0a5e42cd9814244926147673f50

We can do the prelease after lunch.

matusdrobuliak66 commented 1 year ago

In the beginning of release:

In AWS staging there was a hanging sidecar with error: POWER KEY PRESSED on the autoscaling node. Thanks to the AWS CLOUDTRAIL service we were able to find that it was not a system error, but the autoscaling node was killed manually by us.

After the release we found out 2 things:

As we introduced a new variable LOG_FORMAT_LOCAL_DEV_ENABLED to the config file in the osparc-ops-deployment-configuration -> we found out that the value is not propagated correctly to the final container. One of the problems might be that it seems that production & staging shares the same config file (@mrnicegyu11 feel free to add any additional info you found out on the way)
- Even though this ENV is not critical for the functionality
- :warning: WE SHOULD NOT RELEASE TO PRODUCTION until it is fixed :warning: until we understand all the side-effects
There are some warnings & errors occurring in the director-v2 (also seems is not crucial for the functionality - e2e's are green)

mrnicegyu11 commented 1 year ago

re: variable issues: traced in https://git.speag.com/oSparc/osparc-ops-environments/-/issues/567

ITISFoundation / osparc-simcore