Release v1.38.0 and hotfixes 1 and 2

[pcrespov]

In preparation for release. Here an initial (incomplete) list of tasks to prepare before releasing:

• [x] Prepare staging
• [x] Check changelog 🚨
• [x] Check devops ⚠️
• [x] Test assessment: e2e-testing
• [x] Test assessment: targeted-testing
• [x] Test assessment: user-testing ✅
• [x] Release summary
• [x] Release assessment

---

Prepare staging

• Pre-releases :

  • 750

  • 744

  • 🚑️ hotfix: https://github.com/ITISFoundation/osparc-simcore/releases/tag/staging_switzer3
    • pennsieve issue
    • migration w/ new healthcheck

Check changelog 🚨

• https://github.com/ITISFoundation/osparc-simcore/pull/3462 should NOT get in this release
• changelogs
  • https://github.com/ITISFoundation/osparc-simcore/releases/tag/staging_switzer1
  • https://github.com/ITISFoundation/osparc-simcore/releases/tag/staging_switzer2
  • https://github.com/ITISFoundation/osparc-simcore/releases/tag/staging_switzer3

Check devops ⚠️

Test assessment: e2e-testing

• Tue Oct25: AWS staging was overloaded. @mrnicegyu11 sets memory and cpu resource limits (and identical reservations) to dask-scheduler and observe improvement. These limits are implemented in master and in staging for staging_switzer2
• Wed Oct26: AWS staging did not improve. As soon as dask-scheduler peaks CPU, services become very slow. In particular the migration service healthcheck times-outs and is restarted. The healthcheck script as well as pyinstruments logs in the webserver indicate that the code that talks with postgres gets really slow!
• Thr. Oct27: Issue in AWS staging is that we run out of credits: changed migration service healthcheck and hotfixed to staging. Waiting to see results tomorrow.

Test assessment: targeted-testing ✅

• [x] @elisabettai ✅
• [x] @pcrespov ✅

Test assessment: user-testing

• @newton1985 @elisabettai @pcrespov on Fri from 15h-16h Went pretty well. TN proposed several features that will be converted into cases. @newton1985 ✅

Release summary

• what: make release-prod version=1.38.0 git_sha=d9c9ce467944d400e4fd7b6a4564d3e2134fa369
• who: @mrnicegyu11 @pcrespov
• when: MONDAY Oct.31

Release assessment

[pcrespov]

🚑️ hotfix v1.38.1 [FAILED]

• 🐛 Storage: crash when not a file in the project (#3483)

Assessment

• Deploy failed because dask-sidecar image was not in docker-hub

  
  ERROR: for dask-scheduler  manifest for ***/dask-sidecar:hotfix-github-2022-11-01--14-20.ad8b68412f530b8f48a57099e0cdfb8e074e0892 not found: manifest unknown: manifest unknown

ERROR: for dask-sidecar manifest for /dask-sidecar:hotfix-github-2022-11-01--14-20.ad8b68412f530b8f48a57099e0cdfb8e074e0892 not found: manifest unknown: manifest unknown manifest for /dask-sidecar:hotfix-github-2022-11-01--14-20.ad8b68412f530b8f48a57099e0cdfb8e074e0892 not found: manifest unknown: manifest unknown manifest for ***/dask-sidecar:hotfix-github-2022-11-01--14-20.ad8b68412f530b8f48a57099e0cdfb8e074e0892 not found: manifest unknown: manifest unknown

- Removed ``dask-scheduler`` from ``services/docker-compose-deploy.yml`` and re triggered full CI
- TODO: new hotfix required for tomorrow **v1.38.2**

[mrnicegyu11]

Aftermath / Summary Release v1.38.0

• Lengthy dy-sidecar related cleanup necessary: User from Pittsburgh had a large study open. Some dy-services (from e2e/p2e in all likelyhood) could not be deleted due to this issue. Some orphaned containers without sidecars where present on all nodes.
• Weird finding: Potentially legacy, but likely dy-services are started as docker service, e.g. jupyter-neuron and an old s4l version. They had a jovyan user inside. They where orphaned / abandonned.
• Deployment agent shows new issue where when only selected files are pulled the git-repo becomes dirty and the git-repo-watching of the deployment agent failed. This is noticable in a spam flood of osparc-deploy mattermost notifications, mentioning that git pull failed. SSHing into the deployment-agent container, cd /tmp/* and running git reset --hard HEAD fixed the problem on aws-prod, but only after a new change in the watched git repos is detected. The reason for this bug is at this point entirely unclear. This warrants a bug in the deployment agent repo [TODO]
• Osparc-Simcore release went rather smooth on all plattforms (dalco, aws, tip.itis.swiss)
• Maintenance Ticket 325 on gitlab was done
• Maintenance Ticket 242 on gitlab was done
• Maintenance Ticket 335 on gitlab was done
• Maintenance Ticket 318 on gitlab was done
• Maintenance Ticket 329 on gitlab was finished except for dalco11, which is blocked by a user having an open s4l on staging.
• Cron jobs for restarting the dask-sidecar where removed
• [ ] It still needs to be checked if https://github.com/ITISFoundation/osparc-simcore/pull/3384 is resolved, traefik-simcore was disabled as requested.

[pcrespov]

🚑️ hotfix v1.38.2

After some changes in the github actions config and removing dask-scheduler from docker-compose-build, it released correctly.