feat: updates to versions, devcontainer, and bugfixes

[devlace]

Type of PR

• Code changes

Purpose

• update spark cluster versions
• bugfix in util due to spark version upgrade
• update devcontainer
• update opencensus versions
• improve deployment script by specifying principaltype

Does this introduce a breaking change? If yes, details on what can break

Author pre-publish checklist

• [X] Added test to prove my fix is effective or new feature works
• [X] No PII in logs
• [X] Made corresponding changes to the documentation

Validation steps

• ...

Issues Closed or Referenced

• Closes #issue_number
• References #issue_number

[devlace]

[promisinganuj]

LGTM!