unity-sds / unity-cs

Unity Common Services
Apache License 2.0
0 stars 2 forks source link

Fix nightly cron jobs #357

Open galenatjpl opened 6 months ago

hargitayjpl commented 6 months ago
# m h  dom mon dow   command

0 18 * * * cd /home/ubuntu/unity-cs-infra/nightly_tests && ./run.sh --destroy true --run-tests true --project-name unity --venue-name nightly >> /home/ubuntu/nightly_run_`date +\%F_\%H-\%M`.log 2>&1

The cron job expression has been updated, the nighlty will run every day at 6pm

hargitayjpl commented 6 months ago
[@Joshua D Rodriguez](https://jpl.slack.com/team/U4TRSG66M)
, thanks again for creating the new MC release!  I think perhaps Tom pushed a different version that contained a reference to this commit:
https://github.com/unity-sds/unity-proxy/commit/59ec1f65c4227a56c9996714e2292887f99254df
(unity-proxy)
The reason why I think this, is that the error stacktrace above (which I believe is running the latest MC release), mentions EFSAccessPolicy, and the commit above would have prefixed that string like ${var.deployment_name}-EFSAccessPolicy .
I’m wondering if this line:
https://github.com/unity-sds/unity-management-console/blob/86a92cfe8691a0f86c62c0e34ace7c8a54dc2ce6/backend/internal/processes/bootstrap.go#L178
needs to be bumped up to 0.12 ?

The MC failed to spin up do to the above issue. Bumping the version to 0.12 fixed the issue.

hargitayjpl commented 6 months ago

PATH=/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

PATH variable needed to be set in crontab since cron uses a minimal environment, the cron job was failing to find the aws python etc command cr

hargitayjpl commented 6 months ago

The script set_common_ssm_params.sh uses the variable names prefixed with /unity/cs/..., but in AWS Parameter Store, they are actually named with the prefix /unity/ci/.... This mismatch in naming conventions is causing issues. Justed the name of the affected paramters to follow /unity/cs/.... Script now does not error on this issue,

hargitayjpl commented 6 months ago
ubuntu@ip-10-5-29-201:~/unity-cs-infra$ git pull
remote: Enumerating objects: 33, done.
remote: Counting objects: 100% (33/33), done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 33 (delta 21), reused 26 (delta 15), pack-reused 0
error: insufficient permission for adding an object to repository database .git/objects
fatal: failed to write object
fatal: unpack-objects failed

The cronjob used to run the script with sudo privilege's, I believe this messed up the permissions for the .git directory. sudo chown -R ubuntu:ubuntu .git sudo chmod -R u+rwX .git

The above command fixed the permission issues

hargitayjpl commented 6 months ago

The nightly output didn't make it to Slack because the set_common_ssm_params.sh script overwrites the Slack URL

Link to PR that addresses this issue: https://github.com/unity-sds/unity-cs-infra/pull/69

hargitayjpl commented 5 months ago

The nightly now runs without issue in the new bastion host on Unity-CM

galenatjpl commented 5 months ago

fixed