add some changes to vcs

mandresm commented 4 months ago

VCS fixes and improvements.

TODO

[ ] Parallelization of the gitdiff for thrads in a node)
[x] Hashing of binaries to avoid doing a gitdiff all the time (only possible for codes that actually compile, in the future there might be some codes that have no binary, but instead they are python, that's why I think parallelization of git diff is still useful) -> see the chat in Chatgpt3: https://chatgpt.com/share/5386c188-2a7e-499c-a9ea-54c351e2af0d
[x] Testing with the virtual environment activated, there were some problems in the past with it
[ ] Clean the print statements I added

For this PR I will not implement the checking functionality, only the writing of the git status and diff file.

mandresm commented 4 months ago

@pgierz, I'm putting this into clickup and assign it to you, if you don't have time for it we can reassign it to me in some point.

pgierz commented 3 months ago

Hey Miguel, I have something that might be able to handle caching very easily. Have a look at the newest commit.

pgierz commented 3 months ago

Cache seems to be working now as I would expect:

2024-08-07 10:10:03.938 | CRITICAL | esm_runscripts.prepare:add_vcs_info:700 - Experiment information is being stored for usage under:
2024-08-07 10:10:03.940 | CRITICAL | esm_runscripts.prepare:add_vcs_info:701 - >>> /albedo/work/user/pgierz/SciComp/Projects/esm-tools-vcs/experiments/test-007/run_18500201-18500228/log//test-007_vcs_info.yaml
2024-08-07 10:10:03.940 | CRITICAL | esm_runscripts.prepare:add_vcs_info:703 - >>> CACHE FILE EXISTS!
2024-08-07 10:10:03.940 | CRITICAL | esm_runscripts.prepare:add_vcs_info:704 - >>> Last modification time of cache: 1723017728.2125185
2024-08-07 10:10:03.941 | CRITICAL | esm_runscripts.prepare:add_vcs_info:713 - Locating echam
2024-08-07 10:10:03.952 | DEBUG    | esm_runscripts.helpers:is_older_than:477 - Checking modification times:
2024-08-07 10:10:03.952 | DEBUG    | esm_runscripts.helpers:is_older_than:478 - /albedo/work/user/pgierz/SciComp/Projects/esm-tools-vcs/experiments/test-007/log//test-007_vcs_info.yaml: 1723017728.2125185
2024-08-07 10:10:03.952 | DEBUG    | esm_runscripts.helpers:is_older_than:479 - /albedo/work/user/pgierz/SciComp/Projects/esm-tools-vcs/model_codes/awiesm-2.1/echam-6.3.05p2: 1722867496.347289
2024-08-07 10:10:03.952 | CRITICAL | esm_runscripts.prepare:add_vcs_info:722 - Using cached VCS info for echam

pgierz commented 2 months ago

I added an Audit Log, which has all possible info you could possibly wish for to check modification times, access times, and checksums. See in src/esm_runscripts/checksum_helpers.py

This, together with the git stuff, should give all possible information someone would want if their model changed halfway during the experiment.

pgierz commented 2 months ago

Only thing left to do is parallelising the git diff and checksum parts.

esm-tools / esm_tools

add some changes to vcs #1203