kube-HPC / hkube

🐟 High Performance Computing over Kubernetes - Core Repo 🎣
http://hkube.io
MIT License
305 stars 20 forks source link

provide an alternative solution to the usage of tee command #1297

Closed mattanf2 closed 1 year ago

mattanf2 commented 3 years ago

We were previously advised that when calling the python program inside a container. we should call it with a line such as: python -u main.py 2>&1 | tee /hkube-logs/stdout.log

However, it turns out that if we add the "tee /hkube-logs/stdout.log" section to the script and main.py crashes. hkube will not detect the crash and will run the algorithm either forever or untill ttl is reached.

"tee /hkube-logs/stdout.log" section is therefor unusable. I can add code to write logs to the "/hkube-logs/stdout.log" file. However the last time I checked. hkube_python_wrapper uses a lot of print() commands (instead of logging using a logger) that would not be captured this way.

Please advise on how to proceed.

yehiyam commented 2 years ago

example algorithm in python that should replicate that:

import ctypes
import time 
def start(args, hkubeapi):
    time.sleep(1)
    print('sending ctype error')
    data = ctypes.c_char_p(-1).value #<- I think you only need this
    return "hello"