search

google-deepmind / launchpad

Apache License 2.0
309 stars 35 forks source link

Not able to save logs after launchpad.launch #24

Open kmukeshreddy opened 2 years ago

kmukeshreddy commented 2 years ago

I'm not able to save the logs after the termination of the launchpad in .py file. Could you please suggest a solution? and is there any better way to save the logs?

Code:

lp_return = lp.launch(program, launch_type=lp.LaunchType.LOCAL_MULTI_PROCESSING, terminal="current_terminal") lp_return.wait() print('-- acme saved') shutil.make_archive('/content/logs', 'zip', '/root/acme')

log from terminal: [counter/0] I0503 16:58:32.552900 139934399387520 savers.py:207] Caught SIGTERM: forcing a checkpoint save. [counter/0] I0503 16:58:32.553121 139934399387520 savers.py:156] Saving checkpoint: /root/acme/2c71038a-cb02-11ec-9553-0242ac1c0002/checkpoints/counter [reverb/cc/platform/default/server.cc:84] Shutting down replay server Killing entire runtime. Killed /content/gdrive/MyDrive/code/

qstanczyk commented 2 years ago

Killing entire runtime suggests that Program didn't terminate cleanly by itself and Launchpad had to kill it. That results in the code after lp_return.wait() to not be executed. I would suggest investigating why the program doesn't terminate by itself within a timeout