strint
commented
1 year ago @MirrorCY
可以更新下 _cost_cnt，新增了 host mem 统计：
def _cost_cnt(fn):
def new_fn(*args, **kwargs):
print("==> function ", fn.__name__, " try to run...")
flow._oneflow_internal.eager.Sync()
before_used = flow._oneflow_internal.GetCUDAMemoryUsed()
print(fn.__name__, " cuda mem before ", before_used, " MB")
before_host_used = flow._oneflow_internal.GetCPUMemoryUsed()
print(fn.__name__, " host mem before ", before_host_used, " MB")
start_time = time.time()
out = fn(*args, **kwargs)
flow._oneflow_internal.eager.Sync()
end_time = time.time()
print(fn.__name__, " run time ", end_time - start_time, " seconds")
after_used = flow._oneflow_internal.GetCUDAMemoryUsed()
print(fn.__name__, " cuda mem after ", after_used, " MB")
print(fn.__name__, " cuda mem diff ", after_used - before_used, " MB")
after_host_used = flow._oneflow_internal.GetCPUMemoryUsed()
print(fn.__name__, " host mem after ", after_host_used, " MB")
print(fn.__name__, " host mem diff ", after_host_used - before_host_used, " MB")
print("<== function ", fn.__name__, " finish run.")
print("")
return out
return new_fn这是我在 save 阶段的日志
==> Try to run graph save...
==> function get_pipe try to run...
get_pipe cuda mem before 652.0 MB
get_pipe host mem before 2163.0 MB
Fetching 12 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 18606.89it/s]
get_pipe run time 34.62615132331848 seconds
get_pipe cuda mem after 652.0 MB
get_pipe cuda mem diff 0.0 MB
get_pipe host mem after 8467.0 MB
get_pipe host mem diff 6304.0 MB
<== function get_pipe finish run.
==> function pipe_to_cuda try to run...
pipe_to_cuda cuda mem before 652.0 MB
pipe_to_cuda host mem before 8467.0 MB
pipe_to_cuda run time 1.7569012641906738 seconds
pipe_to_cuda cuda mem after 3770.0 MB
pipe_to_cuda cuda mem diff 3118.0 MB
pipe_to_cuda host mem after 9470.0 MB
pipe_to_cuda host mem diff 1003.0 MB
<== function pipe_to_cuda finish run.
==> function config_graph try to run...
config_graph cuda mem before 3770.0 MB
config_graph host mem before 9470.0 MB
config_graph run time 6.890296936035156e-05 seconds
config_graph cuda mem after 3770.0 MB
config_graph cuda mem diff 0.0 MB
config_graph host mem after 9470.0 MB
config_graph host mem diff 0.0 MB
<== function config_graph finish run.
sd init time 36.38862633705139 s.
==> function text_to_image try to run...
text_to_image cuda mem before 3770.0 MB
text_to_image host mem before 9470.0 MB
100%|█████| 50/50 [00:13<00:00, 3.62it/s]
W20230210 01:54:00.468230 291033 cudnn_conv_util.cpp:102] Currently available alogrithm (algo=1, require memory=7472256, idx=1) meeting requirments (max_workspace_size=1073741824, determinism=0) is not fastest. Fastest algorithm (1) requires memory 1074922512
text_to_image run time 15.075793743133545 seconds
text_to_image cuda mem after 7854.0 MB
text_to_image cuda mem diff 4084.0 MB
text_to_image host mem after 10003.0 MB
text_to_image host mem diff 533.0 MB
<== function text_to_image finish run.
==> function text_to_image try to run...
text_to_image cuda mem before 7854.0 MB
text_to_image host mem before 10003.0 MB
/home/xuxiaoyu/dev/oneflow/python/oneflow/nn/modules/module.py:152: UserWarning: Interpolate() is called in a nn.Graph, but not registered into a nn.Graph.
warnings.warn(
100%|█████| 50/50 [00:13<00:00, 3.82it/s]
text_to_image run time 19.789249181747437 seconds
text_to_image cuda mem after 8202.0 MB
text_to_image cuda mem diff 348.0 MB
text_to_image host mem after 7869.0 MB
text_to_image host mem diff -2134.0 MB
<== function text_to_image finish run.
====> diff 0.0013520131
st init and run time 71.27231931686401 s.
==> function save_pipe_sch try to run...
save_pipe_sch cuda mem before 8202.0 MB
save_pipe_sch host mem before 7869.0 MB
save_pipe_sch run time 6.8812150955200195 seconds
save_pipe_sch cuda mem after 8204.0 MB
save_pipe_sch cuda mem diff 2.0 MB
save_pipe_sch host mem after 9686.0 MB
save_pipe_sch host mem diff 1817.0 MB
<== function save_pipe_sch finish run.
==> function save_graph try to run...
save_graph cuda mem before 8204.0 MB
save_graph host mem before 9686.0 MB
save_graph run time 3.215217351913452 seconds
save_graph cuda mem after 8206.0 MB
save_graph cuda mem diff 2.0 MB
save_graph host mem after 9715.0 MB
save_graph host mem diff 29.0 MB
<== function save_graph finish run.
MirrorCY
The running environment is wsl2 Ubuntu 20.04, neither the host nor wsl2 is running any other CUDA programs.
Originally posted by @MirrorCY in https://github.com/Oneflow-Inc/diffusers/issues/75#issuecomment-1424482749