Profiling with PPE for Local

[kuenishi]

PR draft to discuss the base design for https://github.com/pfnet/pfio/issues/258.

With a test script like below:

import pytorch_pfn_extras as ppe

from pfio.v2 import Local, Path

@ppe.profiler.record_function("my_tag_2")
def my_run2(x):
    pass

def my_run1(x):
    pass

def p():
    with ppe.profiler.record("my_tag_1"):
        my_run1('hoge')

    for i in range(10):
        my_run2("far")

    total = 0
    count = 0

    p = Path()
    for f in p.glob("*"):
        s = f.stat()
        total += s.size
        count += 1

    d = 0
    with Local() as fs:
        for f in fs.list():
            if fs.isdir(f.strip()):
                d += 1
                continue

            len(fs.open(f).read())

    print(count - d, "files", d, "dirs", total, "bytes for files")

    ts = ppe.profiler.get_time_summary()

    with ts.summary() as o:
        print(o)

p()

It yields a result like this:

$ python p.py
15 files 13 dirs 8490 bytes for files
(<pytorch_pfn_extras.reporting.DictSummary object at 0x7fae6b3204d0>, {'my_tag_1.min': 9.5367431640625e-06, 'my_tag_1
.max': 9.5367431640625e-06, 'my_tag_2.min': 9.5367431640625e-07, 'my_tag_2.max': 2.86102294921875e-06, 'local-stat.mi
n': 9.059906005859375e-06, 'local-stat.max': 4.9591064453125e-05, 'local-open.min': 1.3828277587890625e-05, 'local-op
en.max': 6.914138793945312e-05, 'local-list.min': 6.9141387939453125e-06, 'local-list.max': 0.00014781951904296875})

[kuenishi]

Latest:

$ python p.py
15 files 13 dirs 8491 bytes for files
(<pytorch_pfn_extras.reporting.DictSummary object at 0x7f5bbc7207d0>, {'my_tag_1.min': 1.0967254638671875e-05, 'my_ta
g_1.max': 1.0967254638671875e-05, 'my_tag_2.min': 7.152557373046875e-07, 'my_tag_2.max': 3.0994415283203125e-06, 'pfi
o.v2.Local:glob.min': 0.0004813671112060547, 'pfio.v2.Local:glob.max': 0.0004813671112060547, 'pfio.v2.Local:stat.min
': 8.106231689453125e-06, 'pfio.v2.Local:stat.max': 0.00011372566223144531, 'pfio.v2.Local:open.min': 1.3113021850585
938e-05, 'pfio.v2.Local:open.max': 6.628036499023438e-05, 'pfio.v2.Local:list.min': 8.344650268554688e-06, 'pfio.v2.L
ocal:list.max': 0.0001614093780517578})

[kuenishi]

Superseded by #340 .