search

blue-oil / blueoil

Bring Deep Learning to small devices
https://blueoil.org
Apache License 2.0
248 stars 86 forks source link

DLK tests for segmentation fail on Arm with hard_quantize=False #512

Closed hadusam closed 4 years ago

hadusam commented 5 years ago

On #511, I tried to add DLK tests for the segmentation task on Arm, but the tests failed when hard_quantize=False.

Is it correct behavior or not? In Classification and Object Detection, hard_quantize=False is tested and passed, so I think this is a bug.

hadusam commented 5 years ago

I found that Out of memory occurred on Arm device when running generated library.

How to reproduce this error

### Generate library file for Arm with hard_quantize=False
# docker run --rm -it -v $(pwd)/tmp:/output blueoil/blueoil:latest python dlk/python/dlk/scripts/generate_project.py -i dlk/examples/segmentation/lm_segnet_v1_quantize_camvid/minimal_graph_with_shape.pb -o /output
# docker run --rm -it -v $(pwd)/tmp:/output blueoil/blueoil:latest bash -c "cd /output/None.prj && cmake -DTOOLCHAIN_NAME=linux_arm -DUSE_NEON=1 . && make lib -j8"
# ls -l tmp/None.prj/*.so
-rwxr-xr-x 1 root root 10341428 10月  8 12:12 tmp/None.prj/lib_arm.so

Put that library and some files to Arm machine.

And run the following commands.

# cd [your work dir]
# ls
lib_arm.so  network_input_output.tar.gz
# tar xvf network_input_output.tar.gz
# python
>>> import numpy as np
>>> from nnlib import NNLib as NNLib
>>> 
>>> nn = NNLib()
>>> nn.load("./lib_arm.so")
True
>>> nn.init()
True
>>> 
>>> proc_input = np.load("./network_input_output/000_images_placeholder:0.npy")
>>> batched_proc_input = np.expand_dims(proc_input, axis=0)
>>> nn.run(batched_proc_input)
Killed

I found that Out of memory occurred (/var/log/syslog)

Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.665832] python invoked oom-killer: gfp_mask=0x26084c0, order=0, oom_score_adj=0
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.665848] python cpuset=/ mems_allowed=0
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.665871] CPU: 1 PID: 2087 Comm: python Tainted: G           O    4.5.0 #1
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.665878] Hardware name: Altera SOCFPGA
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.665921] [<c0018b1c>] (unwind_backtrace) from [<c00144ec>] (show_stack+0x20/0x24)
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.665941] [<c00144ec>] (show_stack) from [<c02f1724>] (dump_stack+0x90/0xa4)
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.665962] [<c02f1724>] (dump_stack) from [<c01343b0>] (dump_header.constprop.7+0x5c/0x16c)
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.665989] [<c01343b0>] (dump_header.constprop.7) from [<c00e985c>] (oom_kill_process+0x2a0/0x4c4)
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666006] [<c00e985c>] (oom_kill_process) from [<c00e9d9c>] (out_of_memory+0x2a8/0x340)
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666024] [<c00e9d9c>] (out_of_memory) from [<c00ef0e4>] (__alloc_pages_nodemask+0x950/0xb88)
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666042] [<c00ef0e4>] (__alloc_pages_nodemask) from [<c01110f8>] (__pte_alloc+0x34/0x158)
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666058] [<c01110f8>] (__pte_alloc) from [<c0113f20>] (handle_mm_fault+0x838/0xe40)
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666074] [<c0113f20>] (handle_mm_fault) from [<c001e6f8>] (do_page_fault+0x148/0x3b0)
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666088] [<c001e6f8>] (do_page_fault) from [<c001eaa4>] (do_translation_fault+0x88/0xc0)
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666102] [<c001eaa4>] (do_translation_fault) from [<c000929c>] (do_DataAbort+0x4c/0xcc)
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666115] [<c000929c>] (do_DataAbort) from [<c0015340>] (__dabt_usr+0x40/0x60)
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666123] Exception stack(0xd60ebfb0 to 0xd60ebff8)
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666134] bfa0:                                     00000480 01c7ad80 b0a00bf8 00006543
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666146] bfc0: bef176d8 bef176d8 00000004 bef17600 00000003 bef177d4 a68fc008 00000006
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666157] bfe0: 00000004 bef171f8 00000004 a8e38912 000f0030 ffffffff
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666164] Mem-Info:
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666184] active_anon:93913 inactive_anon:2410 isolated_anon:0
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666184]  active_file:6 inactive_file:25 isolated_file:0
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666184]  unevictable:0 dirty:2 writeback:0 unstable:0
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666184]  slab_reclaimable:1172 slab_unreclaimable:1444
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666184]  mapped:187 shmem:2459 pagetables:324 bounce:0
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666184]  free:10596 free_pcp:60 free_cma:9924
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666223] Normal free:42384kB min:2848kB low:3560kB high:4272kB active_anon:375652kB inactive_anon:9640kB active_file:76kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):100kB present:524288kB managed:508576kB mlocked:0kB dirty:8kB writeback:0kB mapped:748kB shmem:9836kB slab_reclaimable:4688kB slab_unreclaimable:5776kB kernel_stack:624kB pagetables:1296kB unstable:0kB bounce:0kB free_pcp:240kB local_pcp:0kB free_cma:39696kB writeback_tmp:0kB pages_scanned:1404 all_unreclaimable? yes
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666231] lowmem_reserve[]: 0 0
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666241] Normal: 96*4kB (MEC) 31*8kB (MEC) 41*16kB (UEC) 20*32kB (UME) 7*64kB (UEC) 3*128kB (UC) 1*256kB (U) 1*512kB (C) 0*1024kB 1*2048kB (C) 9*4096kB (C) = 42440kB
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666296] 2504 total pagecache pages
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666305] 0 pages in swap cache
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666312] Swap cache stats: add 0, delete 0, find 0/0
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666318] Free swap  = 0kB
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666323] Total swap = 0kB
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666328] 131072 pages RAM
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666333] 0 pages HighMem/MovableOnly
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666338] 3928 pages reserved
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666343] 32768 pages cma reserved
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666350] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666366] [  830]     0   830     1686      262       6       0        0             0 systemd-journal
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666378] [ 1302]     0  1302     2628      106       6       0        0         -1000 systemd-udevd
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666388] [ 1354]   100  1354     3062       58       5       0        0             0 systemd-timesyn
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666399] [ 1498]     0  1498      832       63       5       0        0             0 systemd-logind
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666409] [ 1500]   107  1500     1251      114       6       0        0          -900 dbus-daemon
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666421] [ 1502]     0  1502    18219      387      18       0        0             0 NetworkManager
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666431] [ 1503]   106  1503     7844      172       8       0        0             0 rsyslogd
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666441] [ 1554]     0  1554     9122      167      11       0        0             0 polkitd
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666451] [ 1625]     0  1625      992      153       4       0        0             0 dhclient
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666462] [ 1677]     0  1677     1995      128       7       0        0         -1000 sshd
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666473] [ 1683]   999  1683   204251     1315      17       0        0             0 buildkite-agent
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666484] [ 1689]     0  1689     1010       31       5       0        0             0 agetty
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666494] [ 1736]     0  1736     5644     2518      14       0        0             0 python3
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666504] [ 1791]     0  1791      452       29       3       0        0             0 agetty
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666515] [ 2021]     0  2021     2976      186       9       0        0             0 sshd
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666525] [ 2027]     0  2027     1205      124       6       0        0             0 systemd
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666535] [ 2030]     0  2030     6004      273       8       0        0             0 (sd-pam)
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666545] [ 2041]     0  2041     1873      335       6       0        0             0 bash
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666555] [ 2087]     0  2087   116874    87551     190       0        0             0 python
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.666563] Out of memory: Kill process 2087 (python) score 669 or sacrifice child
Oct  8 03:43:20 blueoil-de10nano-staging-01 kernel: [ 1239.674114] Killed process 2087 (python) total-vm:467496kB, anon-rss:350204kB, file-rss:0kB, shmem-rss:0kB
tkng commented 5 years ago

maybe related: #473 #476

primenumber commented 5 years ago

kn2row buffer requires ~200MB memory... We must resolve #476 .