search

RedisGears / EdgeRealtimeVideoAnalytics

An example of using Redis Streams, RedisGears, RedisAI and RedisTimeSeries for Realtime Video Analytics (i.e. counting people)
https://redisgears.io
Apache License 2.0
397 stars 69 forks source link

Backend TF not loaded #12

Open Akhtar303 opened 5 years ago

Akhtar303 commented 5 years ago

Hi @itamarhaber @MeirShpilraien I have machine which has no SSE4.1 SSE4.2 AVX AVX2 that's why Backend TF not loaded. go to gear environment and import tensorflow but not import Tensorflow succesfully i.e (python3) onstak@onstak:/opt/redislabs/lib/modules/python3$

import tensorflow Illegal instruction (core dumped) and when I ran init.py tensorflow not load successfully but torch load successfully Error

Backend TF not loaded, will try loading default backend /home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/backends/redisai_tensorflow/lib/libtensorflow_framework.so.1(+0x725a59)[0x7fcfdcf4fa59] Symbol: (null) (base: (nil)) Module: /home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/backends/redisai_tensorflow/lib/libtensorflow_framework.so.1 (base 0x7fcfdc82a000) $ xxd -r -p /tmp/dump.hex /tmp/dump.bin

but torch load successfully TORCH backend loaded from /home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/backends/redisai_torch/redisai_torch.so I am also try this one go to /opt/redislabs/lib/modules/python3$ and install tesorflow like pipenv install tensorflow or

sudo pipenv install tensorflow-1.14.0-cp37-cp37m-linux_x86_64.whl but facing same core dump error so kindly tell me how to solve this problem. Thanks

Akhtar303 commented 5 years ago

@itamarhaber Thanks for reply I am working on EdgeRealtimeVideoAnalytics link = https://github.com/RedisGears/EdgeRealtimeVideoAnalytics

when start redis with redisgear there is no error but when I ran python init.py then Tensorflow didn't load successfully but pytorch load successfully Eroor 80544:M 09 Oct 2019 08:40:50.269 # Backend TF not loaded, will try loading default backend**** redis broke with this error

tting after a reboot. Redis must be restarted after THP is disabled. 80544:M 09 Oct 2019 08:39:51.745 Module 'ai' loaded from /home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/redisai.so 80544:M 09 Oct 2019 08:39:51.844 RedisGears version 99.99.99, git_sha=5beab40cb5e51e66a4b08cf2cabaf9155e9d8f89 80544:M 09 Oct 2019 08:39:51.844 PythonHomeDir:/opt/redislabs/lib/modules/python3/ 80544:M 09 Oct 2019 08:39:51.844 MaxExecutions:1000 80544:M 09 Oct 2019 08:39:51.844 ProfileExecutions:0 80544:M 09 Oct 2019 08:39:51.844 PythonAttemptTraceback:1 80544:M 09 Oct 2019 08:39:51.844 RedisAI api loaded successfully. could not initialize RediSearch_CheckApiVersionCompatibility 80544:M 09 Oct 2019 08:39:51.844 # could not initialize RediSearch api, running without Search support. 80544:M 09 Oct 2019 08:39:51.982 Initializing Python environment with: exec(open('/opt/redislabs/lib/modules/python3/.venv/bin/activate_this.py').read(), {'file': '/opt/redislabs/lib/modules/python3/.venv/bin/activate_this.py'}) 80544:M 09 Oct 2019 08:39:52.256 Module 'rg' loaded from /home/onstak/EdgeRealtimeVideoAnalytics/RedisGears/redisgears.so 80544:M 09 Oct 2019 08:39:52.256 Ready to accept connections 80544:M 09 Oct 2019 08:40:45.719 # Backend TORCH not loaded, will try loading default backend

80544:M 09 Oct 2019 08:40:46.930 * TORCH backend loaded from /home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/backends/redisai_torch/redisai_torch.so 80544:M 09 Oct 2019 08:40:50.269 # Backend TF not loaded, will try loading default backend

=== REDIS BUG REPORT START: Cut & paste starting from here === 80544:M 09 Oct 2019 08:40:50.828 # Redis 5.0.6 crashed by signal: 4 80544:M 09 Oct 2019 08:40:50.828 # Crashed running the instruction at: 0x7ff15bb4fa59 80544:M 09 Oct 2019 08:40:50.828 # Failed assertion: (:0)

------ STACK TRACE ------ EIP: /home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/backends/redisai_tensorflow/lib/libtensorflow_framework.so.1(+0x725a59)[0x7ff15bb4fa59]

Backtrace: redis-server :6379(logStackTrace+0x5a)[0x56095f9424da] redis-server :6379(sigsegvHandler+0xb1)[0x56095f942c91] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7ff1a7df2890] /home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/backends/redisai_tensorflow/lib/libtensorflow_framework.so.1(+0x725a59)[0x7ff15bb4fa59] /lib64/ld-linux-x86-64.so.2(+0x10733)[0x7ff1a87b9733] /lib64/ld-linux-x86-64.so.2(+0x151ff)[0x7ff1a87be1ff] /lib/x86_64-linux-gnu/libc.so.6(_dl_catch_exception+0x6f)[0x7ff1a7b562df] /lib64/ld-linux-x86-64.so.2(+0x147ca)[0x7ff1a87bd7ca] /lib/x86_64-linux-gnu/libdl.so.2(+0xf96)[0x7ff1a8207f96] /lib/x86_64-linux-gnu/libc.so.6(_dl_catch_exception+0x6f)[0x7ff1a7b562df] /lib/x86_64-linux-gnu/libc.so.6(_dl_catch_error+0x2f)[0x7ff1a7b5636f] /lib/x86_64-linux-gnu/libdl.so.2(+0x1735)[0x7ff1a8208735] /lib/x86_64-linux-gnu/libdl.so.2(dlopen+0x71)[0x7ff1a8208051] /home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/redisai.so(RAI_LoadBackend_TensorFlow+0x60)[0x7ff1a55ee0c0] /home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/redisai.so(RedisAI_ModelSet_RedisCommand+0x369)[0x7ff1a55eba59] redis-server :6379(RedisModuleCommandDispatcher+0x68)[0x56095f9702a8] redis-server :6379(call+0xb5)[0x56095f8fbc35] redis-server :6379(processCommand+0x36f)[0x56095f8fc33f] redis-server :6379(processInputBuffer+0x171)[0x56095f90ce31] redis-server :6379(aeProcessEvents+0x150)[0x56095f8f5670] redis-server :6379(aeMain+0x2b)[0x56095f8f5a5b] redis-server :6379(main+0x4f4)[0x56095f8f2604] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7ff1a7a10b97] redis-server :6379(_start+0x2a)[0x56095f8f285a]

------ INFO OUTPUT ------

Server

redis_version:5.0.6 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:9260170b247e88b redis_mode:standalone os:Linux 4.15.0-65-generic x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:atomic-builtin gcc_version:7.4.0 process_id:80544 run_id:972bf07279b81aac3e38a2dba32475cf2e576eba tcp_port:6379 uptime_in_seconds:59 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:10329362 executable:/home/onstak/EdgeRealtimeVideoAnalytics/redis-server config_file:/home/onstak/EdgeRealtimeVideoAnalytics/redisedge/redisedge.conf

Clients

connected_clients:2 client_recent_max_input_buffer:198311946 client_recent_max_output_buffer:0 blocked_clients:0

Memory

used_memory:544659192 used_memory_human:519.43M used_memory_rss:258641920 used_memory_rss_human:246.66M used_memory_peak:544659192 used_memory_peak_human:519.43M used_memory_peak_perc:197.18% used_memory_overhead:250158447 used_memory_startup:791728 used_memory_dataset:294500745 used_memory_dataset_perc:54.15% allocator_allocated:209405176 allocator_active:209842176 allocator_resident:284135424 total_system_memory:16795607040 total_system_memory_human:15.64G used_memory_lua:37888 used_memory_lua_human:37.00K used_memory_scripts:0 used_memory_scripts_human:0B number_of_cached_scripts:0 maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction allocator_frag_ratio:1.00 allocator_frag_bytes:437000 allocator_rss_ratio:1.35 allocator_rss_bytes:74293248 rss_overhead_ratio:0.91 rss_overhead_bytes:-25493504 mem_fragmentation_ratio:1.24 mem_fragmentation_bytes:49529160 mem_not_counted_for_evict:0 mem_replication_backlog:0 mem_clients_slaves:0 mem_clients_normal:249366647 mem_aof_buffer:0 mem_allocator:jemalloc-5.1.0 active_defrag_running:0 lazyfree_pending_objects:0

Persistence

loading:0 rdb_changes_since_last_save:1 rdb_bgsave_in_progress:0 rdb_last_save_time:1570610391 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:-1 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:0 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0

Stats

total_connections_received:2 total_commands_processed:3 instantaneous_ops_per_sec:0 total_net_input_bytes:249333157 total_net_output_bytes:16 instantaneous_input_kbps:120600.10 instantaneous_output_kbps:0.00 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 evicted_keys:0 keyspace_hits:0 keyspace_misses:1 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:0 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0

Replication

role:master connected_slaves:0 master_replid:3c767078d46a203529e05eabbce964153e41b7eb master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0

CPU

used_cpu_sys:0.411274 used_cpu_user:0.518047 used_cpu_sys_children:0.000000 used_cpu_user_children:0.000000

Commandstats

cmdstat_ai.scriptset:calls=1,usec=1251631,usec_per_call=1251631.00 cmdstat_exists:calls=1,usec=2,usec_per_call=2.00 cmdstat_ping:calls=1,usec=2,usec_per_call=2.00

Cluster

cluster_enabled:0

Keyspace

db0:keys=1,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------ id=4 addr=127.0.0.1:46538 fd=13 name= age=5 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exists id=5 addr=[::1]:39200 fd=14 name= age=5 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=249332795 obl=0 oll=0 omem=0 events=r cmd=ai.modelset

------ CURRENT CLIENT INFO ------ id=5 addr=[::1]:39200 fd=14 name= age=5 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=249332795 obl=0 oll=0 omem=0 events=r cmd=ai.modelset argv[0]: 'AI.MODELSET' argv[1]: 'model' argv[2]: 'TF' argv[3]: 'CPU' argv[4]: 'INPUTS' argv[5]: 'input_1' argv[6]: 'input_image_shape' argv[7]: 'OUTPUTS' argv[8]: 'concat_11' argv[9]: 'concat_12' argv[10]: 'concat_13' argv[11]: ' [ input_1 Placeholder*

dtype06 shape-:+ ��������� ��������� ��������� � conv2d_1/kernelConst� �[��>�ҷ��ג���7��Z=��׾������=���>��q��T߾�>eO>�B��v?h�L�ϼ �=�&��#���?d�����9>ڒ4��H�w2>B���@ ���ƾ'�2��v9��>����^ܲ<��a�L��� ����>�9��>�$�=d�T��;ޖ���E>�>�����=�������<P��h���L>o>�ɔ��x�by �ڞ����.<�l�=-�����->�?���<X���=[��D� ?0V������o�>�$�>��=�^}=�;Q�|�;i�p�W��ؽ�8�����>6��>ovм�#=/�67�V�=Ā�=�e�>!���{�=M>�i��Pi�i����7>X��Zk/<���>�_̿,I��n�

�ܥ�ݿ�>��>��W��ܭ����=��M>���=�N">%�D��߭>���=�r�>����R�>V��LU?��|=�"���p�?›�=m�fh���>/���nݽ���Tt>L��=�F>�G@�\|X�n�����=L�¾�q ��nZ>��g>���P7># r���U=nT��o�h=P����A=</�>�Ǿ�[O��<b?�?��ս;����=�:�g%?2Ք?'�=;C3���b>���>��?4��|�R=O�%?&��>2�^��R�=Nl�=v<��>���>0���d̹���8�m�w?�a>~����_T=u6*�d�ý��>cu���-?R��`դ����<!c��T/�����q���AT��\@�<٦ �̥(>[ؽ8{⼳�X?�'

------ REGISTERS ------ 80544:M 09 Oct 2019 08:40:50.859 # RAX:0000560961ffe4e0 RBX:00007ffd733f0650 RCX:0000000000000000 RDX:00005609619e7010 RDI:00007ffd733f0630 RSI:00007ffd733f0540 RBP:00007ffd733f0710 RSP:00007ffd733f0600 R8 :0000560961f5b8b0 R9 :00005609620710f8 R10:0000000000000027 R11:0000000000000001 R12:00007ffd733f1108 R13:00005609619e7ec0 R14:00007ff15cd37390 R15:0000560962075230 RIP:00007ff15bb4fa59 EFL:0000000000010202 CSGSFS:002b000000000033 80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f060f) -> 0000000000000000 80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f060e) -> 00007ffd733f0730 80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f060d) -> 0000000000000000 80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f060c) -> 00007ff1a89b53e0 80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f060b) -> 00007ff1a7ddf8f8 80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f060a) -> 0000000000000348 80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0609) -> 00007ff1a87b5c09 80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0608) -> 0000000000000001 80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0607) -> 00007ff1a87b5af9 80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0606) -> 0000002b00000007 80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0605) -> 0000560962075230 80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0604) -> 00007ff165df7b90 80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0603) -> 0000000000000007 80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0602) -> 00007ffd733f0760 80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0601) -> 00007ff165dfac90 80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0600) -> 0000000000000000

------ FAST MEMORY TEST ------ 80544:M 09 Oct 2019 08:40:50.860 # Bio thread for job type #0 terminated 80544:M 09 Oct 2019 08:40:50.860 # Bio thread for job type #1 terminated 80544:M 09 Oct 2019 08:40:50.860 # Bio thread for job type #2 terminated Preparing to test memory region 56095fc6e000 (2252800 bytes) Preparing to test memory region 5609619e7000 (7188480 bytes) Preparing to test memory region 7ff15cdcb000 (69632 bytes) Preparing to test memory region 7ff165d7f000 (491520 bytes) Preparing to test memory region 7ff166000000 (408944640 bytes) Preparing to test memory region 7ff17e66d000 (335544320 bytes) Preparing to test memory region 7ff192e34000 (16384 bytes) Preparing to test memory region 7ff193082000 (8192 bytes) Preparing to test memory region 7ff19ef6c000 (278528 bytes) Preparing to test memory region 7ff19f6be000 (3670016 bytes) Preparing to test memory region 7ff1a0936000 (4096 bytes) Preparing to test memory region 7ff1a1a4c000 (12288 bytes) Preparing to test memory region 7ff1a207c000 (3145728 bytes) Preparing to test memory region 7ff1a2580000 (6815744 bytes) Preparing to test memory region 7ff1a2dd1000 (8388608 bytes) Preparing to test memory region 7ff1a35d2000 (8388608 bytes) Preparing to test memory region 7ff1a3dd3000 (8388608 bytes) Preparing to test memory region 7ff1a4dbc000 (139264 bytes) Preparing to test memory region 7ff1a4ddf000 (8388608 bytes) Preparing to test memory region 7ff1a57fe000 (8388608 bytes) Preparing to test memory region 7ff1a5fff000 (8388608 bytes) Preparing to test memory region 7ff1a6800000 (8388608 bytes) Preparing to test memory region 7ff1a7000000 (8388608 bytes) Preparing to test memory region 7ff1a7ddc000 (16384 bytes) Preparing to test memory region 7ff1a7ffb000 (16384 bytes) Preparing to test memory region 7ff1a89ae000 (32768 bytes) *** Preparing to test memory region 7ff1a89d2000 (4096 bytes) .O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------ Symbol: (null) (base: (nil)) Module: /home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/backends/redisai_tensorflow/lib/libtensorflow_framework.so.1 (base 0x7ff15b42a000) $ xxd -r -p /tmp/dump.hex /tmp/dump.bin $ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

   Please report the crash by opening an issue on github:

       http://github.com/antirez/redis/issues

Suspect RAM error? Use redis-server --test-memory to verify it.

Illegal instruction I think this error generated because I have no avx and avx2 that's why Tensorflow not build correctly when I build redisai with bash get_deps.sh cpu after this error I installed tensorflow manully at redisgear successfully i.e /opt/redislabs/lib/modules/python3

import tensorflow as tf print(tf.version) 1.12.0 but I have same Error 80544:M 09 Oct 2019 08:40:50.269 # Backend TF not loaded, will try loading default backend and redis has broken. so how can I install Tensorflow 1.14.0 without AVX and AVX to load Tensorflow successfully

Machine : ubuntu 18.04 Python :3.7 AVX , AVX2 ,MKL and FMA : NO

Thanks

MeirShpilraien commented 5 years ago

@Akhtar303 Can you send me you redisai_tensorflow.so file? Maybe I will be able to understand from the stack trace where it crashed?

MeirShpilraien commented 5 years ago

Also how did you compiled redisai?

Akhtar303 commented 5 years ago

Also how did you compiled redisai? @MeirShpilraien I follow these steps bash get_deps.sh cpu mkdir build cd build cmake .. make && make install cd .. Thanks

Akhtar303 commented 5 years ago

@Akhtar303 Can you send me you redisai_tensorflow.so file? Maybe I will be able to understand from the stack trace where it crashed? @MeirShpilraien Thanks Here is Link https://drive.google.com/file/d/1Q4OAlWkTrg_vObCS482Ohz9f9R_2EmRX/view Thanks

Akhtar303 commented 5 years ago

@MeirShpilraien have you find any Cause for this problem. Thanks @MeirShpilraien