search

IBM / MAX-Object-Detector

Localize and identify multiple objects in a single image.
https://developer.ibm.com/exchanges/models/all/max-object-detector/
Apache License 2.0
290 stars 224 forks source link

Running Object Detector on older AMD x86 machine #130

Closed corvy closed 4 years ago

corvy commented 4 years ago

Hello, is there a way to run the detector on an older x86 machine? I am trying to reuse an older machine in my house to do object detection and camera recordings. But I cannot get neither :latest nor :v1.3.0 tags to run. 

root@ubuntu-server:~# docker run -p 5000:5000 codait/max-object-detector:v1.3.0
2020-04-13 09:36:00.831007: F tensorflow/core/platform/cpu_feature_guard.cc:37] The TensorFlow library was compiled to use SSE4.1 instructions, but these aren't available on your machine.
Aborted (core dumped)
root@ubuntu-server:~# docker run -p 5000:5000 codait/max-object-detector:latest
Illegal instruction (core dumped)

This is my CPU information: 

root@ubuntu-server:~# cat /proc/cpuinfo 
processor   : 0
vendor_id   : AuthenticAMD
cpu family  : 16
model       : 4
model name  : AMD Phenom(tm) II X4 955 Processor
stepping    : 3
microcode   : 0x10000c8
cpu MHz     : 800.000
cache size  : 512 KB
physical id : 0
siblings    : 4
core id     : 0
cpu cores   : 4
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 5
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr hw_pstate vmmcall npt lbrv svm_lock nrip_save
bugs        : tlb_mmatch apic_c1e fxsave_leak sysret_ss_attrs null_seg amd_e400 spectre_v1 spectre_v2
bogomips    : 6421.40
TLB size    : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

processor   : 1
vendor_id   : AuthenticAMD
cpu family  : 16
model       : 4
model name  : AMD Phenom(tm) II X4 955 Processor
stepping    : 3
microcode   : 0x10000c8
cpu MHz     : 3200.000
cache size  : 512 KB
physical id : 0
siblings    : 4
core id     : 1
cpu cores   : 4
apicid      : 1
initial apicid  : 1
fpu     : yes
fpu_exception   : yes
cpuid level : 5
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr hw_pstate vmmcall npt lbrv svm_lock nrip_save
bugs        : tlb_mmatch apic_c1e fxsave_leak sysret_ss_attrs null_seg amd_e400 spectre_v1 spectre_v2
bogomips    : 6421.40
TLB size    : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

processor   : 2
vendor_id   : AuthenticAMD
cpu family  : 16
model       : 4
model name  : AMD Phenom(tm) II X4 955 Processor
stepping    : 3
microcode   : 0x10000c8
cpu MHz     : 800.000
cache size  : 512 KB
physical id : 0
siblings    : 4
core id     : 2
cpu cores   : 4
apicid      : 2
initial apicid  : 2
fpu     : yes
fpu_exception   : yes
cpuid level : 5
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr hw_pstate vmmcall npt lbrv svm_lock nrip_save
bugs        : tlb_mmatch apic_c1e fxsave_leak sysret_ss_attrs null_seg amd_e400 spectre_v1 spectre_v2
bogomips    : 6421.40
TLB size    : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

processor   : 3
vendor_id   : AuthenticAMD
cpu family  : 16
model       : 4
model name  : AMD Phenom(tm) II X4 955 Processor
stepping    : 3
microcode   : 0x10000c8
cpu MHz     : 800.000
cache size  : 512 KB
physical id : 0
siblings    : 4
core id     : 3
cpu cores   : 4
apicid      : 3
initial apicid  : 3
fpu     : yes
fpu_exception   : yes
cpuid level : 5
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr hw_pstate vmmcall npt lbrv svm_lock nrip_save
bugs        : tlb_mmatch apic_c1e fxsave_leak sysret_ss_attrs null_seg amd_e400 spectre_v1 spectre_v2
bogomips    : 6421.40
TLB size    : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate
xuhdev commented 4 years ago

Could you try to build the image? That might work. Clone the repo and run:

docker build -t max-object-detector .
corvy commented 4 years ago

Thanks for the tip @xuhdev

Tried to compile, that worked, but still I get an error. Tried to run with "docker events":

root@ubuntu-server:~/temp/MAX-Object-Detector# docker build -t max-object-detector-local .
Sending build context to Docker daemon  2.398MB
Step 1/15 : FROM codait/max-base:v1.3.2
 ---> b2e47f6a8e46
Step 2/15 : RUN apt-get update && apt-get -y install libatlas3-base && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> a213789dee29
Step 3/15 : ARG model_bucket=https://max-cdn.cdn.appdomain.cloud/max-object-detector/1.0.1
 ---> Using cache
 ---> 34015a7a6950
Step 4/15 : ARG model_file=model.tar.gz
 ---> Using cache
 ---> 8cce208975c0
Step 5/15 : ARG data_file=data.tar.gz
 ---> Using cache
 ---> a1a89db6b46d
Step 6/15 : ARG use_pre_trained_model=true
 ---> Using cache
 ---> 1271bcf0c5bf
Step 7/15 : WORKDIR /workspace
 ---> Using cache
 ---> 934e070898cd
Step 8/15 : RUN if [ "$use_pre_trained_model" = "true" ] ; then    wget -nv --show-progress --progress=bar:force:noscroll ${model_bucket}/${model_file} --output-document=assets/${model_file} &&            tar -x -C assets/ -f assets/${model_file} -v && rm assets/${model_file} &&     wget -nv --show-progress --progress=bar:force:noscroll ${model_bucket}/${data_file} --output-document=assets/${data_file} &&            tar -x -C assets/ -f assets/${data_file} -v && rm assets/${data_file}; fi
 ---> Using cache
 ---> 659a326fa850
Step 9/15 : RUN wget -O - -nv --show-progress --progress=bar:force:noscroll https://github.com/IBM/MAX-Object-Detector-Web-App/archive/v2.0.tar.gz |   tar zxvf - --strip-components=1 --wildcards 'MAX-Object-Detector-Web-App-*/static'
 ---> Using cache
 ---> 76c6d4f36b38
Step 10/15 : COPY requirements.txt /workspace
 ---> Using cache
 ---> 7544d6e481bf
Step 11/15 : RUN pip install -r requirements.txt
 ---> Using cache
 ---> 76a1eb7070f3
Step 12/15 : COPY . /workspace
 ---> Using cache
 ---> 27c3a1b1e18a
Step 13/15 : RUN if [ "$use_pre_trained_model" = "true" ] ; then       sha512sum -c sha512sums.txt ;     else       if [ -d "./custom_assets/" ] ; then         rm -rf ./assets && ln -s ./custom_assets ./assets ;       fi     fi
 ---> Using cache
 ---> 7013de9d293e
Step 14/15 : EXPOSE 5000
 ---> Using cache
 ---> a1b774a2098b
Step 15/15 : CMD python app.py
 ---> Using cache
 ---> cdd1f44a98c8
Successfully built cdd1f44a98c8
Successfully tagged max-object-detector-local:latest
2020-04-14T06:59:08.463136306Z image tag sha256:cdd1f44a98c8aea5b5c9c261fef98f917eb1617f8bf3f57dd42154f1a9ee2392 (name=max-object-detector-local:latest)
root@ubuntu-server:~/temp/MAX-Object-Detector# docker images ls
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
root@ubuntu-server:~/temp/MAX-Object-Detector# docker image ls
REPOSITORY                  TAG                 IMAGE ID            CREATED             SIZE
max-object-detector-local   latest              cdd1f44a98c8        9 hours ago         1.25GB
max-object-detector         latest              cdd1f44a98c8        9 hours ago         1.25GB
codait/max-base             v1.3.2              b2e47f6a8e46        7 weeks ago         552MB
ccrisan/motioneye           master-amd64        692eccfe45a6        4 months ago        411MB
root@ubuntu-server:~/temp/MAX-Object-Detector# docker run -it -p 5000:5000 max-object-detector-local 
2020-04-14T06:59:44.724168240Z container create f1dc3747609f61b82bdaaa8d661d7d193243678092162bdbd185d61eef8ce02b (image=max-object-detector-local, name=awesome_wozniak)
2020-04-14T06:59:44.728566097Z container attach f1dc3747609f61b82bdaaa8d661d7d193243678092162bdbd185d61eef8ce02b (image=max-object-detector-local, name=awesome_wozniak)
                                                                                                                                                                        2020-04-14T06:59:44.764823888Z network connect 731c26b30da00608ea6358ab300d81388fb4e1e891e3e37d3d06f28c024f7673 (container=f1dc3747609f61b82bdaaa8d661d7d193243678092162bdbd185d61eef8ce02b, name=bridge, type=bridge)
                                                                                                                                                                  2020-04-14T06:59:45.486215542Z container start f1dc3747609f61b82bdaaa8d661d7d193243678092162bdbd185d61eef8ce02b (image=max-object-detector-local, name=awesome_wozniak)
                                                                                                             2020-04-14T06:59:45.495438436Z container resize f1dc3747609f61b82bdaaa8d661d7d193243678092162bdbd185d61eef8ce02b (height=36, image=max-object-detector-local, name=awesome_wozniak, width=220)
                                                                               Illegal instruction (core dumped)
2020-04-14T06:59:46.182605547Z container die f1dc3747609f61b82bdaaa8d661d7d193243678092162bdbd185d61eef8ce02b (exitCode=132, image=max-object-detector-local, name=awesome_wozniak)
2020-04-14T06:59:46.322506300Z network disconnect 731c26b30da00608ea6358ab300d81388fb4e1e891e3e37d3d06f28c024f7673 (container=f1dc3747609f61b82bdaaa8d661d7d193243678092162bdbd185d61eef8ce02b, name=bridge, type=bridge)
root@ubuntu-server:~/temp/MAX-Object-Detector#
corvy commented 4 years ago

Looks like the illegal instruction is a problem on tag latest.

How could I run a build on v1.3.0? Do I check out a specific version of the github rep?

See this test on a Intel CPU:

Digest: sha256:ca9b10849ac84b21ab3e4b66d92c2c7ff0a3320be5b38b3ec249737465cfb4ec Status: Downloaded newer image for codait/max-object-detector:latest Illegal instruction

This computer is also older so it does not have SSE instructions. 

Status: Downloaded newer image for codait/max-object-detector:v1.3.0
2020-04-14 07:12:02.350973: F tensorflow/core/platform/cpu_feature_guard.cc:37] The TensorFlow library was compiled to use SSE4.1 instructions, but these aren't available on your machine.
Aborted
root@web:~#

Will try to do a manual compile also on this computer to compare.

corvy commented 4 years ago

Found a way to checkout v1.3.0 so I will continue to test. 

root@ubuntu-server:~/temp/MAX-Object-Detector# git checkout tags/v1.3.0
Note: checking out 'tags/v1.3.0'.
xuhdev commented 4 years ago

This is probably because codait/max-base is based on x64. You probably have to also rebuild an i686 version of that: https://github.com/IBM/MAX-Base/blob/master/Dockerfile

Actually, to do that, you also have to rebuild the miniconda3 base on i686: https://hub.docker.com/r/continuumio/miniconda3/dockerfile (a 386 version of the base is available)

corvy commented 4 years ago

Version 1.1.0 worked. Strange. 

root@ubuntu-server:~/temp/MAX-Object-Detector# docker run -it -p 5000:5000 codait/max-object-detector:v1.1.0
Unable to find image 'codait/max-object-detector:v1.1.0' locally
v1.1.0: Pulling from codait/max-object-detector
cc1a78bfd46b: Already exists 
49eab01d36f3: Already exists 
c2c2cfea0213: Already exists 
419499c9a4cf: Already exists 
f7550509e92e: Already exists 
07ffab9d2da8: Already exists 
66ba7c4c6ba3: Already exists 
47ed8bee4580: Pull complete 
67bbc4c2da04: Pull complete 
05e276dd8add: Pull complete 
ca9bf41d3989: Pull complete 
18ca47bb8a10: Pull complete 
0041cffa457f: Pull complete 
e546c103de4b: Pull complete 
26cf8716aabd: Pull complete 
71b54601090f: Pull complete 
bffba3d773da: Pull complete 
f3d7f53f317a: Pull complete 
70607a26d50b: Pull complete 
f7a15ebb2fd7: Pull complete 
b6f2f4185597: Pull complete 
Digest: sha256:21e1e2aacf6d8d35e234a00f8348c387f0c35a58de1710b5a966f6c9c761777c
Status: Downloaded newer image for codait/max-object-detector:v1.1.0
 * Serving Flask app "MAX Object Detector" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
corvy commented 4 years ago

I think this iss boils down to AVX. This container looks promising, but I was wondering if it would be possible to get an official container with no AVX?

https://hub.docker.com/r/nklint/max-object-detector-no-avx

xuhdev commented 4 years ago

If this is an AVX issue, then it seems like we have to rebuild TensorFlow by ourselves (or perhaps use a service (by human). Then you may change tensorflow in the requirement file.

@bdwyer2 Do you think we should provide such a TF version across all relevant MAX models?