search

lindawangg / COVID-Net

COVID-Net Open Source Initiative
Other
1.15k stars 480 forks source link

Running the inference.py file is getting Killed after a while #108

Closed qomhmd closed 3 years ago

qomhmd commented 3 years ago

Before posting, have you looked at the FAQ page? Yes, they weren't related. Somewhat similar to the previous, closed issue #41 occurs.

Steps to Reproduce

  1. Creating a (mini)conda virtual environment.
  2. Installing all pre-requisites including Tensorflow using conda/pip. The recommended way to install TF using pip is not compatible with my seemingly old, non-avx CPU, so I installed using TF=1.15 package from the anaconda channel.
  3. Running: (COVID) root@vm000:path# python /opt/github.com/lindawangg/COVID-Net/inference.py --weightspath /opt/ai/covid/models/COVIDNet-CXR4-A --metaname model.meta --ckptname model-18540 --imagepath /opt/uploads/medical_imaging/cxr/ex-covid.jpeg

Expected behavior

The usual inferenc.py response regarding confidence of the three diagnoses.

Actual behavior

After a while (about 100 sec) from executing $ python inference.py, the word "Killed" appears.

(COVID) root@vm000:path# python /opt/github.com/lindawangg/COVID-Net/inference.py --weightspath /opt/ai/covid/models/COVIDNet-CXR4-A --metaname model.meta --ckptname model-18540 --imagepath /opt/uploads/medical_imaging/cxr/ex-covid.jpeg

WARNING:tensorflow:From /opt/github.com/lindawangg/COVID-Net/inference.py:23: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2020-11-26 13:35:19.995661: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2194710000 Hz
2020-11-26 13:35:19.996578: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55fce9791d00 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-11-26 13:35:19.996957: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
OMP: Info #212: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #210: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0
OMP: Info #156: KMP_AFFINITY: 1 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #159: KMP_AFFINITY: 1 packages x 1 cores/pkg x 1 threads/core (1 total cores)
OMP: Info #214: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0
OMP: Info #250: KMP_AFFINITY: pid 4567 tid 4567 thread 0 bound to OS proc set 0
2020-11-26 13:35:19.999040: I tensorflow/core/common_runtime/process_util.cc:115] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
WARNING:tensorflow:From /opt/github.com/lindawangg/COVID-Net/inference.py:24: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

WARNING:tensorflow:From /opt/github.com/lindawangg/COVID-Net/inference.py:25: The name tf.train.import_meta_graph is deprecated. Please use tf.compat.v1.train.import_meta_graph instead.

Killed

Environment

qomhmd commented 3 years ago

Related log using $ sudo dmesg.

[ 4337.143850] Out of memory: Kill process 5081 (python3) score 784 or sacrifice child [ 4337.144138] Killed process 5081 (python3) total-vm:3675760kB, anon-rss:818304kB, file-rss:136kB, shmem-rss:0kB [ 4337.255929] oom_reaper: reaped process 5081 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

qomhmd commented 3 years ago

Finally, reports appears after increasing my ram to 2GB. Waiting time was more than 5 minutes the first time, 1.5 minutes the next.