search

oist / Usiigaci

Usiigaci: stain-free cell tracking in phase contrast microscopy enabled by supervised machine learning
MIT License
192 stars 68 forks source link

failed to create mask folder & failed to deploy the inference, skipping... #3

Closed hiroalchem closed 5 years ago

hiroalchem commented 5 years ago

I had error below

Loading model from: trained_network/Usiigaci_1.h5 prediction for: /home/Usiigaci/test/T98G_sample model run 1 of 3 failed to create mask folder 97%|#########################################8 | 37/38 [00:49<00:01, 1.33s/it]failed to deploy the inference, skipping... Loading model from: trained_network/Usiigaci_2.h5 prediction for: /home/Usiigaci/test/T98G_sample model run 2 of 3 failed to create mask folder

0%| | 0/38 [00:00<?, ?it/s]^Cfailed to deploy the inference, skipping...

Loading model from: trained_network/Usiigaci_3.h5 prediction for: /home/Usiigaci/test/T98G_sample model run 3 of 3 failed to create mask folder

hftsai commented 5 years ago

Hi I observed this sometimes on particular machines. It could be due to the inference script found the log file and try to do inference, then get skipped. I'll try to track down where the problem is

in the mean time if you don't need to log the output

try commenting out line 318

sys.stdout=Logger()

See if it is resolved. Please do let me know

hiroalchem commented 5 years ago

Thank you for your reply. I commented out line 318 however, I still got stuck like below

prediction for:  /home/hkawai/obj/Usiigaci/test2/T98G_sample
model run 1 of 3
 97%|#########################################################################################################################################2   | 37/38 [00:54<00:01,  1.46s/it]failed to deploy the inference, skipping...
>Loading model from:  trained_network/Usiigaci_2.h5
prediction for:  /home/hkawai/obj/Usiigaci/test2/T98G_sample
model run 2 of 3

 11%|##############9

and when I stopped it with "ctrl +c"


 to deploy the inference, skipping...
>Loading model from:  trained_network/Usiigaci_3.h5
prediction for:  /home/Usiigaci/test2/T98G_sample
model run 3 of 3
Exception ignored in: <object repr() failed>.
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tqdm/_tqdm.py", line 887, in __del__
    self.close()
  File "/usr/local/lib/python3.5/dist-packages/tqdm/_tqdm.py", line 1098, in close
    if self.disable:
AttributeError: 'tqdm' object has no attribute 'disable'
Merging multiple models predictions.
^CTraceback (most recent call last):
  File "Inference.py", line 356, in <module>
    postprocess(mask_duplicate_dir, avg_prediction_dir)
  File "Inference.py", line 268, in postprocess
    for filename in tqdm(filenames):
  File "/usr/local/lib/python3.5/dist-packages/tqdm/_tqdm.py", line 447, in __new__
    with cls._lock:
  File "/usr/local/lib/python3.5/dist-packages/tqdm/_tqdm.py", line 77, in __enter__
    self.acquire()
  File "/usr/local/lib/python3.5/dist-packages/tqdm/_tqdm.py", line 70, in acquire
    lock.acquire()
KeyboardInterrupt
Exception ignored in: <object repr() failed>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tqdm/_tqdm.py", line 887, in __del__
    self.close()
  File "/usr/local/lib/python3.5/dist-packages/tqdm/_tqdm.py", line 1098, in close
    if self.disable:
AttributeError: 'tqdm' object has no attribute 'disable'
^C
hftsai commented 5 years ago

hah. i see the problem. there should be only 37 files in the T98G sample file. I made a avi movie that doesn't below to it. it should all be only tif files in a folder to do the inference.

Please remove the T98G_sample.avi before running it.

I will figure out a better solution later.

hiroalchem commented 5 years ago

I removed .avi file, however same error occurred.

hftsai commented 5 years ago

Please make sure there should only be image files in the folder. I haven't run into the inference process stuck in the middle before. Which implies some problems on the tqdm or the tensorflow.

To check, can you run the inference script with just one model? See if can generate results on your system. If not then it's something wrong with tensorflow installation.

hiroalchem commented 5 years ago

I am sorry that I found ".ipynb_checkpoints" in the image directory. I removed it and then everything worked fine! Thank you.

hftsai commented 5 years ago

hew~ good to hear. Please let me know if it works for you. If there's any problem, i will try to help out as much as I can. If it doesn't work, please let me know too. then it can be a direction we can try to improve the software.

best,