search

microsoft / aerial_wildlife_detection

Tools for detecting wildlife in aerial images using active learning
MIT License
226 stars 58 forks source link

Model Workflow Designer #29

Open scfulford opened 3 years ago

scfulford commented 3 years ago

Hello!

We're having some trouble with some model prediction workflows getting stuck in the Running Workflow section rather than moving into the Finished section. We've used it to successfully for a few image prediction data sets a few months ago and this seems to be a new issue as of a couple weeks ago. Since the workflow designer page is still labelled as a work in progress, is it best to just hold off on model predictions until its finished or do you know if something else could be causing an issue?

Thank you!!

bkellenb commented 3 years ago

Hello!

Thanks for opening the issue. You are right, model processes sometimes really appear to be stuck in an unfinished state in the browser. However, they often still finish, even if the GUI says differently. This is because it is not as straight-forward as it seems to detect whether a task actually finished or not. I am still trying to resolve this issue; it also has to do with some bugs of the job dispatcher (mostly because Celery chains/chords not reporting back when they are finished).

Error messages should always go through—if you see a white cross on a red circle in the top right corner of a process, it has failed for one or another reason.