DigitalSlideArchive / digital_slide_archive

The official deployment of the Digital Slide Archive and HistomicsTK.
https://digitalslidearchive.github.io
Apache License 2.0
110 stars 50 forks source link

Error on loading CLI task #187

Closed ds2268 closed 2 years ago

ds2268 commented 2 years ago

I often get the following error when loading docker image through Slicer CLI Web Plugin:

Started to Load Docker images Got image image/image:latest, cli XXXX metadata Loaded metadata from pre-existing local image image/image:latest Error with job syntax error: line 1, column 0

When I run the image locally, it gives me no problem. The problem only shows from time to time for the same image?

Any clue, what is causing this problem?

ds2268 commented 2 years ago

The issue seems to be with the use of Python's logging instead of ordinary prints. Don't know what caused it, but when I remove logging and switch to prints, loading the tasks works again.

manthey commented 2 years ago

Do you have a minimal example? It be good to fix this.

ds2268 commented 2 years ago

It should break up if you add logging to the average_color example:

`import logging

log = logging.getLogger(name) # github transforms underscores _ before and after logging.basicConfig(stream=sys.stdout, level=logging.INFO)`

I have built my plugin based on this example and it broke when commented/uncommented the above logging initialization code. The only difference was that I used a custom Docker image based on pytorch/pytorch:1.9.1-cuda11.1-cudnn8-runtime, but this shouldn't be a problem. I have run the container locally directly and it did run without a problem. I have also uploaded the task without logging - so that this was done successfully and then built the image with logging, but did not again add it - already added, when executed, there were no errors, but there was also no output.

manthey commented 2 years ago

This is actually because the ctk_cli module emits a warning (WARNING:ctk_cli.module:attribute of 'file' ignored: reference='ImageFile'), and slicer_cli_web reads it and fails to parse the xml. Specifically, we extend the slicer_cli xml, and those extensions cause warnings unless we import our extensions first. That is,

This import will end up throwing a warning:

from ctk_cli import CLIArgumentParser 

This won't:

# This isn't used, but has the side effect or adjusting the ctk_cli parser
# to accept more parameters.  It must be before we import ctk_cli
from slicer_cli_web.ctk_cli_adjustment import CLIModule  # noqa
from ctk_cli import CLIArgumentParser  # noqa

You've set logging to got to stdout, so the warnings ends up in the same stream as the xml and then doesn't parse. Obviously, it is handy to log to stdout in lieu of printing, but in that instance we either need to ignore the ctk_cli warnings, or import our adjustment to avoid creating them, or log warnings to stderr and only info to stdout, or use something more complex than logging.basicConfig so that ctk_cli.module logs elsewhere.

You could, for instance, add

logging.getLogger('ctk_cli.module').setLevel(logging.ERROR)

after your basicConfig line.

ds2268 commented 2 years ago

Closing the issue. Switched to "prints" for now.