Exception in ASGI application

[laurencei]

Hi there!

Thanks for this amazing package!

I've got it installed on Home Assistant and the default models work - i.e. I can detect a bike etc.

I wanted to add a custom model, but I'm getting an "Exception in ASGI application" when I try to run the image.

To confirm

• The doods webpage loads, and the configuration loads.
• I can see my model loaded with the labels. I can do a "default" image with the same image no problem.

Below is the full log.

Attached is the model.

2023-01-19 16:20:00,227 - doods.doods - INFO - Registered detector type:tflite name:default
2023-01-19 16:20:00,235 - doods.doods - INFO - Registered detector type:tflite name:bins
2023-01-19 16:20:00,498 - uvicorn.error - INFO - Started server process [7]
2023-01-19 16:20:00,498 - uvicorn.error - INFO - Waiting for application startup.
2023-01-19 16:20:00,499 - uvicorn.error - INFO - Application startup complete.
2023-01-19 16:20:00,499 - uvicorn.error - INFO - Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
2023-01-19 16:20:03,034 - uvicorn.access - INFO - 192.168.1.158:65298 - "GET / HTTP/1.1" 304
2023-01-19 16:20:03,082 - uvicorn.access - INFO - 192.1
[bin_model.zip](https://github.com/snowzach/doods2/files/10453903/bin_model.zip)
68.1.158:65298 - "GET /detectors HTTP/1.1" 200
2023-01-19 16:20:09,572 - uvicorn.access - INFO - 192.168.1.158:65299 - "POST /image HTTP/1.1" 500
2023-01-19 16:20:09,573 - uvicorn.error - ERROR - Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/protocols/http/h11_impl.py", line 373, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/fastapi/applications.py", line 208, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.8/dist-packages/prometheus_fastapi_instrumentator/middleware.py", line 103, in __call__
    raise exc
  File "/usr/local/lib/python3.8/dist-packages/prometheus_fastapi_instrumentator/middleware.py", line 101, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/usr/local/lib/python3.8/dist-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/usr/local/lib/python3.8/dist-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 61, in app
    response = await func(request)
  File "/usr/local/lib/python3.8/dist-packages/fastapi/routing.py", line 226, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.8/dist-packages/fastapi/routing.py", line 159, in run_endpoint_function
    return await dependant.call(**values)
  File "/opt/doods/api.py", line 113, in image
    detect_response = self.doods.detect(detect_request)
  File "/opt/doods/doods.py", line 139, in detect
    ret = detector.detect(image)
  File "/opt/doods/detectors/tflite.py", line 114, in detect
    for i in range(int(count)):
TypeError: only size-1 arrays can be converted to Python scalars

[laurencei]

bin_model.zip

Whoops - sorry - here is the model.

[snowzach]

tflite models need a very specific input and output format.. Supporting a model that's outside of that input and output format is probably beyond my capability. Sorry.. :-/

[laurencei]

@snowzach - nah, of course - no problem.

Where do you generate your tflite files? i.e. what is the "input/output" format I should be looking for?

I was doing the Google Colab one, so is there another way that you know of?

[snowzach]

Honestly, I am just using the default tflite models. If you can come up with python code that uses the model you have, it might be easier to see what could be done with it. The code that uses tflite files is here: https://github.com/snowzach/doods2/blob/master/detectors/tflite.py

[laurencei]

Honestly, I am just using the default tflite models. If you can come up with python code that uses the model you have, it might be easier to see what could be done with it. The code that uses tflite files is here: https://github.com/snowzach/doods2/blob/master/detectors/tflite.py

Ok. So I'm trying to modify the tflite.py file, but I'm not a docker expert.

If I do this command in the project root: docker build -t doodstest docker - the container builds, but when I run docker run -it -p 8080:8080 doodstest - I get an error python3: can't open file 'main.py': [Errno 2] No such file or directory

So I'm obviously doing something wrong to build the container? Is there a recommended way to make a change to the python code file and persist it to be able to run it? (I dont expect you to teach me Docker or anything - just trying to understand your build process so I can try to fix the issue and provide a PR).

[cjborchert]

@laurencei did you make any further progress? I think I'm running into the same issue when I try to make use of a model created using the Google Colab script. Model loads, but first image processing request crashes DOODS with the below exception (whether through Home Assistant or the DOODS web portal). I described a bit more about how I made the model on the Home Assistant community forums. It seems like there must be a way to do this. Perhaps the Google Colab script uses a different method for generating the model than the default models.

2023-02-19 22:41:02,689 - doods.doods - INFO - Registered detector type:tflite name:default
2023-02-19 22:41:02,698 - doods.doods - INFO - Registered detector type:tflite name:youtubeads
2023-02-19 22:41:09,673 - doods.doods - INFO - Registered detector type:tensorflow name:tensorflow
2023-02-19 22:41:09,796 - uvicorn.error - INFO - Started server process [7]
2023-02-19 22:41:09,796 - uvicorn.error - INFO - Waiting for application startup.
2023-02-19 22:41:09,797 - uvicorn.error - INFO - Application startup complete.
2023-02-19 22:41:09,798 - uvicorn.error - INFO - Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
2023-02-19 23:30:29,465 - uvicorn.access - INFO - XXX.XXX.XX.XX:XXXXX - "POST /image HTTP/1.1" 500
2023-02-19 23:30:29,467 - uvicorn.error - ERROR - Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/protocols/http/h11_impl.py", line 373, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/fastapi/applications.py", line 208, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.8/dist-packages/prometheus_fastapi_instrumentator/middleware.py", line 103, in __call__
    raise exc
  File "/usr/local/lib/python3.8/dist-packages/prometheus_fastapi_instrumentator/middleware.py", line 101, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/usr/local/lib/python3.8/dist-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/usr/local/lib/python3.8/dist-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 61, in app
    response = await func(request)
  File "/usr/local/lib/python3.8/dist-packages/fastapi/routing.py", line 226, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.8/dist-packages/fastapi/routing.py", line 159, in run_endpoint_function
    return await dependant.call(**values)
  File "/opt/doods/api.py", line 113, in image
    detect_response = self.doods.detect(detect_request)
  File "/opt/doods/doods.py", line 139, in detect
    ret = detector.detect(image)
  File "/opt/doods/detectors/tflite.py", line 114, in detect
    for i in range(int(count)):
TypeError: only size-1 arrays can be converted to Python scalars

[laurencei]

@cjborchert - no, I didnt have any luck, Havent had much time - but really want to solve it.

If you find a way please let me know, otherwise I will try again in a few weeks when I get some free time :)

Or if there is another way to train custom models that work with this, I might switch to that...

[cjborchert]

@laurencei Ok, I'll let you know if I get anywhere. I don't really know what I'm doing, but can try some brute force debugging. If we're really lucky, getting the input/output shape correct is as simple as chosing the right model spec (shape is mentioned in the EfficientNet-Lite0 model spec API docs). That spec could be easily changed in the Google Colab notebook.

When I get a chance in the evenings, I'll try creating all those model specs with my data to see if one of them works in DOODS, or perhaps try to figure out the model spec of the default tflite model packaged with DOODS.

[snowzach]

Okay, it looks like the output just had the count/boxes/confidence and classes in a different order... Pull the latest docker image and see if it works...

[cjborchert]

Okay, it looks like the output just had the count/boxes/confidence and classes in a different order... Pull the latest docker image and see if it works...

@snowzach Amazing, it's working! This is going to be fun. Logs looking like this now with my custom model loaded:

2023-02-21 17:52:19,255 - doods.doods - INFO - Registered detector type:tflite name:default
2023-02-21 17:52:19,263 - doods.doods - INFO - Registered detector type:tflite name:youtubeads
2023-02-21 17:52:26,104 - doods.doods - INFO - Registered detector type:tensorflow name:tensorflow
2023-02-21 17:52:26,231 - uvicorn.error - INFO - Started server process [7]
2023-02-21 17:52:26,232 - uvicorn.error - INFO - Waiting for application startup.
2023-02-21 17:52:26,233 - uvicorn.error - INFO - Application startup complete.
2023-02-21 17:52:26,234 - uvicorn.error - INFO - Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
2023-02-21 18:00:08,579 - uvicorn.access - INFO - 192.168.50.100:58522 - "GET /detectors HTTP/1.1" 200
2023-02-21 18:00:08,634 - uvicorn.access - INFO - 192.168.50.100:58528 - "GET /detectors HTTP/1.1" 200
2023-02-21 18:06:03,608 - doods.doods - INFO - DetectResponse(id='', image=None, detections=[], error=None, duration=257.4712848290801)
2023-02-21 18:06:03,610 - uvicorn.access - INFO - 192.168.50.100:41776 - "POST /detect HTTP/1.1" 200
2023-02-21 18:07:52,218 - uvicorn.access - INFO - 192.168.50.37:62962 - "GET /detectors HTTP/1.1" 200
2023-02-21 18:08:27,558 - doods.doods - INFO - DetectResponse(id='manual', image=None, detections=[Detection(region_id=None, top=0.029689401388168335, left=-0.03299739956855774, bottom=0.8498482704162598, right=0.9125350713729858, label='person', confidence=63.28125, image=None), Detection(region_id=None, top=0.09326356649398804, left=0.0009166896343231201, bottom=0.8017772436141968, right=0.6555933952331543, label='person', confidence=55.078125, image=None)], error=None, duration=138.89911002479494)
2023-02-21 18:08:27,635 - uvicorn.access - INFO - 192.168.50.37:62963 - "POST /image HTTP/1.1" 200
2023-02-21 18:08:36,762 - doods.doods - INFO - DetectResponse(id='manual', image=None, detections=[Detection(region_id=None, top=0.07453380525112152, left=0.05353961139917374, bottom=0.13912010192871094, right=0.18053725361824036, label='unknown:0.0', confidence=83.203125, image=None)], error=None, duration=249.52724785543978)
2023-02-21 18:08:36,838 - uvicorn.access - INFO - 192.168.50.37:62981 - "POST /image HTTP/1.1" 200
2023-02-21 18:09:23,138 - doods.doods - INFO - DetectResponse(id='manual', image=None, detections=[Detection(region_id=None, top=0.07621215283870697, left=0.0513460636138916, bottom=0.13971097767353058, right=0.1827308088541031, label='unknown:0.0', confidence=85.15625, image=None), Detection(region_id=None, top=0.8314181566238403, left=0.8389620780944824, bottom=0.8881784677505493, right=0.9468444585800171, label='ad_notice', confidence=78.515625, image=None)], error=None, duration=246.565684909001)
2023-02-21 18:09:23,211 - uvicorn.access - INFO - 192.168.50.37:63003 - "POST /image HTTP/1.1" 200

[cjborchert]

@snowzach I spoke slightly too soon. It seems like the objects are being detected with great accuracy, however the labels are not working right. I've so far been unable to get this working via the DOODS addon configuration.yaml or by messing around with the labels file. I only have two labels in the tflite model, and it seems that one of the objects from the model is always getting assigned the first label listed in the labelFile, regardless of what that is. Any additional lines in the labelFile seem to be ignored. Perhaps something is going wrong with the labelsStartFromZero: true parameter in the addon configuration.yaml. Whether I set this parameter to true, false, or remove it altogether from the config seems to make no difference. If I add in more than 2 labels to the labelFile, or add a 'Labels:' header, it doesn't matter (the first line is always used as the label for the 'skip ads' object, and the ad countdown object always gets 'unknown:0.0').

Here is a detecion log showing labels 'skip_button' and 'unknown:0.0' (the first one is correct, but second should be 'ad_notice'). The relevant labelFile is shown below:

labelFile:

skip_button
ad_notice

Log:

2023-02-21 22:20:14,876 - doods.doods - INFO - DetectResponse(id='manual', image=None, detections=[Detection(region_id=None, top=0.8273845314979553, left=0.8294242024421692, bottom=0.8930769562721252, right=0.9501143097877502, label='skip_button', confidence=87.109375, image=None), Detection(region_id=None, top=0.0823274627327919, left=0.055440209805965424, bottom=0.14267252385616302, right=0.17409822344779968, label='unknown:0.0', confidence=81.25, image=None)], error=None, duration=261.32299890741706)
2023-02-21 22:20:14,953 - uvicorn.access - INFO - 192.168.50.37:63840 - "POST /image HTTP/1.1" 200

Web portal output:

When I swap the labelFile around, the correctly detected 'skip_button' object is now incorrectly labeled 'ad_notice', and the correctly detected 'ad_notice' object remains incorrectly labeled 'unknown:0.0'.

labelFile:

ad_notice
skip_button

Log:

2023-02-21 22:34:41,935 - doods.doods - INFO - DetectResponse(id='manual', image=None, detections=[Detection(region_id=None, top=0.8273845314979553, left=0.8294242024421692, bottom=0.8930769562721252, right=0.9501143097877502, label='ad_notice', confidence=87.109375, image=None), Detection(region_id=None, top=0.0823274627327919, left=0.055440209805965424, bottom=0.14267252385616302, right=0.17409822344779968, label='unknown:0.0', confidence=81.25, image=None)], error=None, duration=260.9857909847051)
2023-02-21 22:34:42,012 - uvicorn.access - INFO - 192.168.50.37:63886 - "POST /image HTTP/1.1" 200

Web portal output:

[snowzach]

Hmm... something seems fishy... I remember seeing that one of your classes is 2 but the unknown one there is coming in with class 0. Maybe you have class 0 and 2 and no class 1? I would create a label file with say 5 values:

class0
class1
class2
class3
class4

and then set labelsStartFromZero: true also and see what shows up in your image... Adjust your labels file accordingly.

[snowzach]

You can also define specific class numbers with a labels file that looks like this:

0 skip_button
2 ad_notice

[cjborchert]

Hmmm... I set labelsStartFromZero: true under my custom tflite detector in the addon config, and then adjusted the labelFile (restarting the addon before each test).

labelFile:

class0
class1
class2
class3
class4

results in this:

labelFile:

0 skip_button
2 ad_notice

results in this:

[cjborchert]

This did the trick! Didn't realize I needed indices in the labelFile. Thanks @snowzach, my YouTube ad dismissing neural network is alive!

labelsStartFromZero: true

labelFile:

0 ad_notice
1 skip_button

[snowzach]

Great! Will close this.

[laurencei]

awesome @snowzach! This is brilliant - thank you!!

[Innocentius-alt]

Okay, it looks like the output just had the count/boxes/confidence and classes in a different order... Pull the latest docker image and see if it works...

Hello, Did you also update hassio-addons to fix this issue?