search

TomBursch / kitchenowl

KitchenOwl is a self-hosted grocery list and recipe manager. The backend is made with Flask and the frontend with Flutter. Easily add items to your shopping list before you go shopping. You can also create recipes and add items based on what you want to cook.
https://kitchenowl.org/
GNU Affero General Public License v3.0
1.24k stars 68 forks source link

Bug: Error on adding recipe from hellofresh #424

Open Karamellwuerfel opened 6 months ago

Karamellwuerfel commented 6 months ago

Is this urgent?

No

What parts are affected

Backend

What is the server version

93

What is the client version

0.4.21

What platform are you using

Linux

What's the problem 🤔

Hi, when I try to import a recipe from hellofresh, a 500 error occurs and it seems to have a problem in the python backend code (see log).

HelloFresh Link to import: https://www.hellofresh.de/recipes/torchietti-in-tomaten-paprika-sosse-5fc66ae618d71c10e64e51d4

Thanks!

Share your logs

[04/Apr/2024:09:21:33 +0000] "POST /api/household/1/recipe/scrape HTTP/2.0" 500 20 "https://kitchen.[secret].de/household/1/recipes" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:124.0) Gecko/20100101 Firefox/124.0"

------

[2024-04-04 09:25:17,040] ERROR in config: argument of type 'NoneType' is not iterable
Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 867, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 852, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask_jwt_extended/view_decorators.py", line 170, in decorator
    return current_app.ensure_sync(fn)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/kitchenowl/app/helpers/authorize_household.py", line 26, in decorator
    return func(*args, **kwargs)  # case server admin
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/kitchenowl/app/helpers/validate_args.py", line 26, in func_wrapper
    return func(arguments, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/kitchenowl/app/controller/recipe/recipe_controller.py", line 202, in scrapeRecipe
    recipe.name = scraper.title()
                  ^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/exception_handling.py", line 64, in decorated_method_wrapper
    return decorated(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/html_tags_stripper.py", line 75, in decorated_method_wrapper
    decorated_func_result = decorated(self, *args, **kwargs)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/normalize_string.py", line 34, in decorated_method_wrapper
    return normalize_string(decorated(self, *args, **kwargs))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/schemaorg_fill.py", line 48, in decorated_method_wrapper
    return decorated(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/hellofresh.py", line 14, in title
    return self.schema.title()
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/_schemaorg.py", line 103, in title
    return normalize_string(self.data.get("name"))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/_utils.py", line 144, in normalize_string
    unescaped_string = html.unescape(string)
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/html/__init__.py", line 130, in unescape
    if '&' not in s:
       ^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable
ERROR:app.config:argument of type 'NoneType' is not iterable
Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 867, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 852, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask_jwt_extended/view_decorators.py", line 170, in decorator
    return current_app.ensure_sync(fn)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/kitchenowl/app/helpers/authorize_household.py", line 26, in decorator
    return func(*args, **kwargs)  # case server admin
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/kitchenowl/app/helpers/validate_args.py", line 26, in func_wrapper
    return func(arguments, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/kitchenowl/app/controller/recipe/recipe_controller.py", line 202, in scrapeRecipe
    recipe.name = scraper.title()
                  ^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/exception_handling.py", line 64, in decorated_method_wrapper
    return decorated(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/html_tags_stripper.py", line 75, in decorated_method_wrapper
    decorated_func_result = decorated(self, *args, **kwargs)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/normalize_string.py", line 34, in decorated_method_wrapper
    return normalize_string(decorated(self, *args, **kwargs))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/schemaorg_fill.py", line 48, in decorated_method_wrapper
    return decorated(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/hellofresh.py", line 14, in title
    return self.schema.title()
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/_schemaorg.py", line 103, in title
    return normalize_string(self.data.get("name"))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/_utils.py", line 144, in normalize_string
    unescaped_string = html.unescape(string)
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/html/__init__.py", line 130, in unescape
    if '&' not in s:
       ^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable

Share your configuration

irrelevant
TomBursch commented 6 months ago

Should be fixed with the just published latest version.

Karamellwuerfel commented 6 months ago

Hi @TomBursch, I updated to the last two versions of your images and still get the following error in backend:

kitchenowlback     | [2024-04-04 12:03:48,441] ERROR in config: argument of type 'NoneType' is not iterable
kitchenowlback     | Traceback (most recent call last):
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 870, in full_dispatch_request
kitchenowlback     |     rv = self.dispatch_request()
kitchenowlback     |          ^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 855, in dispatch_request
kitchenowlback     |     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/flask_jwt_extended/view_decorators.py", line 170, in decorator
kitchenowlback     |     return current_app.ensure_sync(fn)(*args, **kwargs)
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/usr/src/kitchenowl/app/helpers/authorize_household.py", line 26, in decorator
kitchenowlback     |     return func(*args, **kwargs)  # case server admin
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/usr/src/kitchenowl/app/helpers/validate_args.py", line 26, in func_wrapper
kitchenowlback     |     return func(arguments, *args, **kwargs)
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/usr/src/kitchenowl/app/controller/recipe/recipe_controller.py", line 202, in scrapeRecipe
kitchenowlback     |     recipe.name = scraper.title()
kitchenowlback     |                   ^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/exception_handling.py", line 64, in decorated_method_wrapper
kitchenowlback     |     return decorated(self, *args, **kwargs)
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/html_tags_stripper.py", line 75, in decorated_method_wrapper
kitchenowlback     |     decorated_func_result = decorated(self, *args, **kwargs)
kitchenowlback     |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/normalize_string.py", line 34, in decorated_method_wrapper
kitchenowlback     |     return normalize_string(decorated(self, *args, **kwargs))
kitchenowlback     |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/schemaorg_fill.py", line 48, in decorated_method_wrapper
kitchenowlback     |     return decorated(self, *args, **kwargs)
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/hellofresh.py", line 14, in title
kitchenowlback     |     return self.schema.title()
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/_schemaorg.py", line 103, in title
kitchenowlback     |     return normalize_string(self.data.get("name"))
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/_utils.py", line 187, in normalize_string
kitchenowlback     |     unescaped_string = html.unescape(string)
kitchenowlback     |                        ^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/usr/local/lib/python3.11/html/__init__.py", line 130, in unescape
kitchenowlback     |     if '&' not in s:
kitchenowlback     |        ^^^^^^^^^^^^
kitchenowlback     | TypeError: argument of type 'NoneType' is not iterable
kitchenowlback     | ERROR:app.config:argument of type 'NoneType' is not iterable
kitchenowlback     | Traceback (most recent call last):
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 870, in full_dispatch_request
kitchenowlback     |     rv = self.dispatch_request()
kitchenowlback     |          ^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 855, in dispatch_request
kitchenowlback     |     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/flask_jwt_extended/view_decorators.py", line 170, in decorator
kitchenowlback     |     return current_app.ensure_sync(fn)(*args, **kwargs)
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/usr/src/kitchenowl/app/helpers/authorize_household.py", line 26, in decorator
kitchenowlback     |     return func(*args, **kwargs)  # case server admin
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/usr/src/kitchenowl/app/helpers/validate_args.py", line 26, in func_wrapper
kitchenowlback     |     return func(arguments, *args, **kwargs)
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/usr/src/kitchenowl/app/controller/recipe/recipe_controller.py", line 202, in scrapeRecipe
kitchenowlback     |     recipe.name = scraper.title()
kitchenowlback     |                   ^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/exception_handling.py", line 64, in decorated_method_wrapper
kitchenowlback     |     return decorated(self, *args, **kwargs)
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/html_tags_stripper.py", line 75, in decorated_method_wrapper
kitchenowlback     |     decorated_func_result = decorated(self, *args, **kwargs)
kitchenowlback     |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/normalize_string.py", line 34, in decorated_method_wrapper
kitchenowlback     |     return normalize_string(decorated(self, *args, **kwargs))
kitchenowlback     |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/schemaorg_fill.py", line 48, in decorated_method_wrapper
kitchenowlback     |     return decorated(self, *args, **kwargs)
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/hellofresh.py", line 14, in title
kitchenowlback     |     return self.schema.title()
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/_schemaorg.py", line 103, in title
kitchenowlback     |     return normalize_string(self.data.get("name"))
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/_utils.py", line 187, in normalize_string
kitchenowlback     |     unescaped_string = html.unescape(string)
kitchenowlback     |                        ^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/usr/local/lib/python3.11/html/__init__.py", line 130, in unescape
kitchenowlback     |     if '&' not in s:
kitchenowlback     |        ^^^^^^^^^^^^
kitchenowlback     | TypeError: argument of type 'NoneType' is not iterable
kitchenowlback     | [pid: 44|app: 0|req: 11/10] 172.26.0.2 () {64 vars in 1416 bytes} [Thu Apr  4 12:03:43 2024] POST /api/household/1/recipe/scrape => generated 20 bytes in 5333 msecs (HTTP/1.1 500) 2 headers in 98 bytes (3 switches on core 199)

My config uses image: tombursch/kitchenowl-web:latest and image: tombursch/kitchenowl:latest. In the web-frontend I got the versions grafik

Thx for your help!

TomBursch commented 6 months ago

Strange, it works for me both on my personal instance and the public one.

Karamellwuerfel commented 6 months ago

My steps to update were

  1. docker-compose pull
  2. docker-compose up --force-recreate --build -d
  3. docker image prune -f

On web-frontend the error looks like this: grafik

Browser network: grafik

Karamellwuerfel commented 5 months ago

@thatso can you confirm, imports with hellofresh are working? Try this url: https://www.hellofresh.de/recipes/torchietti-in-tomaten-paprika-sosse-5fc66ae618d71c10e64e51d4 Thanks!

I changed setup to the AIO docker-compose. Versions are the same as after my update, but it's not working for me. Maybe it could be a problem with my nginx proxy.

thatso commented 5 months ago

@Karamellwuerfel: did you mix up issues? I don't use hellofresh. :roll_eyes:

Karamellwuerfel commented 5 months ago

@Karamellwuerfel: did you mix up issues? I don't use hellofresh. 🙄

Oh, I'm sorry!

Karamellwuerfel commented 5 months ago

Is it maybe a server configuration issue?

02:~$ ping hellofresh.de
PING hellofresh.de (54.72.116.110) 56(84) bytes of data.

--- hellofresh.de ping statistics ---
56 packets transmitted, 0 received, 100% packet loss, time 56297ms
02:~$ curl hellofresh.de
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
</body>
</html>

02:~$ curl www.hellofresh.de
<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title>Attention Required! | Cloudflare</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" /><![endif]-->
<style>body{margin:0;padding:0}</style>

<!--[if gte IE 10]><!-->
<script>
  if (!navigator.cookieEnabled) {
    window.addEventListener('DOMContentLoaded', function () {
      var cookieEl = document.getElementById('cookie-alert');
      cookieEl.style.display = 'block';
    })
  }
</script>
<!--<![endif]-->

</head>

Can someone help me on cloudflare authentication issue?

TomBursch commented 5 months ago

Good to know, this has nothing to do with your setup nor with KitchenOwl. Hello fresh is using Cloudflare which, to combat dos attacks, can block bots (which KitchenOwl is when scraping the website). It seems like they flagged your KitchenOwl instance.