hay-kot
commented
2 years ago Tracking in
-
853
Closed jmighion closed 2 years ago
hay-kot
commented
2 years ago Tracking in
jmighion
commented
2 years ago The latest version 0.5.4 fixes most of the import problems, but the image doesn't come through.
From the debugger :
{
"@context": "http://schema.org",
"@type": [
"Recipe"
],
"headline": "Dan Dan Noodles Recipe",
"datePublished": "2011-04-13T11:10:00.000-04:00",
"dateModified": "2020-05-06T07:50:11.000-04:00",
"author": [
{
"@type": "Person",
"name": "J. Kenji López-Alt",
"description": "Kenji is the former culinary director for Serious Eats and a current culinary consultant for the site. He is also a New York Times food columnist and the author of The Food Lab: Better Home Cooking Through Science.",
"jobTitle": "Culinary Consultant",
"url": "https://www.seriouseats.com/j-kenji-l-pez-alt-5118720",
"sameAs": [
"https://www.instagram.com/kenjilopezalt/",
"http://www.kenjilopezalt.com/"
],
"knowsAbout": [
"Food science, Recipe development, Equipment reviews, Restaurant reviews , General food features"
]
}
],
"description": "This recipes is part of Chinese Restaurant Appetizer Week Perhaps the biggest key to making excellent Dan Dan Noodles is to make your own roasted chili oil. When done right, it get a rich, fruity, smoky flavor that none of...",
"image": [
{
"@type": "ImageObject",
"url": "https://www.seriouseats.com/thmb/sVd3x4SokJuDyoLhUt4nyPj04d4=/610x343/smart/filters:no_upscale()/__opt__aboutcom__coeus__resources__content_migration__serious_eats__seriouseats.com__recipes__images__20110406-chinese-appetizers-dan-dan-noodle-5-2af33cd5faa24bb7919bce845f012c7e.jpg",
"height": 343,
"width": 610
},
{
"@type": "ImageObject",
"url": "https://www.seriouseats.com/thmb/PrnoGBg3BEhnPhwCzoJNA-xzvD4=/610x458/smart/filters:no_upscale()/__opt__aboutcom__coeus__resources__content_migration__serious_eats__seriouseats.com__recipes__images__20110406-chinese-appetizers-dan-dan-noodle-5-2af33cd5faa24bb7919bce845f012c7e.jpg",
"height": 458,
"width": 610
},
{
"@type": "ImageObject",
"url": "https://www.seriouseats.com/thmb/fvjNutfgzh69UrN-l1daP6nGYZw=/458x458/smart/filters:no_upscale()/__opt__aboutcom__coeus__resources__content_migration__serious_eats__seriouseats.com__recipes__images__20110406-chinese-appetizers-dan-dan-noodle-5-2af33cd5faa24bb7919bce845f012c7e.jpg",
"height": 458,
"width": 458
}
],
"publisher": {
"@type": "Organization",
"name": "Serious Eats",
"url": "https://www.seriouseats.com",
"logo": {
"@type": "ImageObject",
"url": "https://www.seriouseats.com/thmb/O7qOTYhS4IAKSKnr4JmNDURwjhw=/320x320/filters:no_upscale():max_bytes(150000):strip_icc()/Serious_Eats_Schema_Logo-033d1e058bdb4c8d9e0ada84a4485482.png",
"width": 320,
"height": 320
},
"brand": "Serious Eats",
"publishingPrinciples": "https://www.seriouseats.com/about-us-5120006#editorial-guidelines"
},
"name": "Dan Dan Noodles Recipe",
"aggregateRating": {
"@type": "AggregateRating",
"ratingValue": "4.4",
"ratingCount": "5"
},
"keywords": "appetizers, Asian, Chinese Appetizer Recipe Week, Dan Dan noodles, noodle, sichuan",
"recipeCategory": [
"Appetizers and Hors d'Oeuvres"
],
"recipeCuisine": [
"Chinese"
],
"recipeIngredient": [
"For the Roasted Chili Oil Vinaigrette:",
"1/2 cup roasted chili oil (see note)",
"3 tablespoons Chinkiang vinegar (see note)",
"1 tablespoon soy sauce",
"1 tablespoon sugar",
"2 cloves of garlic, grated on a microplane grater",
"To Serve:",
"12 ounces fresh chinese noodles (or 6 ounces dried noodles)",
"2 teaspoons vegetable oil",
"2 ounces ground pork",
"2 tablespoons finely chopped preserve Sichuan Vegetable ( zha cai , see note)",
"2 ounces roasted peanuts, lightly crushed in a mortar and pestle",
"1 tablespoon Sichuan peppercorns, finely ground in a mortar and pestle",
"2 cloves garlic, grated on a microplane",
"2 tablespoons finely sliced scallion greens"
],
"recipeInstructions": [
{
"@type": "HowToStep",
"text": "For the Roasted Chili Vinaigrette : Combine all vinaigrette ingredients in a bowl and stir vigorously to mix."
},
{
"@type": "HowToStep",
"text": "To Finish : Bring a large pot of salted water to a boil. Add noodles and cook according to package directions. Drain. While noodles are cooking, heat oil in a wok or a small skillet over high heat until smoking. Add pork and preserved vegetable and cook, stirring and shaking constantly, using spatule or spoon to break up pork until cooked through, about 1 minute. Transfer to a small bowl and set aside."
},
{
"@type": "HowToStep",
"text": "Transfer noodles to serving bowl and top with pork mixture. Stir vinaigrette and spoon over and around the noodles (you may not want to use all of it). Sprinkle with roasted peanuts, sichuan peppercorn, grated garlic, and scallion greens. Serve immediately."
}
],
"totalTime": "PT30M",
"review": [
{
"@type": "Review",
"author": {
"@type": "Person",
"name": "westernNyChef"
},
"reviewBody": "There is indeed a typo in this recipe! According to the serious eats video, there should be 6 oz of pork. This just makes sense - 2 oz is not nearly enough.\n\nhttps://www.youtube.com/watch?v=uhYEy3PA0RI"
},
{
"@type": "Review",
"author": {
"@type": "Person",
"name": "Anonymous"
},
"reviewBody": "Wheres the sesame paste???"
},
{
"@type": "Review",
"author": {
"@type": "Person",
"name": "kskal"
},
"reviewBody": "\"whole Chinese chilis\" fresh or dried?"
},
{
"@type": "Review",
"author": {
"@type": "Person",
"name": "Kungpow"
},
"reviewBody": "perhaps a drop or two of sesame oil and some chopped cilantro?"
},
{
"@type": "Review",
"author": {
"@type": "Person",
"name": "Laserlaw"
},
"reviewBody": "Phenomenal. Thanks!"
}
],
"mainEntityOfPage": {
"@type": [
"WebPage"
],
"@id": "https://www.seriouseats.com/dan-dan-noodles-recipe",
"breadcrumb": {
"@type": "BreadcrumbList",
"itemListElement": [
{
"@type": "ListItem",
"position": 1,
"item": {
"@id": "https://www.seriouseats.com/recipes-by-ingredient-recipes-5117749",
"name": "Recipes By Ingredients"
}
},
{
"@type": "ListItem",
"position": 2,
"item": {
"@id": "https://www.seriouseats.com/noodle-recipes-5117610",
"name": "Noodles"
}
},
{
"@type": "ListItem",
"position": 3,
"item": {
"@id": "https://www.seriouseats.com/dan-dan-noodles-recipe",
"name": "Dan Dan Noodles Recipe"
}
}
]
}
},
"about": []
}Logs:
ERROR: 04-Dec-21 05:31:14 Error parsing recipe func_call for 'performTime'
ERROR: 04-Dec-21 05:31:14 Error parsing recipe func_call for 'cookTime'
<class 'str'>
Dan Dan Noodles Recipe
ERROR: 04-Dec-21 05:31:14 Error parsing recipe func_call for 'image'
ERROR: 04-Dec-21 05:31:14 Error parsing recipe func_call for 'description'
<class 'str'>
This recipes is part of Chinese Restaurant Appetizer Week Perhaps the biggest key to making excellent Dan Dan Noodles is to make your own roasted chili oil. When done right, it get a rich, fruity, smoky flavor that none of...
ERROR: 04-Dec-21 05:31:14 Error parsing recipe func_call for 'nutrition'
ERROR: 04-Dec-21 05:31:14 Error parsing recipe func_call for 'recipeYield'
<class 'NoneType'>
<class 'str'>
For the Roasted Chili Oil Vinaigrette:
<class 'str'>
1/2 cup roasted chili oil (see note)
<class 'str'>
3 tablespoons Chinkiang vinegar (see note)
<class 'str'>
1 tablespoon soy sauce
<class 'str'>
1 tablespoon sugar
<class 'str'>
2 cloves of garlic, grated on a microplane grater
<class 'str'>
To Serve:
<class 'str'>
12 ounces fresh chinese noodles (or 6 ounces dried noodles)
<class 'str'>
2 teaspoons vegetable oil
<class 'str'>
2 ounces ground pork
<class 'str'>
2 tablespoons finely chopped preserve Sichuan Vegetable ( zha cai , see note)
<class 'str'>
2 ounces roasted peanuts, lightly crushed in a mortar and pestle
<class 'str'>
1 tablespoon Sichuan peppercorns, finely ground in a mortar and pestle
<class 'str'>
2 cloves garlic, grated on a microplane
<class 'str'>
2 tablespoons finely sliced scallion greens
INFO: 04-Dec-21 05:31:14 Scraped Instructions: (Type: <class 'str'>)
For the Roasted Chili Vinaigrette : Combine all vinaigrette ingredients in a bowl and stir vigorously to mix.
To Finish : Bring a large pot of salted water to a boil. Add noodles and cook according to package directions. Drain. While noodles are cooking, heat oil in a wok or a small skillet over high heat until smoking. Add pork and preserved vegetable and cook, stirring and shaking constantly, using spatule or spoon to break up pork until cooked through, about 1 minute. Transfer to a small bowl and set aside.
Transfer noodles to serving bowl and top with pork mixture. Stir vinaigrette and spoon over and around the noodles (you may not want to use all of it). Sprinkle with roasted peanuts, sichuan peppercorn, grated garlic, and scallion greens. Serve immediately.
<class 'str'>
For the Roasted Chili Vinaigrette : Combine all vinaigrette ingredients in a bowl and stir vigorously to mix.
<class 'str'>
For the Roasted Chili Vinaigrette : Combine all vinaigrette ingredients in a bowl and stir vigorously to mix.
<class 'str'>
To Finish : Bring a large pot of salted water to a boil. Add noodles and cook according to package directions. Drain. While noodles are cooking, heat oil in a wok or a small skillet over high heat until smoking. Add pork and preserved vegetable and cook, stirring and shaking constantly, using spatule or spoon to break up pork until cooked through, about 1 minute. Transfer to a small bowl and set aside.
<class 'str'>
To Finish : Bring a large pot of salted water to a boil. Add noodles and cook according to package directions. Drain. While noodles are cooking, heat oil in a wok or a small skillet over high heat until smoking. Add pork and preserved vegetable and cook, stirring and shaking constantly, using spatule or spoon to break up pork until cooked through, about 1 minute. Transfer to a small bowl and set aside.
<class 'str'>
Transfer noodles to serving bowl and top with pork mixture. Stir vinaigrette and spoon over and around the noodles (you may not want to use all of it). Sprinkle with roasted peanuts, sichuan peppercorn, grated garlic, and scallion greens. Serve immediately.
<class 'str'>
Transfer noodles to serving bowl and top with pork mixture. Stir vinaigrette and spoon over and around the noodles (you may not want to use all of it). Sprinkle with roasted peanuts, sichuan peppercorn, grated garlic, and scallion greens. Serve immediately.
INFO: 04-Dec-21 05:31:14 Cleaned Instructions: (Type: <class 'list'>)
[{'text': 'For the Roasted Chili Vinaigrette : Combine all vinaigrette ingredients in a bowl and stir vigorously to mix.'}, {'text': 'To Finish : Bring a large pot of salted water to a boil. Add noodles and cook according to package directions. Drain. While noodles are cooking, heat oil in a wok or a small skillet over high heat until smoking. Add pork and preserved vegetable and cook, stirring and shaking constantly, using spatule or spoon to break up pork until cooked through, about 1 minute. Transfer to a small bowl and set aside.'}, {'text': 'Transfer noodles to serving bowl and top with pork mixture. Stir vinaigrette and spoon over and around the noodles (you may not want to use all of it). Sprinkle with roasted peanuts, sichuan peppercorn, grated garlic, and scallion greens. Serve immediately.'}]
ERROR: 04-Dec-21 05:31:14 Error parsing recipe func_call for 'totalTime'
ERROR: 04-Dec-21 05:31:14 Error parsing recipe func_call for 'prepTime'
INFO: 04-Dec-21 05:31:14 Image [{'@type': 'ImageObject', 'url': 'https://www.seriouseats.com/thmb/sVd3x4SokJuDyoLhUt4nyPj04d4=/610x343/smart/filters:no_upscale()/__opt__aboutcom__coeus__resources__content_migration__serious_eats__seriouseats.com__recipes__images__20110406-chinese-appetizers-dan-dan-noodle-5-2af33cd5faa24bb7919bce845f012c7e.jpg', 'height': 343, 'width': 610}, {'@type': 'ImageObject', 'url': 'https://www.seriouseats.com/thmb/PrnoGBg3BEhnPhwCzoJNA-xzvD4=/610x458/smart/filters:no_upscale()/__opt__aboutcom__coeus__resources__content_migration__serious_eats__seriouseats.com__recipes__images__20110406-chinese-appetizers-dan-dan-noodle-5-2af33cd5faa24bb7919bce845f012c7e.jpg', 'height': 458, 'width': 610}, {'@type': 'ImageObject', 'url': 'https://www.seriouseats.com/thmb/fvjNutfgzh69UrN-l1daP6nGYZw=/458x458/smart/filters:no_upscale()/__opt__aboutcom__coeus__resources__content_migration__serious_eats__seriouseats.com__recipes__images__20110406-chinese-appetizers-dan-dan-noodle-5-2af33cd5faa24bb7919bce845f012c7e.jpg', 'height': 458, 'width': 458}]
INFO: 04-Dec-21 05:31:14 Image URL: [{'@type': 'ImageObject', 'url': 'https://www.seriouseats.com/thmb/sVd3x4SokJuDyoLhUt4nyPj04d4=/610x343/smart/filters:no_upscale()/__opt__aboutcom__coeus__resources__content_migration__serious_eats__seriouseats.com__recipes__images__20110406-chinese-appetizers-dan-dan-noodle-5-2af33cd5faa24bb7919bce845f012c7e.jpg', 'height': 343, 'width': 610}, {'@type': 'ImageObject', 'url': 'https://www.seriouseats.com/thmb/PrnoGBg3BEhnPhwCzoJNA-xzvD4=/610x458/smart/filters:no_upscale()/__opt__aboutcom__coeus__resources__content_migration__serious_eats__seriouseats.com__recipes__images__20110406-chinese-appetizers-dan-dan-noodle-5-2af33cd5faa24bb7919bce845f012c7e.jpg', 'height': 458, 'width': 610}, {'@type': 'ImageObject', 'url': 'https://www.seriouseats.com/thmb/fvjNutfgzh69UrN-l1daP6nGYZw=/458x458/smart/filters:no_upscale()/__opt__aboutcom__coeus__resources__content_migration__serious_eats__seriouseats.com__recipes__images__20110406-chinese-appetizers-dan-dan-noodle-5-2af33cd5faa24bb7919bce845f012c7e.jpg', 'height': 458, 'width': 458}]
ERROR: 04-Dec-21 05:31:14 Image {url} could not be requested
Traceback (most recent call last):
File "/app/mealie/services/image/image.py", line 58, in scrape_image
r = requests.get(url, stream=True, headers={"User-Agent": _FIREFOX_UA})
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/sessions.py", line 649, in send
adapter = self.get_adapter(url=request.url)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/sessions.py", line 742, in get_adapter
raise InvalidSchema("No connection adapters were found for {!r}".format(url))
requests.exceptions.InvalidSchema: No connection adapters were found for "{'@type': 'ImageObject', 'url': 'https://www.seriouseats.com/thmb/sVd3x4SokJuDyoLhUt4nyPj04d4=/610x343/smart/filters:no_upscale()/__opt__aboutcom__coeus__resources__content_migration__serious_eats__seriouseats.com__recipes__images__20110406-chinese-appetizers-dan-dan-noodle-5-2af33cd5faa24bb7919bce845f012c7e.jpg', 'height': 343, 'width': 610}"
ERROR: 04-Dec-21 05:31:14 Image {url} could not be requested
Traceback (most recent call last):
File "/app/mealie/services/image/image.py", line 58, in scrape_image
r = requests.get(url, stream=True, headers={"User-Agent": _FIREFOX_UA})
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/sessions.py", line 649, in send
adapter = self.get_adapter(url=request.url)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/sessions.py", line 742, in get_adapter
raise InvalidSchema("No connection adapters were found for {!r}".format(url))
requests.exceptions.InvalidSchema: No connection adapters were found for "{'@type': 'ImageObject', 'url': 'https://www.seriouseats.com/thmb/PrnoGBg3BEhnPhwCzoJNA-xzvD4=/610x458/smart/filters:no_upscale()/__opt__aboutcom__coeus__resources__content_migration__serious_eats__seriouseats.com__recipes__images__20110406-chinese-appetizers-dan-dan-noodle-5-2af33cd5faa24bb7919bce845f012c7e.jpg', 'height': 458, 'width': 610}"
ERROR: 04-Dec-21 05:31:14 Image {url} could not be requested
Traceback (most recent call last):
File "/app/mealie/services/image/image.py", line 58, in scrape_image
r = requests.get(url, stream=True, headers={"User-Agent": _FIREFOX_UA})
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/sessions.py", line 649, in send
adapter = self.get_adapter(url=request.url)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/sessions.py", line 742, in get_adapter
raise InvalidSchema("No connection adapters were found for {!r}".format(url))
requests.exceptions.InvalidSchema: No connection adapters were found for "{'@type': 'ImageObject', 'url': 'https://www.seriouseats.com/thmb/fvjNutfgzh69UrN-l1daP6nGYZw=/458x458/smart/filters:no_upscale()/__opt__aboutcom__coeus__resources__content_migration__serious_eats__seriouseats.com__recipes__images__20110406-chinese-appetizers-dan-dan-noodle-5-2af33cd5faa24bb7919bce845f012c7e.jpg', 'height': 458, 'width': 458}"
ERROR: 04-Dec-21 05:31:14 Fatal Image Request Exception
Traceback (most recent call last):
File "/app/mealie/services/image/image.py", line 76, in scrape_image
r = requests.get(image_url, stream=True, headers={"User-Agent": _FIREFOX_UA})
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/sessions.py", line 528, in request
prep = self.prepare_request(req)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/sessions.py", line 456, in prepare_request
p.prepare(
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/models.py", line 316, in prepare
self.prepare_url(url, params)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/requests/models.py", line 390, in prepare_url
raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL '': No schema supplied. Perhaps you meant http://?
ERROR: 04-Dec-21 05:31:14 Error Scraping Image: 'NoneType' object has no attribute 'name'
10.0.2.100:0 - "POST /api/recipes/create-url HTTP/1.1" 201
/app/data/recipes/dan-dan-noodles-recipe/images/min-original.webpThe url for the image in the json and logs work in a browser. Not sure why they were not able to be imported.
cadamswaite
commented
2 years ago So the image scraper expects a list of strings, e.g.
"image": [
"https://www.seriouseats.com/thmb/sVd3x4SokJuDyoLhUt4nyPj04d4",
"https://www.seriouseats.com/thmb/fvjNutfgzh69UrN-l1daP6nGYZw",
...
]and picks the best image from the list. Seriouseats is giving a dictionary, which is perfectly ok according to the standard.
I did a quick PR, linked above, that should extract the URL from the dictionary
First Check
What is the issue you are experiencing?
I'm attempting to add a recipe via a URL : https://www.seriouseats.com/dan-dan-noodles-recipe
Version : docker.io/hkotel/mealie:latest (0.5.3)
Logs :
I saw a couple other bugs similar to this #792 and #791. https://github.com/hay-kot/mealie/issues/791#issuecomment-965734815 had me test with the debugger and got :
I checked the recipe scraper repo and found https://github.com/hhursev/recipe-scrapers/issues/450. Looks like they may have run into this same problem with a site change to seriouseats.com and it may have been resolved. If so, when would that be in the next mealie container?
Deployment
Docker (Linux)
Deployment Details
No response
Mealie Version
0.5.3