search

mealie-recipes / mealie

Mealie is a self hosted recipe manager and meal planner with a RestAPI backend and a reactive frontend application built in Vue for a pleasant user experience for the whole family. Easily add recipes into your database by providing the url and mealie will automatically import the relevant data or add a family recipe with the UI editor
https://docs.mealie.io
GNU Affero General Public License v3.0
6.87k stars 708 forks source link

Mass import of recipes using the API causes mealie to crash #791

Closed robertmain closed 2 years ago

robertmain commented 2 years ago

First Check

What is the issue you are experiencing?

I'm migrating from recipes stored in google calendar to Mealie and I'm using the API to bulk insert recipes into the system. The recipes I'm trying to bulk insert as as follows:

[
  'http://fitcakery.com/home/2013/9/30/caprese-crepes',
  'http://littlespicejar.com/one-pot-greek-chicken-and-rice-pilaf/',
  'http://thetoastykitchen.com/spring-vegetable-soup/?utm_medium=social&utm_source=pinterest&utm_campaign=tailwind_tribes&utm_content=tribes&utm_term=638688072_24819740_11553',
  'http://wholeandheavenlyoven.com/2015/04/02/springtime-veggie-pasta-primavera-2/',
  'http://www.thepreservesproject.com/asparagus-lemon-risotto/',
  'https://diethood.com/chicken-ratatouille-recipe/',
  'https://healthyrecipesflatley.blogspot.com/2019/04/easy-vegetable-crustless-quiche-dairy.html',
  'https://jenelizabethsjournals.com/2015/02/10/one-pot-pasta/',
  'https://jenelizabethsjournals.com/2015/02/10/one-pot-pasta/',
  'https://mycrazygoodlife.com/healthy-tuscan-chicken-pasta-instant-pot-slow-cooker-stovetop/',
  'https://mycrazygoodlife.com/instant-pot-21-day-fix-burrito-bowl/',
  'https://ohsweetbasil.com/quick-and-easy-lemon-orzo-with-parmesan-and-peas-recipe/',
  'https://pinchofyum.com/creamy-chicken-quinoa-broccoli-casserole',
  'https://pinchofyum.com/creamy-chicken-quinoa-broccoli-casserole',
  'https://showmetheyummy.com/healthy-mexican-casserole/',
  'https://slimsanity.com/healthy-broccoli-chicken-casserole/',
  'https://thesaltymarshmallow.com/best-easy-instant-pot-chili/',
  'https://www.acouplecooks.com/creamy-goat-cheese-pasta/',
  'https://www.allrecipes.com/recipe/222137/healthier-ultimate-twice-baked-potatoes/',
  'https://www.ambitiouskitchen.com/healthy-chicken-pot-pie/',
  'https://www.ambitiouskitchen.com/roasted-butternut-squash-broccoli-cheddar-chicken-couscous/',
  'https://www.bbc.com/food/recipes/basictomatoandbasils_67840',
  'https://www.bhg.com/recipe/poultry/fast-chicken-fettuccine/',
  'https://www.bonappetit.com/recipe/risotto-with-butternut-squash-leeks-and-basil',
  'https://www.chelseasmessyapron.com/sweet-potato-burritos/',
  'https://www.chelseasmessyapron.com/the-best-ever-chicken-fajita-bowls/',
  'https://www.crazyforcrust.com/mexican-chicken-soup/',
  'https://www.delish.com/cooking/recipe-ideas/recipes/a54291/one-pan-balsamic-chicken-and-asparagus-recipe/',
  'https://www.dessertfortwo.com/instant-pot-chicken-tacos/',
  'https://www.dessertfortwo.com/instant-pot-salsa-chicken/',
  'https://www.dessertfortwo.com/instant-pot-salsa-chicken/',
  'https://www.dinneratthezoo.com/spaghetti-salad/',
  'https://www.eatyourselfskinny.com/sweet-potato-black-bean-quinoa-bake/',
  'https://www.geniuskitchen.com/recipe/savory-mushroom-spinach-cheese-crepes-373390?soc=socialsharingpinterest',
  'https://www.gimmesomeoven.com/cucumber-quinoa-salad-recipe/',
  'https://www.healthymealplans.com/recipe-details/classic-one-pot-pasta',
  'https://www.healthymealplans.com/recipe-details/lazy-lasagna-bake',
  'https://www.healthymealplans.com/recipe-details/turkey-and-wild-rice-soup',
  'https://www.lecremedelacrumb.com/easy-healthy-baked-lemon-chicken/',
  'https://www.lecremedelacrumb.com/instant-pot-pot-roast-potatoes/',
  'https://www.loveandoliveoil.com/2007/09/spicy-salsa-turkey-burgers.html',
  'https://www.loveandoliveoil.com/2015/02/chicken-tortilla-soup.html',
  'https://www.purewow.com/recipes/chicken-snap-pea-stir-fry-recipe',
  'https://www.purewow.com/recipes/skillet-pasta-squash-ricotta-basil-recipe',
  'https://www.recipegirl.com/quinoa-stuffed-peppers/',
  'https://www.recipegirl.com/quinoa-stuffed-peppers/',
  'https://www.simplyhappyfoodie.com/instant-pot-mini-sweet-potato-chili/',
  'https://www.skinnytaste.com/baked-potato-soup/',
  'https://www.skinnytaste.com/butternut-squash-and-spinach-lasagna/',
  'https://www.skinnytaste.com/crust-less-summer-zucchini-pie/',
  'https://www.skinnytaste.com/lighter-baked-macaroni-and-cheese/#recipe',
  'https://www.skinnytaste.com/skinny-baked-broccoli-macaroni-and/',
  'https://www.skinnytaste.com/spinach-lasagna-rolls/',
  'https://www.skinnytaste.com/stuffed-pepper-soup/',
  'https://www.tablefortwoblog.com/quinoa-bowls-with-roasted-vegetables-and-chicken/',
  'https://www.tasteofhome.com/recipes/butternut-portobello-lasagna/',
  'https://www.tasteofhome.com/recipes/linguine-with-broccoli-rabe-peppers/',
  'https://www.thekitchn.com/recipe-broccoli-and-cheese-risotto-256187',
  'https://www.theroastedroot.net/roasted-winter-vegetable-quinoa-salad-with-cider-vinaigrette/',
  'https://www.wellplated.com/caprese-chicken-pasta/',
];

I'm less familiar with python than other languages, but watching the logs seems to suggest that the app is crashing and then restarting when I try to do this. I had been trying to insert them in parallel, and when that failed, I tried inserting smaller chunks in parallel (i.e: rather than try to insert the whole 60-odd at once, try to just do 10 or 20 at once). This also seemed to crash the app.

I've provided logging output below:

mealie      | 172.18.0.1:0 - "GET /api/users/1/image HTTP/1.1" 404
mealie      | 172.18.0.1:0 - "POST /api/recipes/test-scrape-url HTTP/1.1" 500
mealie      | 172.18.0.1:0 - "POST /api/recipes/test-scrape-url HTTP/1.1" 500
mealie      | [2021-11-07 21:04:33 +0000] [53] [CRITICAL] WORKER TIMEOUT (pid:54)
postgres    | 2021-11-07 21:04:34.032 UTC [111] LOG:  unexpected EOF on client connection with an open transaction
postgres    | 2021-11-07 21:04:34.032 UTC [110] LOG:  unexpected EOF on client connection with an open transaction
postgres    | 2021-11-07 21:04:34.032 UTC [107] LOG:  unexpected EOF on client connection with an open transaction
postgres    | 2021-11-07 21:04:34.032 UTC [106] LOG:  unexpected EOF on client connection with an open transaction
postgres    | 2021-11-07 21:04:34.032 UTC [108] LOG:  unexpected EOF on client connection with an open transaction
postgres    | 2021-11-07 21:04:34.032 UTC [105] LOG:  unexpected EOF on client connection with an open transaction
postgres    | 2021-11-07 21:04:34.032 UTC [103] LOG:  unexpected EOF on client connection with an open transaction
postgres    | 2021-11-07 21:04:34.033 UTC [77] LOG:  unexpected EOF on client connection with an open transaction
mealie      | {"level":"error","ts":1636319074.0374625,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47096","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["103"],"Connection":["close"],"Accept":["application/json, text/plain, */*"]}},"duration":78.440422503,"status":502,"err_id":"utcf1vfbg","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.038025,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47116","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["100"],"Connection":["close"]}},"duration":78.440582671,"status":502,"err_id":"a7iybzgvv","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.038714,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47192","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["66"],"Connection":["close"]}},"duration":78.438832845,"status":502,"err_id":"vjndydieb","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0389664,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47134","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["73"],"Connection":["close"]}},"duration":78.440944764,"status":502,"err_id":"nn7yyzzka","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0376565,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47094","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["94"],"Connection":["close"]}},"duration":78.441045853,"status":502,"err_id":"akxzyjk64","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.038347,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47152","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["67"],"Connection":["close"]}},"duration":78.43956618,"status":502,"err_id":"3m068nw6b","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0395017,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47098","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Connection":["close"],"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["94"]}},"duration":78.442685807,"status":502,"err_id":"tij9envgk","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.038413,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47114","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Connection":["close"],"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["74"]}},"duration":78.44083602,"status":502,"err_id":"c9yygvyam","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0377998,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47108","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Connection":["close"],"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["69"]}},"duration":78.440477335,"status":502,"err_id":"q1dhmzigg","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0388863,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47120","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Content-Length":["91"],"Connection":["close"],"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"]}},"duration":78.441132429,"status":502,"err_id":"7g4f2desg","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0405805,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47136","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["68"],"Connection":["close"],"Accept":["application/json, text/plain, */*"]}},"duration":78.442265041,"status":502,"err_id":"2w83acvks","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
postgres    | 2021-11-07 21:04:34.033 UTC [93] LOG:  unexpected EOF on client connection with an open transaction
postgres    | 2021-11-07 21:04:34.033 UTC [75] LOG:  unexpected EOF on client connection with an open transaction
mealie      | {"level":"error","ts":1636319074.04097,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47164","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Connection":["close"],"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["69"]}},"duration":78.441803297,"status":502,"err_id":"8v9v02ai9","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0402243,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47176","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["80"],"Connection":["close"],"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"]}},"duration":78.440749253,"status":502,"err_id":"rg2b0qh4e","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0402658,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47156","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Connection":["close"],"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["102"]}},"duration":78.441378347,"status":502,"err_id":"njp08czym","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0407612,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47160","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["69"],"Connection":["close"]}},"duration":78.441752464,"status":502,"err_id":"qdd3syue7","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0411646,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47140","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Connection":["close"],"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["70"]}},"duration":78.442756676,"status":502,"err_id":"1zr8dzpac","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0412703,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47144","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["64"],"Connection":["close"]}},"duration":78.442714752,"status":502,"err_id":"qfxnmtd0i","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.041322,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47148","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["91"],"Connection":["close"]}},"duration":78.442617608,"status":502,"err_id":"9436uv0bt","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0382607,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47104","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["69"],"Connection":["close"],"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"]}},"duration":78.441103539,"status":502,"err_id":"mb1ibvufg","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0388443,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47130","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Connection":["close"],"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["63"]}},"duration":78.440710199,"status":502,"err_id":"nams8bsh4","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0389254,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47124","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["73"],"Connection":["close"]}},"duration":78.441063132,"status":502,"err_id":"7e3hgnb0s","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.041384,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47168","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["89"],"Connection":["close"]}},"duration":78.442086059,"status":502,"err_id":"ed6mcuyda","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0451965,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47090","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["94"],"Connection":["close"],"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"]}},"duration":78.448391609,"status":502,"err_id":"j0b1rxee8","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0414903,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47180","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"User-Agent":["axios/0.21.1"],"Content-Length":["61"],"Connection":["close"],"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"]}},"duration":78.44190391,"status":502,"err_id":"aiv4mw50k","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0415394,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47184","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Connection":["close"],"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["115"]}},"duration":78.441830131,"status":502,"err_id":"0z1iemj7p","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0415812,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47172","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["67"],"Connection":["close"],"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"]}},"duration":78.442223342,"status":502,"err_id":"vd8hxcjr5","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0416012,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47196","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["66"],"Connection":["close"]}},"duration":78.441642512,"status":502,"err_id":"jnu9ikwzw","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0416565,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47198","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["57"],"Connection":["close"]}},"duration":78.441571774,"status":502,"err_id":"rqwybdfuf","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.0453818,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47084","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["94"],"Connection":["close"],"Accept":["application/json, text/plain, */*"]}},"duration":78.449564434,"status":502,"err_id":"e6b4xd7w2","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
mealie      | {"level":"error","ts":1636319074.041441,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"172.18.0.1:47188","proto":"HTTP/1.1","method":"POST","host":"localhost:9090","uri":"/api/recipes/test-scrape-url","headers":{"Connection":["close"],"Accept":["application/json, text/plain, */*"],"Authorization":["Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb25nX3Rva2VuIjp0cnVlLCJpZCI6MSwiZXhwIjoxNzkzOTk4OTc2fQ.9cpR_foNofxceHA5BF7zGpK_a2CC8AtTCVKGlpf3E4g"],"Content-Type":["application/json;charset=utf-8"],"User-Agent":["axios/0.21.1"],"Content-Length":["66"]}},"duration":78.441575885,"status":502,"err_id":"18mgadrgd","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
postgres    | 2021-11-07 21:04:34.034 UTC [88] LOG:  unexpected EOF on client connection with an open transaction
postgres    | 2021-11-07 21:04:34.035 UTC [112] LOG:  unexpected EOF on client connection with an open transaction
postgres    | 2021-11-07 21:04:34.036 UTC [104] LOG:  unexpected EOF on client connection with an open transaction
postgres    | 2021-11-07 21:04:34.039 UTC [109] LOG:  unexpected EOF on client connection with an open transaction
mealie      | [2021-11-07 21:04:34 +0000] [53] [WARNING] Worker with pid 54 was terminated due to signal 6
mealie      | [2021-11-07 21:04:34 +0000] [182] [INFO] Booting worker with pid: 182
mealie      | [2021-11-07 21:04:34 +0000] [182] [INFO] Started server process [182]
mealie      | [2021-11-07 21:04:34 +0000] [182] [INFO] Waiting for application startup.
mealie      | INFO: 07-Nov-21 21:04:34  Adding job tentatively -- it will be properly scheduled when the scheduler starts
mealie      | INFO: 07-Nov-21 21:04:34  Adding job tentatively -- it will be properly scheduled when the scheduler starts
mealie      | INFO: 07-Nov-21 21:04:34  ----INIT SCHEDULE OBJECT-----
mealie      | INFO: 07-Nov-21 21:04:34  Adding job tentatively -- it will be properly scheduled when the scheduler starts
postgres    | 2021-11-07 21:04:34.400 UTC [71] WARNING:  there is already a transaction in progress
mealie      | INFO: 07-Nov-21 21:04:34  Adding job tentatively -- it will be properly scheduled when the scheduler starts
mealie      | Pending jobs:
mealie      |     purge_events_database (trigger: interval[1 day, 0:00:00], pending)
mealie      |     update_webhook_schedule (trigger: interval[0:30:00], pending)
mealie      |     backups (trigger: cron[hour='0', minute='0'], pending)
mealie      |     Home (trigger: cron[hour='0', minute='0'], pending)
mealie      | INFO: 07-Nov-21 21:04:34  None
mealie      | INFO: 07-Nov-21 21:04:34  Added job "purge_events_database" to job store "default"
mealie      | INFO: 07-Nov-21 21:04:34  Added job "update_webhook_schedule" to job store "default"
mealie      | INFO: 07-Nov-21 21:04:34  Added job "backups" to job store "default"
mealie      | INFO: 07-Nov-21 21:04:34  Added job "Home" to job store "default"
mealie      | INFO: 07-Nov-21 21:04:34  Scheduler started
mealie      | INFO: 07-Nov-21 21:04:34  -----SYSTEM STARTUP----- 
mealie      | 
mealie      | INFO: 07-Nov-21 21:04:34  ------APP SETTINGS------
mealie      | INFO: 07-Nov-21 21:04:34  {
mealie      |     "PRODUCTION": true,
mealie      |     "BASE_URL": "http://localhost:8080",
mealie      |     "IS_DEMO": false,
mealie      |     "API_PORT": 9000,
mealie      |     "API_DOCS": true,
mealie      |     "DB_ENGINE": "postgres",
mealie      |     "POSTGRES_SERVER": "postgres",
mealie      |     "POSTGRES_PORT": "5432",
mealie      |     "POSTGRES_DB": "mealie",
mealie      |     "DB_URL_PUBLIC": "postgresql://*****:*****@postgres:5432/mealie",
mealie      |     "DEFAULT_GROUP": "Home",
mealie      |     "DEFAULT_EMAIL": "changeme@email.com",
mealie      |     "TOKEN_TIME": 2,
mealie      |     "RECIPE_PUBLIC": true,
mealie      |     "RECIPE_SHOW_NUTRITION": true,
mealie      |     "RECIPE_SHOW_ASSETS": true,
mealie      |     "RECIPE_LANDSCAPE_VIEW": true,
mealie      |     "RECIPE_DISABLE_COMMENTS": false,
mealie      |     "RECIPE_DISABLE_AMOUNT": false,
mealie      |     "SCHEDULER_DATABASE": "sqlite:////app/data/scheduler.db"
mealie      | }
mealie      | None
mealie      | [2021-11-07 21:04:34 +0000] [182] [INFO] Application startup complete.

Deployment

Docker (Linux)

Deployment Details

Currently running on my Debian Linux 9 laptop for dev purposes. Running via docker-compose. Compose file as follows:

version: "3.8"
services:
  mealie:
    container_name: mealie
    image: hkotel/mealie:latest
    restart: always
    ports:
      - 9090:80
    depends_on:
      - postgres
    environment:
      PUID: 1000
      PGID: 1000
      TZ: America/New York

      DB_ENGINE: postgres
      POSTGRES_USER: mealie
      POSTGRES_PASSWORD: mealie
      POSTGRES_SERVER: postgres
      POSTGRES_PORT: 5432
      POSTGRES_DB: mealie

      RECIPE_PUBLIC: 'true'
      RECIPE_SHOW_NUTRITION: 'true'
      RECIPE_SHOW_ASSETS: 'true'
      RECIPE_LANDSCAPE_VIEW: 'true'
      RECIPE_DISABLE_COMMENTS: 'false'
      RECIPE_DISABLE_AMOUNT: 'false'
    volumes:
      - ./mealie/data/:/app/data
  postgres:
    container_name: postgres
    image: postgres
    restart: always
    environment:
      POSTGRES_PASSWORD: mealie
      POSTGRES_USER: mealie
      POSTGRES_DB: mealie

Mealie Version

0.5.3

robertmain commented 2 years ago

From what I understand of the logging output above, it seems like the app is not happy with the sudden workload of trying to parse 60-off HTML pages from different website concurrently(which is understandable).

I've had similar results trying this with smaller pieces of the data too. Is there a better way to do this?

hay-kot commented 2 years ago

Can you post your script? Atleast the part that interacts with Mealie?

robertmain commented 2 years ago

Yes of course, sorry.

import { axiosInstance } from './api';

(async () => {
    const urls = [
      'http://fitcakery.com/home/2013/9/30/caprese-crepes',
      'http://littlespicejar.com/one-pot-greek-chicken-and-rice-pilaf/',
      'http://thetoastykitchen.com/spring-vegetable-soup/?utm_medium=social&utm_source=pinterest&utm_campaign=tailwind_tribes&utm_content=tribes&utm_term=638688072_24819740_11553',
      'http://wholeandheavenlyoven.com/2015/04/02/springtime-veggie-pasta-primavera-2/',
      'http://www.thepreservesproject.com/asparagus-lemon-risotto/',
      'https://diethood.com/chicken-ratatouille-recipe/',
      'https://healthyrecipesflatley.blogspot.com/2019/04/easy-vegetable-crustless-quiche-dairy.html',
      'https://jenelizabethsjournals.com/2015/02/10/one-pot-pasta/',
      'https://jenelizabethsjournals.com/2015/02/10/one-pot-pasta/',
      'https://mycrazygoodlife.com/healthy-tuscan-chicken-pasta-instant-pot-slow-cooker-stovetop/',
      'https://mycrazygoodlife.com/instant-pot-21-day-fix-burrito-bowl/',
      'https://ohsweetbasil.com/quick-and-easy-lemon-orzo-with-parmesan-and-peas-recipe/',
      'https://pinchofyum.com/creamy-chicken-quinoa-broccoli-casserole',
      'https://pinchofyum.com/creamy-chicken-quinoa-broccoli-casserole',
      'https://showmetheyummy.com/healthy-mexican-casserole/',
      'https://slimsanity.com/healthy-broccoli-chicken-casserole/',
      'https://thesaltymarshmallow.com/best-easy-instant-pot-chili/',
      'https://www.acouplecooks.com/creamy-goat-cheese-pasta/',
      'https://www.allrecipes.com/recipe/222137/healthier-ultimate-twice-baked-potatoes/',
      'https://www.ambitiouskitchen.com/healthy-chicken-pot-pie/',
      'https://www.ambitiouskitchen.com/roasted-butternut-squash-broccoli-cheddar-chicken-couscous/',
      'https://www.bbc.com/food/recipes/basictomatoandbasils_67840',
      'https://www.bhg.com/recipe/poultry/fast-chicken-fettuccine/',
      'https://www.bonappetit.com/recipe/risotto-with-butternut-squash-leeks-and-basil',
      'https://www.chelseasmessyapron.com/sweet-potato-burritos/',
      'https://www.chelseasmessyapron.com/the-best-ever-chicken-fajita-bowls/',
      'https://www.crazyforcrust.com/mexican-chicken-soup/',
      'https://www.delish.com/cooking/recipe-ideas/recipes/a54291/one-pan-balsamic-chicken-and-asparagus-recipe/',
      'https://www.dessertfortwo.com/instant-pot-chicken-tacos/',
      'https://www.dessertfortwo.com/instant-pot-salsa-chicken/',
      'https://www.dessertfortwo.com/instant-pot-salsa-chicken/',
      'https://www.dinneratthezoo.com/spaghetti-salad/',
      'https://www.eatyourselfskinny.com/sweet-potato-black-bean-quinoa-bake/',
      'https://www.geniuskitchen.com/recipe/savory-mushroom-spinach-cheese-crepes-373390?soc=socialsharingpinterest',
      'https://www.gimmesomeoven.com/cucumber-quinoa-salad-recipe/',
      'https://www.healthymealplans.com/recipe-details/classic-one-pot-pasta',
      'https://www.healthymealplans.com/recipe-details/lazy-lasagna-bake',
      'https://www.healthymealplans.com/recipe-details/turkey-and-wild-rice-soup',
      'https://www.lecremedelacrumb.com/easy-healthy-baked-lemon-chicken/',
      'https://www.lecremedelacrumb.com/instant-pot-pot-roast-potatoes/',
      'https://www.loveandoliveoil.com/2007/09/spicy-salsa-turkey-burgers.html',
      'https://www.loveandoliveoil.com/2015/02/chicken-tortilla-soup.html',
      'https://www.purewow.com/recipes/chicken-snap-pea-stir-fry-recipe',
      'https://www.purewow.com/recipes/skillet-pasta-squash-ricotta-basil-recipe',
      'https://www.recipegirl.com/quinoa-stuffed-peppers/',
      'https://www.recipegirl.com/quinoa-stuffed-peppers/',
      'https://www.simplyhappyfoodie.com/instant-pot-mini-sweet-potato-chili/',
      'https://www.skinnytaste.com/baked-potato-soup/',
      'https://www.skinnytaste.com/butternut-squash-and-spinach-lasagna/',
      'https://www.skinnytaste.com/crust-less-summer-zucchini-pie/',
      'https://www.skinnytaste.com/lighter-baked-macaroni-and-cheese/#recipe',
      'https://www.skinnytaste.com/skinny-baked-broccoli-macaroni-and/',
      'https://www.skinnytaste.com/spinach-lasagna-rolls/',
      'https://www.skinnytaste.com/stuffed-pepper-soup/',
      'https://www.tablefortwoblog.com/quinoa-bowls-with-roasted-vegetables-and-chicken/',
      'https://www.tasteofhome.com/recipes/butternut-portobello-lasagna/',
      'https://www.tasteofhome.com/recipes/linguine-with-broccoli-rabe-peppers/',
      'https://www.thekitchn.com/recipe-broccoli-and-cheese-risotto-256187',
      'https://www.theroastedroot.net/roasted-winter-vegetable-quinoa-salad-with-cider-vinaigrette/',
      'https://www.wellplated.com/caprese-chicken-pasta/',
    ];

    const scraped = urls
    .map((url) => axiosInstance.post('recipes/test-scrape-url', { url }));

    const data = await Promise.all(scraped);

    console.log(data);
})();
hay-kot commented 2 years ago

Ah. Gotcha. Since mealie, doesn't implement any sort of que you're running up against concurrent read/write errors to the database, SQLite in particular is fairly limited for concurrency.

I'd suggest just your running the script synchronously instead of it's async implementation. Await each request, and then you should be good to go.

robertmain commented 2 years ago

In this case I'm using postgres - however I wonder if I'm causing some kind of collision with the auto-increment database keys...

robertmain commented 2 years ago

So, I've refactored my script to run serially instead of concurrently. 

import { axiosInstance } from './api';

(async () => {
    const stuff = [];
    const links = [
      // .. recipes
    ];
    for (const url of links){
      console.log(`Scraping: ${url}`);
      let { data } = await axiosInstance.post('recipes/create-url', { url });
      stuff.push(data);
    }
})();

The scrape URL now works, but some aspect of the recipe/create-url endpoint seems to be breaking...

mealie      | ERROR: 07-Nov-21 23:40:02         Error parsing recipe func_call for 'totalTime'
mealie      | ERROR: 07-Nov-21 23:40:02         Error parsing recipe func_call for 'prepTime'
mealie      | INFO: 07-Nov-21 23:40:02  Image ['https://wholeandheavenlyoven.com/wp-content/uploads/2015/03/Springtime-Veggie-Pasta-Primavera10.jpg']
mealie      | INFO: 07-Nov-21 23:40:02  Image URL: ['https://wholeandheavenlyoven.com/wp-content/uploads/2015/03/Springtime-Veggie-Pasta-Primavera10.jpg']
mealie      | INFO: 07-Nov-21 23:40:03  File Name Suffix .jpg
mealie      | /app/data/recipes/springtime-veggie-pasta-primavera/images/original.jpg
mealie      | INFO: 07-Nov-21 23:40:03  original.jpg Minified: 115.67 kB -> 106.92 kB -> 15.77 kB
mealie      | 172.21.0.1:0 - "POST /api/recipes/create-url HTTP/1.1" 201
mealie      | /app/data/recipes/springtime-veggie-pasta-primavera/images/min-original.webp
mealie      | ERROR: 07-Nov-21 23:40:03         Failed to extract rdfa, raises 'str' object has no attribute 'decode'
mealie      | Traceback (most recent call last):
mealie      |   File "/opt/pysetup/.venv/lib/python3.9/site-packages/extruct/_extruct.py", line 108, in extract
mealie      |     output[syntax] = list(extract(document, base_url=base_url))
mealie      |   File "/opt/pysetup/.venv/lib/python3.9/site-packages/extruct/rdfa.py", line 154, in extract_items
mealie      |     jsonld_string = g.serialize(format='json-ld', auto_compact=not expanded).decode('utf-8')
mealie      | AttributeError: 'str' object has no attribute 'decode'
mealie      | 172.21.0.1:0 - "POST /api/recipes/create-url HTTP/1.1" 500
mealie      | [2021-11-07 23:40:03 +0000] [51] [ERROR] Exception in ASGI application
mealie      | Traceback (most recent call last):
mealie      |   File "/opt/pysetup/.venv/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi
mealie      |     result = await app(self.scope, self.receive, self.send)
mealie      |   File "/opt/pysetup/.venv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
mealie      |     return await self.app(scope, receive, send)
mealie      |   File "/opt/pysetup/.venv/lib/python3.9/site-packages/fastapi/applications.py", line 199, in __call__
mealie      |     await super().__call__(scope, receive, send)
mealie      |   File "/opt/pysetup/.venv/lib/python3.9/site-packages/starlette/applications.py", line 111, in __call__
mealie      |     await self.middleware_stack(scope, receive, send)
mealie      |   File "/opt/pysetup/.venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
mealie      |     raise exc from None
mealie      |   File "/opt/pysetup/.venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
mealie      |     await self.app(scope, receive, _send)
mealie      |   File "/opt/pysetup/.venv/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
mealie      |     raise exc from None
mealie      |   File "/opt/pysetup/.venv/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
mealie      |     await self.app(scope, receive, sender)
mealie      |   File "/opt/pysetup/.venv/lib/python3.9/site-packages/starlette/routing.py", line 566, in __call__
mealie      |     await route.handle(scope, receive, send)
mealie      |   File "/opt/pysetup/.venv/lib/python3.9/site-packages/starlette/routing.py", line 227, in handle
mealie      |     await self.app(scope, receive, send)
mealie      |   File "/opt/pysetup/.venv/lib/python3.9/site-packages/starlette/routing.py", line 41, in app
mealie      |     response = await func(request)
mealie      |   File "/opt/pysetup/.venv/lib/python3.9/site-packages/fastapi/routing.py", line 201, in app
mealie      |     raw_response = await run_endpoint_function(
mealie      |   File "/opt/pysetup/.venv/lib/python3.9/site-packages/fastapi/routing.py", line 150, in run_endpoint_function
mealie      |     return await run_in_threadpool(dependant.call, **values)
mealie      |   File "/opt/pysetup/.venv/lib/python3.9/site-packages/starlette/concurrency.py", line 34, in run_in_threadpool
mealie      |     return await loop.run_in_executor(None, func, *args)
mealie      |   File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
mealie      |     result = self.fn(*self.args, **self.kwargs)
mealie      |   File "/app/mealie/routes/recipe/recipe_crud_routes.py", line 68, in parse_recipe_url
mealie      |     recipe = create_from_url(url.url)
mealie      |   File "/app/mealie/services/scraper/scraper.py", line 35, in create_from_url
mealie      |     elif og_dict := extract_open_graph_values(url):
mealie      |   File "/app/mealie/services/scraper/scraper.py", line 59, in extract_open_graph_values
mealie      |     if recipe.get("name", "") == "":
mealie      | AttributeError: 'NoneType' object has no attribute 'get'

It managed about 3 recipes(including this one, curiously) before it gave up and returned 500 to my import script

Oddly, despite this - it seems to have managed to grab the ingredients and instructions well enough...

image

image

robertmain commented 2 years ago

Ah, it looks like the mealie API is returning a 500 for these API calls despite the apparent success in importing them...

hay-kot commented 2 years ago

It looks like you're running into issues related to your specific site as such, I'm going to close this issue since your initial problem was solved.

If you want help resolving the additional errors with importing from certain sites I'd suggest

1) Verifying that the website returns valid JSON - You can check with https://demo.mealie.io/recipes/debugger?test_url= 2) If it doesn't you can check the library we use to see if they have specific support for that site. https://github.com/hhursev/recipe-scrapers 3) If it does but you're still experiencing issues, you can open another issue with the problem URL and the corresponding logs related to the URL.