mealie-recipes / mealie

Mealie is a self hosted recipe manager and meal planner with a RestAPI backend and a reactive frontend application built in Vue for a pleasant user experience for the whole family. Easily add recipes into your database by providing the url and mealie will automatically import the relevant data or add a family recipe with the UI editor
https://docs.mealie.io
GNU Affero General Public License v3.0
7.2k stars 723 forks source link

Migration from nextcloud fails with error 'NoneType' object is not iterable #2427

Closed dfense23 closed 1 year ago

dfense23 commented 1 year ago

First Check

What is the issue you are experiencing?

I try to migrate from Tandoor. I exported all recipes in nextcloud format and uploaded the zip to Mealie. After clicking import I get the following error on every recipe: 'NoneType' object is not iterable

I manually checked one recipe for malformed json, but have not found an issue (only replaced url:null with url:"", just in case, this lead to the same error.)

I did not find any additional info (or error) in the log.

Any help would be appreciated and I hope this is the right place to raise this.

Deployment

Docker (Synology)

Deployment Details

No response

Mealie Version

0.5.6

michael-genson commented 1 year ago

If you're on version 0.5.6, that's no longer being updated, I recommend migrating to the latest 1.0.0 version: https://nightly.mealie.io/documentation/getting-started/migrating-to-mealie-v1/

Try upgrading and see if the issue persists, if it's still a problem we can get it fixed!

dfense23 commented 1 year ago

Thanks for the quick answer. I installed the beta and was able to import the nextcloud recipes, allthough without tags and without pictures (not a complaint, just an info).

michael-genson commented 1 year ago

Glad to hear it worked!

without tags and without pictures

Hmm, we should definitely take a look at that then, we should be bringing those over (assuming the data is there)

dfense23 commented 1 year ago

Ok, I did some more testing. I imported my recipes once via nextcloud, once via paprika:

Below is a list what works(checked), and does not work (unchecked):

Paprika:

Nextcloud:

Everything else looks good in both (Ingredients, Prep) My source includes everything, I checked in the export files.

The times are a bit wonky. In the Paprika app I have Prep 15min, Cook 1 hour in one example. This ends up being just 1 hour Cook Time via Paprika Export and via Nextcloud 15 min Prep and 16 min Total (assume taking 1 hour as 1 minute).

Hope this helps. Again, not complaining, just informing (although, obviously I am a bit stuck migrating, I do not need nutritional info, but the other three are rather important to me....)

michael-genson commented 1 year ago

Thanks for this. Are you able to share some or all of your recipe exports? I'll see if I can take a look at it this week and having samples that aren't working fully definitely helps

dfense23 commented 1 year ago

Of course, happy to, I am not doing the heavy lifting here ;-)

A good test case is 3-Ingredient Chipotle-Lime Grilled Steak (thats the one I used for above info)

The two files are not identical, the nextcloud one has more recipes, as I migrated from Paprika to Tandoor and added a couple there. Also, it is a horrible mix between American (language and units) and German (languge and sane units ;-) ). But at least it makes a good test case...

Let me know if I can help in any way.

recipe_paprika_export_direct_from_app.zip recipe_nexctcloud_export_from_tandoor.zip

michael-genson commented 1 year ago

I was able to fix the paprika issues (PR coming later), though I probably can't parse the nutritional info without a lot more effort due to the format it exports in.

For Nextcloud, you noted that categories aren't imported. Did you mean tags, or does Nextcloud actually have categories? In the data I see "keywords" but I'm guessing those should be tags rather than categories

michael-genson commented 1 year ago

I was also able to partially fix Nextcloud, but due to its data structure things come out a bit weird. I recommend using the Paprika exports instead. For instance:

The cook times are wrong: image image

The serving size has no unit, so it's just a bare number: image image

And the instructions aren't using the JSON schema correctly, instead putting everything in one step: image image

Since that's a data issue in the export there's not much we can do about it. Meanwhile, Paprika looks pretty good, minus the missing nutritional info (note in this recipe the description is blank, otherwise we would import it):

image image image

(Not pictured, the original URL is there too now, whereas its missing in the Nextcloud data, instead just sitting in the instructions for some reason)

I'll wait for your feedback, then open a PR fixing the Paprika issues (and some small issues with Nextcloud, though as I stated above the data just isn't that great)

dfense23 commented 1 year ago

Wow, thanks a lot. I think the Paprika way is reasonable and works for me, I can fogure out the missing recipes and reenter them manually. I personally do not care about the nutritional info, so no problem for me. Important to me are picture, tags, and source url. Also, just to clarify, the source was Tandoor, not Nextcloud. Nextcloud was just the only format match between Tandoor and Mealie, so the source might be off due to the Nextcloud format implementation in Tandoor. (If you want to provide a migration path from Tandoor to Mealie is up to you and a whole other discussion ;-) )

Question: Paprika has only categories, but just because the name is the same, I do not think it should be imported into categories in mealie. Categories and Tools are much more specific than Tags (this distinction I like a lot regarding mealie).

So, I would propose to dump all Paprika "categories" in the less-specific "tags" in Mealie. Then from there it would be possible to re-organize into tools and categories. E.G. In Paprika, with only the Categories available, tags like "bbq", "lunch", "French" and "easy" are all lumped together in categories (although, Paprika has hierarchical categories, which helps). In mealie, the "bbq" tag could then become an equipment, "lunch" a categorie and the rest stays in "tags". I would not expect functionality to do that, I could do this myself (manually, through the API or in the DB (already tested, it works)). It's just that starting from categories makes this more complicated and higher effort (lots of tags, far less categories and tools).

michael-genson commented 1 year ago

Paprika has only categories, but just because the name is the same, I do not think it should be imported into categories in mealie

Yeah if Paprika only has support for categories and they're used like tags, and doesn't have support for tags, I think this makes sense. I did the same thing with CopyMeThat. I can make that change

If you want to provide a migration path from Tandoor to Mealie is up to you and a whole other discussion ;-)

Do you have an export to share? I could definitely take a look. The way that we have migrations set up in Mealie makes it pretty easy to add new sources, provided we can read and map the data properly

dfense23 commented 1 year ago

Sure, I made sure two recipes are clean and the major fields are populated. No nutritional info though.

I exported the "default" format from Tandoor. Have at it...

Sorry, I am no expert in github, is the PR finished? Or what is the status I need to lookout for to pull a fresh container to get the fix? Also, no pressure, but do you have a timeline for the Tandoor import? If it becomes available soon, I would use this to migrate to mealie, if not, I take the Paprika route and add the missing recipes manually. No pressure, just a planing question ;-)

tandoor_default_export_2023-06-29.zip

michael-genson commented 1 year ago

I exported the "default" format from Tandoor

Could you take a look and confirm the data is in there? I'm getting a blank zip file when I try to look at it.

what is the status I need to lookout for to pull a fresh container to get the fix?

Once #2434 is merged (it will say "Merged" in the upper left and the color will change to purple; right now it's green and says "Open". This issue will also automatically be closed so you'll probably get a notification/email) a new nightly build will automatically start being created. Usually takes a while, last build took 4 hours.

You can either watch Discord (there will be an automated notification in the "nightly" channel) or check the Docker Nightly Production actions (if you can see them, I can't remember if GitHub makes those available to everyone or just contributors).

do you have a timeline for the Tandoor import

No idea, I'll try to look at that once I'm able to open it! Could be quick, could take a while, could be completely impossible, who knows. More likely to be fast though, I think.

dfense23 commented 1 year ago

Sorry, that's the only one I did not check... So, I am not able to export single, or filter based partial recipe collections from Tandoor (either empty, or just no download at all). I can only do a full export of all >500 recipes. The recipes are numbered and individually zipped. The two certainly good ones are 492 and 475. I did not want to risk the integrity of the file by manually editing/deleting after exporting, so here is the whole thing. tandoor_default_export_full_2023-06-29.zip

No idea, I'll try to look at that once I'm able to open it! Could be quick, could take a while, could be completely impossible, who knows. More likely to be fast though, I think.

Sure, I did not mean right know, just would apperciate if you let me know if it's deemed impossible after a quick glance, then I don't have to wait.

michael-genson commented 1 year ago

That one worked, thanks! Looks like it should be easy to get this working. I had a bit of time to spare so I started on it, we'll see if I can get a PR in today or tomorrow. If not, barring anything unexpected, I should have a PR in by the end of the weekend at the latest

michael-genson commented 1 year ago

Seems to be working: image

I did some extra work to make the ingredient amounts look nice, including sections when available: image

I added "hours" and "minutes" to the times as needed: image

Each section was one step. Is this correct? I'm not familiar enough with Tandoor. If it's wrong, should we be splitting steps by newlines? image

Tags: image

Also, I noticed none of these have the original URL in them. Are they missing from the export? I don't see them in the data for the two recipes you suggested I look at. Might be an issue with Tandoor's export feature, if so.

dfense23 commented 1 year ago

Wow, looks great! Sorry for the late answer, I was travelling. This is what it looks like in Tandoor: image

Your interpertation of the two steps is correct, the text of the recipe is somewhat inconsistent with the numbererd list in the second part. As you can see, the source is in Tandoor, but you are right, it is missing in the source. Too bad...

michael-genson commented 1 year ago

I opened a PR on Tandoor to see if we can get it added, but I'm not sure how often new versions are deployed for Tandoor, etc, so it could take a while. In the meantime I've added the mapping in Mealie so once the data is in the export it should import.

I have a bit more manual testing on the Mealie side to make sure the changes are good to go, then I'll open a PR and keep you updated!

michael-genson commented 1 year ago

Looks like it was merged last night - not sure if there's a develop version of Tandoor or if you need to wait until the next release, but those exports should contain the URL now

michael-genson commented 1 year ago

PR for the Tandoor import: https://github.com/hay-kot/mealie/pull/2438 Once that's merged and a new nightly image is built, you can update your nightly instance and see the Tandoor migration option.

Of course the missing source URL will still be a problem if you're unable to update your Tandoor instance and generate a new export