Migration from nextcloud fails with error 'NoneType' object is not iterable

[dfense23]

First Check

• [X] This is not a feature request
• [X] I added a very descriptive title to this issue.
• [X] I used the GitHub search to find a similar issue and didn't find it.
• [X] I searched the Mealie documentation, with the integrated search.
• [X] I already read the docs and didn't find an answer.
• [X] I have checked for existing issues that have been resolved in v1-beta

What is the issue you are experiencing?

I try to migrate from Tandoor. I exported all recipes in nextcloud format and uploaded the zip to Mealie. After clicking import I get the following error on every recipe: 'NoneType' object is not iterable

I manually checked one recipe for malformed json, but have not found an issue (only replaced url:null with url:"", just in case, this lead to the same error.)

I did not find any additional info (or error) in the log.

Any help would be appreciated and I hope this is the right place to raise this.

Deployment

Docker (Synology)

Deployment Details

No response

Mealie Version

0.5.6

[michael-genson]

If you're on version 0.5.6, that's no longer being updated, I recommend migrating to the latest 1.0.0 version: https://nightly.mealie.io/documentation/getting-started/migrating-to-mealie-v1/

Try upgrading and see if the issue persists, if it's still a problem we can get it fixed!

[dfense23]

Thanks for the quick answer. I installed the beta and was able to import the nextcloud recipes, allthough without tags and without pictures (not a complaint, just an info).

[michael-genson]

Glad to hear it worked!

without tags and without pictures

Hmm, we should definitely take a look at that then, we should be bringing those over (assuming the data is there)

[dfense23]

Ok, I did some more testing. I imported my recipes once via nextcloud, once via paprika:

Below is a list what works(checked), and does not work (unchecked):

Paprika:

• [x] Pictures
• [x] Categories
• [ ] Source URL
• [ ] Nutritional Info
• [x] Cooking times (somewhat, see below)
• [x] Servings

Nextcloud:

• [ ] Pictures
• [ ] Categories
• [x] Source URL
• [x] Nutritional Info
• [ ] Cooking times
• [ ] Servings

Everything else looks good in both (Ingredients, Prep) My source includes everything, I checked in the export files.

The times are a bit wonky. In the Paprika app I have Prep 15min, Cook 1 hour in one example. This ends up being just 1 hour Cook Time via Paprika Export and via Nextcloud 15 min Prep and 16 min Total (assume taking 1 hour as 1 minute).

Hope this helps. Again, not complaining, just informing (although, obviously I am a bit stuck migrating, I do not need nutritional info, but the other three are rather important to me....)

[michael-genson]

Thanks for this. Are you able to share some or all of your recipe exports? I'll see if I can take a look at it this week and having samples that aren't working fully definitely helps

[dfense23]

Of course, happy to, I am not doing the heavy lifting here ;-)

A good test case is 3-Ingredient Chipotle-Lime Grilled Steak (thats the one I used for above info)

The two files are not identical, the nextcloud one has more recipes, as I migrated from Paprika to Tandoor and added a couple there. Also, it is a horrible mix between American (language and units) and German (languge and sane units ;-) ). But at least it makes a good test case...

Let me know if I can help in any way.

recipe_paprika_export_direct_from_app.zip recipe_nexctcloud_export_from_tandoor.zip

[michael-genson]

I was able to fix the paprika issues (PR coming later), though I probably can't parse the nutritional info without a lot more effort due to the format it exports in.

For Nextcloud, you noted that categories aren't imported. Did you mean tags, or does Nextcloud actually have categories? In the data I see "keywords" but I'm guessing those should be tags rather than categories

[michael-genson]

I was also able to partially fix Nextcloud, but due to its data structure things come out a bit weird. I recommend using the Paprika exports instead. For instance:

The cook times are wrong:

The serving size has no unit, so it's just a bare number:

And the instructions aren't using the JSON schema correctly, instead putting everything in one step:

Since that's a data issue in the export there's not much we can do about it. Meanwhile, Paprika looks pretty good, minus the missing nutritional info (note in this recipe the description is blank, otherwise we would import it):

(Not pictured, the original URL is there too now, whereas its missing in the Nextcloud data, instead just sitting in the instructions for some reason)

I'll wait for your feedback, then open a PR fixing the Paprika issues (and some small issues with Nextcloud, though as I stated above the data just isn't that great)

[dfense23]

Wow, thanks a lot. I think the Paprika way is reasonable and works for me, I can fogure out the missing recipes and reenter them manually. I personally do not care about the nutritional info, so no problem for me. Important to me are picture, tags, and source url. Also, just to clarify, the source was Tandoor, not Nextcloud. Nextcloud was just the only format match between Tandoor and Mealie, so the source might be off due to the Nextcloud format implementation in Tandoor. (If you want to provide a migration path from Tandoor to Mealie is up to you and a whole other discussion ;-) )

Question: Paprika has only categories, but just because the name is the same, I do not think it should be imported into categories in mealie. Categories and Tools are much more specific than Tags (this distinction I like a lot regarding mealie).

So, I would propose to dump all Paprika "categories" in the less-specific "tags" in Mealie. Then from there it would be possible to re-organize into tools and categories. E.G. In Paprika, with only the Categories available, tags like "bbq", "lunch", "French" and "easy" are all lumped together in categories (although, Paprika has hierarchical categories, which helps). In mealie, the "bbq" tag could then become an equipment, "lunch" a categorie and the rest stays in "tags". I would not expect functionality to do that, I could do this myself (manually, through the API or in the DB (already tested, it works)). It's just that starting from categories makes this more complicated and higher effort (lots of tags, far less categories and tools).

[michael-genson]

Paprika has only categories, but just because the name is the same, I do not think it should be imported into categories in mealie

Yeah if Paprika only has support for categories and they're used like tags, and doesn't have support for tags, I think this makes sense. I did the same thing with CopyMeThat. I can make that change

If you want to provide a migration path from Tandoor to Mealie is up to you and a whole other discussion ;-)

Do you have an export to share? I could definitely take a look. The way that we have migrations set up in Mealie makes it pretty easy to add new sources, provided we can read and map the data properly

[dfense23]

Sure, I made sure two recipes are clean and the major fields are populated. No nutritional info though.

I exported the "default" format from Tandoor. Have at it...

Sorry, I am no expert in github, is the PR finished? Or what is the status I need to lookout for to pull a fresh container to get the fix? Also, no pressure, but do you have a timeline for the Tandoor import? If it becomes available soon, I would use this to migrate to mealie, if not, I take the Paprika route and add the missing recipes manually. No pressure, just a planing question ;-)

tandoor_default_export_2023-06-29.zip

[michael-genson]

I exported the "default" format from Tandoor

Could you take a look and confirm the data is in there? I'm getting a blank zip file when I try to look at it.

what is the status I need to lookout for to pull a fresh container to get the fix?

Once #2434 is merged (it will say "Merged" in the upper left and the color will change to purple; right now it's green and says "Open". This issue will also automatically be closed so you'll probably get a notification/email) a new nightly build will automatically start being created. Usually takes a while, last build took 4 hours.

You can either watch Discord (there will be an automated notification in the "nightly" channel) or check the Docker Nightly Production actions (if you can see them, I can't remember if GitHub makes those available to everyone or just contributors).

do you have a timeline for the Tandoor import

No idea, I'll try to look at that once I'm able to open it! Could be quick, could take a while, could be completely impossible, who knows. More likely to be fast though, I think.

[dfense23]

Sorry, that's the only one I did not check... So, I am not able to export single, or filter based partial recipe collections from Tandoor (either empty, or just no download at all). I can only do a full export of all >500 recipes. The recipes are numbered and individually zipped. The two certainly good ones are 492 and 475. I did not want to risk the integrity of the file by manually editing/deleting after exporting, so here is the whole thing. tandoor_default_export_full_2023-06-29.zip

No idea, I'll try to look at that once I'm able to open it! Could be quick, could take a while, could be completely impossible, who knows. More likely to be fast though, I think.

Sure, I did not mean right know, just would apperciate if you let me know if it's deemed impossible after a quick glance, then I don't have to wait.

[michael-genson]

That one worked, thanks! Looks like it should be easy to get this working. I had a bit of time to spare so I started on it, we'll see if I can get a PR in today or tomorrow. If not, barring anything unexpected, I should have a PR in by the end of the weekend at the latest

[michael-genson]

Seems to be working:

I did some extra work to make the ingredient amounts look nice, including sections when available:

I added "hours" and "minutes" to the times as needed:

Each section was one step. Is this correct? I'm not familiar enough with Tandoor. If it's wrong, should we be splitting steps by newlines?

Tags:

Also, I noticed none of these have the original URL in them. Are they missing from the export? I don't see them in the data for the two recipes you suggested I look at. Might be an issue with Tandoor's export feature, if so.

[dfense23]

Wow, looks great! Sorry for the late answer, I was travelling. This is what it looks like in Tandoor:

Your interpertation of the two steps is correct, the text of the recipe is somewhat inconsistent with the numbererd list in the second part. As you can see, the source is in Tandoor, but you are right, it is missing in the source. Too bad...

[michael-genson]

I opened a PR on Tandoor to see if we can get it added, but I'm not sure how often new versions are deployed for Tandoor, etc, so it could take a while. In the meantime I've added the mapping in Mealie so once the data is in the export it should import.

I have a bit more manual testing on the Mealie side to make sure the changes are good to go, then I'll open a PR and keep you updated!

[michael-genson]

Looks like it was merged last night - not sure if there's a develop version of Tandoor or if you need to wait until the next release, but those exports should contain the URL now

[michael-genson]

PR for the Tandoor import: https://github.com/hay-kot/mealie/pull/2438 Once that's merged and a new nightly image is built, you can update your nightly instance and see the Tandoor migration option.

Of course the missing source URL will still be a problem if you're unable to update your Tandoor instance and generate a new export