search

wedesoft / anymeal

AnyMeal is a free and open source recipe management software developed using SQLite3 and Qt5. It can manage a cookbook with more than 250,000 MealMaster recipes, thereby allowing to import, export, search, display, edit, and print them. AnyMeal is available for GNU/Linux and Microsoft Windows.
https://wedesoft.github.io/anymeal
GNU General Public License v3.0
57 stars 5 forks source link

Export fails not listed in error file #46

Closed Oldbwl closed 8 months ago

Oldbwl commented 9 months ago

I carried out an export of the entire database to create an offsite backup of my data. Some recipes failed, but I cannot identify what the errors are or what recipes not exported.

Steps:

  1. Show All recipes
  2. Choosing toolbar icon to export
  3. Message appears showing progress
  4. Export concludes showing 19 recipes failed
  5. Failed recipes do not appear in "/tmp/errors.mmf" - or indeed anywhere

As I have a 2620 recipe database a manual search without a clue to search for is a non starter. In the past when I only exported a small number I was (possibly) able to find an error - the export routine did not seem to like the following in the Description: "sauté," - the comma was significant and subsequently removed manually. Changing the region code did not help either.

Regardless of the error cause at this time, it would be helpful to identify the problem recipes to identify with certainty the reason for the fail and eliminate - be it user error or bug etc

Regards

wedesoft commented 9 months ago

Maybe you need to use UTF-8 as output encoding. The default Mealmaster output encoding is ISO-8859-1 and it will fail if a character is not part of the encoding. And you are right, it would be useful if the failing recipes are identified in some way.

Oldbwl commented 9 months ago

it failed on utf8 too., that was my reason for reporting.

On Fri, 9 Feb 2024, 18:13 Jan Wedekind, @.***> wrote:

Maybe you need to use UTF-8 as output encoding. The default Mealmaster output encoding is ISO-8859-1 and it will fail if a character is not part of the encoding. And you are right, it would be useful if the failing recipes are identified in some way.

— Reply to this email directly, view it on GitHub https://github.com/wedesoft/anymeal/issues/46#issuecomment-1936391946, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABUNSCMJBKRGCBWNNIDKU5TYSZRLHAVCNFSM6AAAAABDBMKRHKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZWGM4TCOJUGY . You are receiving this because you authored the thread.Message ID: @.***>

wedesoft commented 9 months ago

This is probably a bug with the Recode library. An identity recoding should not fail. I have release Anymeal-1.24. The new release skips recoding when input/output is UTF-8 anyway. Let me know if export still doesn't work with the new release.

wedesoft commented 9 months ago

Exporting recipes with "sauté," should work with ISO-8859-1 as well though.

Oldbwl commented 9 months ago

The UTF-8 to UTF-8 works, thank you. This is an excellent solution to the archiving and restoring the database.

I would very much like to see the UTF-8 input to ISO-8859-1 export work too. I am happy for just the list of failed recipes so that I can adjust them to read, input and export in both codeings. This is because the exported recipes are held in RecipeKeeper Android app, and that only imports in the ISO-8859-1 code.

Your turn around on this was excellent. 5*

Dan

wedesoft commented 9 months ago

Can you try to export all using UTF-8 and then use recode utf-8..iso-8859-1 test.mm? If that does not work, maybe use another command-line tool or online tool for changing the encoding.

wedesoft commented 9 months ago

Can you try to convert using iconv -f UTF-8 -t ISO-8859-1 test.mm > output.mm? If that works, one could look into switching Anymeal to libiconv.

Oldbwl commented 9 months ago

I installed recode as it was missing on my set.

I used recode -v iso.8859-1 111.mmf

and the output was

Request: ISO-8859-1..:libiconv:..CHAR Shrunk to: ISO-8859-1..CHAR Recoding 111.mmf... done

The resultant file had examples of "Sauté" now showing as "Sauté" and sometimes as "sauté‚". There were other errors.

I will now look at your next email

Dan

On Sun, 11 Feb 2024 at 13:02, Jan Wedekind @.***> wrote:

Can you try to export all using UTF-8 and then use recode utf-8..iso-8859-1 test.mm? If that does not work, maybe use another command-line tool or online tool.

— Reply to this email directly, view it on GitHub https://github.com/wedesoft/anymeal/issues/46#issuecomment-1937732563, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABUNSCL5KAQVHDMJ2P7JZLLYTC6OHAVCNFSM6AAAAABDBMKRHKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZXG4ZTENJWGM . You are receiving this because you authored the thread.Message ID: @.***>

-- Dan - Growing old disgracefully

Oldbwl commented 9 months ago

I copied your terminal code and pasted in the directory holding my file.

The response was "iconv: illegal input sequence at position 40601"

The output file was created - but truncated at character 40601 as expected. The illegal input sequence was "sauté,"

I hope this info helps

Dan

On Sun, 11 Feb 2024 at 13:14, Jan Wedekind @.***> wrote:

Can you try to convert using iconv -f UTF-8 -t ISO-8859-1 test.mm > output.mm? If that works, one could look into switching Anymeal to libiconv.

— Reply to this email directly, view it on GitHub https://github.com/wedesoft/anymeal/issues/46#issuecomment-1937750576, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABUNSCNTQGMBN4Y7M3P54CTYTC73LAVCNFSM6AAAAABDBMKRHKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZXG42TANJXGY . You are receiving this because you authored the thread.Message ID: @.***>

-- Dan - Growing old disgracefully

wedesoft commented 9 months ago

Can you email/upload the file? Creating a recipe with "sauté," doesn't cause any problems on my end (t.txt). Maybe there is an invisble character which didn't get copied in your response?

Oldbwl commented 9 months ago

OK, further testing has been done and things are officially very weird now

Using an export from the currant version 1.24 where no recode has been done shows that the comma following at least some of the words "sauté" is the illegal character. If I do a search by typing the word "sauté," it will sail right past the offending article and end up with an instance several 1000's of lines further on. If I copy the version found where the crash occurs, it will find it straight away ignoring all the rest.

Really all I need is the list of recipes which fail, and then I can edit out all the issues.

On Sun, 11 Feb 2024 at 20:11, Dan Freedman @.***> wrote:

I copied your terminal code and pasted in the directory holding my file.

The response was "iconv: illegal input sequence at position 40601"

The output file was created - but truncated at character 40601 as expected. The illegal input sequence was "sauté,"

I hope this info helps

Dan

On Sun, 11 Feb 2024 at 13:14, Jan Wedekind @.***> wrote:

Can you try to convert using iconv -f UTF-8 -t ISO-8859-1 test.mm > output.mm? If that works, one could look into switching Anymeal to libiconv.

— Reply to this email directly, view it on GitHub https://github.com/wedesoft/anymeal/issues/46#issuecomment-1937750576, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABUNSCNTQGMBN4Y7M3P54CTYTC73LAVCNFSM6AAAAABDBMKRHKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZXG42TANJXGY . You are receiving this because you authored the thread.Message ID: @.***>

-- Dan - Growing old disgracefully

-- Dan - Growing old disgracefully

Oldbwl commented 9 months ago

Here is the entire output, go to line 1479

On Sun, 11 Feb 2024 at 20:44, Jan Wedekind @.***> wrote:

Can you email/upload the file? Creating a recipe with "sauté," doesn't cause any problems on my end (t.txt https://github.com/wedesoft/anymeal/files/14233994/t.txt). Maybe there is an invisble character which didn't get copied in your response?

— Reply to this email directly, view it on GitHub https://github.com/wedesoft/anymeal/issues/46#issuecomment-1937864667, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABUNSCJLRPKNDMOMEWZLRA3YTEUUBAVCNFSM6AAAAABDBMKRHKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZXHA3DINRWG4 . You are receiving this because you authored the thread.Message ID: @.***>

-- Dan - Growing old disgracefully

wedesoft commented 9 months ago

I didn't get your attachment. Maybe it is a non-standard comma. E.g. this one: "，". Maybe I'll add functionality to write recipes failing conversion to /tmp/errors.mmf.

wedesoft commented 9 months ago

Ok, I have released version 1.25. I added functionality to write recipes which fail recoding during export to an error file.

Oldbwl commented 9 months ago

excellent, many thanks.

On Mon, 12 Feb 2024, 20:39 Jan Wedekind, @.***> wrote:

Ok, I have released version 1.25. I added functionality to write recipes which fail recoding during export to an error file.

— Reply to this email directly, view it on GitHub https://github.com/wedesoft/anymeal/issues/46#issuecomment-1939535316, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABUNSCPYN3LYGKLH6D5SHITYTJ4Y3AVCNFSM6AAAAABDBMKRHKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZZGUZTKMZRGY . You are receiving this because you authored the thread.Message ID: @.***>

wedesoft commented 8 months ago

I switched to iconv and released version 1.26. I hope this will work better since it is part of GNU libc.

Oldbwl commented 8 months ago

Good stuff. Thanks

On Fri, 23 Feb 2024, 12:43 Jan Wedekind, @.***> wrote:

I switched to iconv and released version 1.26. I hope this will work better since it is part of GNU libc.

— Reply to this email directly, view it on GitHub https://github.com/wedesoft/anymeal/issues/46#issuecomment-1961262080, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABUNSCIC57ZNXLCQ6ZVZ3DTYVCFIDAVCNFSM6AAAAABDBMKRHKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRRGI3DEMBYGA . You are receiving this because you authored the thread.Message ID: @.***>