search

tmo1 / sms-ie

SMS Import / Export is a simple Android app that imports and exports SMS and MMS messages, call logs, and contacts from and to JSON / NDJSON files.
GNU General Public License v3.0
312 stars 38 forks source link

Error importing messages #152

Closed DavidHenryThoreau closed 5 months ago

DavidHenryThoreau commented 5 months ago

I've got this error on 

Android : 14 UKQ1.230917.001
Error importing messages

An error was encountered while importing messages :

"java.lan.illegalArgumentException: The non-sync-callers AND non-blocked-url should not specify DELETED for inserting."

See logcat for more information.
tmo1 commented 5 months ago

Thanks for the report. This is a known issue with MIUI (documented in the README) - please read the discussion here, and follow-up there.

tmo1 commented 5 months ago

Duplicate of #103

DavidHenryThoreau commented 5 months ago

Thanks for reply but I'm getting error and messages-nnn-MIUI-stripped.json is empty

jq: error (at messages.ndjson:18575): Cannot index string with string "deleted"
jq: error (at messages.ndjson:18576): Cannot index string with string "deleted"
jq: error (at messages.ndjson:18577): Cannot index string with string "deleted"
jq: error (at messages.ndjson:18578): Cannot index string with string "deleted"
jq: error (at messages.ndjson:18579): Cannot index string with string "deleted"
jq: error (at messages.ndjson:18580): Cannot index string with string "deleted"
jq: error (at messages.ndjson:18581): Cannot index string with string "deleted"
jq: error (at messages.ndjson:18576): Cannot index string with string "display_name"
jq: error (at messages.ndjson:18577): Cannot index string with string "display_name"
jq: error (at messages.ndjson:18578): Cannot index string with string "display_name"
jq: error (at messages.ndjson:18579): Cannot index string with string "display_name"
jq: error (at messages.ndjson:18580): Cannot index string with string "display_name"
jq: error (at messages.ndjson:18581): Cannot index string with string "display_name"
tmo1 commented 5 months ago

Please post some of the (redacted) lines that are causing the error, and I'll take a look.

DavidHenryThoreau commented 5 months ago 

{"_id":"18","thread_id":"14","date":"1620739770","date_sent":"1620730823","msg_box":"1","read":"1","m_id":"148C2480-B248-71EB-8618-0050569C3D99","ct_t":"application\/vnd.wap.multipart.related","ct_l":"http:\/\/2x.orange.fr\/mmsc?XCPPKPOREREJNa","m_cls":"personal","m_type":"132","v":"18","m_size":"594830","pri":"129","tr_id":"TOTOTO","retr_st":"128","retr_txt":"1000:Ok","retr_txt_cs":"106","d_rpt":"129","locked":"0","seen":"1","timed":"0","date_ms_part":"0","deleted":"0","sync_state":"0","marker":"0","bind_id":"0","mx_status":"0","need_download":"0","out_time":"0","preview_type":"0","preview_data_ts":"0","sim_id":"1","block_type":"0","advanced_seen":"3","mx_type":"0","favorite_date":"0","text_only":"0","__sender_address":{"_id":"22","msg_id":"18","address":"+33970580490","type":"137","charset":"106","__display_name":"TTTïs"},"__recipient_addresses":[{"_id":"23","msg_id":"18","address":"+33878235890","type":"151","charset":"106","__display_name":"oi"}],"__parts":[{"_id":"40","mid":"18","seq":"-1","ct":"application\/smil","name":"smil.smil","chset":"106","cd":"attachment","cid":"<smil.smil>","cl":"smil.smil","text":"<smil><head><layout><root-layout\/>\n<region 
id=\"Image\" fit=\"meet\" top=\"0\" left=\"0\" height=\"80%\" 
width=\"100%\"\/>\n<region id=\"Text\" top=\"80%\" left=\"0\" 
height=\"20%\" width=\"100%\"\/>\n<\/layout>\n<\/head>\n<body><par 
dur=\"5000ms\"><img src=\"image0000001.jpg\" 
region=\"Image\"\/>\n<\/par>\n<par dur=\"5000ms\"><text 
src=\"text0000021.txt\" 
region=\"Text\"\/>\n<\/par>\n<\/body>\n<\/smil>\n"},{"_id":"41","mid":"18","seq":"0","ct":"image\/jpeg","name":"image0000001.jpg","cd":"attachment","cid":"<image0000001.jpg>","cl":"image0000001.jpg","_data":"\/data\/user_de\/0\/com.android.providers.telephony\/app_parts\/PART_1620739770387_image0000001.jpg"},{"_id":"42","mid":"18","seq":"0","ct":"text\/plain","name":"text0000021.txt","chset":"106","cd":"attachment","cid":"<text0000021.txt>","cl":"text0000021.txt","text":"Elle 
a bien ! "}]}
tmo1 commented 5 months ago

This doesn't seem to be valid (ND)JSON. Are you sure that this is exactly what SMS I/E output?

DavidHenryThoreau commented 5 months ago

Yes, I've opened with text editor and copy/paste line 18575 and anonymize some parts.

tmo1 commented 5 months ago

I think that there may be something wrong with your copying and pasting - perhaps you added or removed spacing or line breaks? Please make sure to copy and paste exactly. To verify that you've done it correctly, make sure it validates as JSON.

[Edit: spelling]

DavidHenryThoreau commented 4 months ago

Steps I've done :

DavidHenryThoreau commented 4 months ago

{ seems to match }

tmo1 commented 4 months ago

Again, I need actual NDJSON output from the app to troubleshoot this. Please make you're posting the exact output, and please check that it validates as NDJSON, as above.

DavidHenryThoreau commented 3 months ago

I've ident it and it's ok for https://jsonlint.com/

{
    "_id": "18",
    "thread_id": "14",
    "date": "1620739770",
    "date_sent": "1620730823",
    "msg_box": "1",
    "read": "1",
    "m_id": "148C2480-B248-71EB-8618-0050569C3D99",
    "ct_t": "application\/vnd.wap.multipart.related",
    "ct_l": "http:\/\/2x.orange.fr\/mmsc?XCPPKPOREREJNa",
    "m_cls": "personal",
    "m_type": "132",
    "v": "18",
    "m_size": "594830",
    "pri": "129",
    "tr_id": "TOTOTO",
    "retr_st": "128",
    "retr_txt": "1000:Ok",
    "retr_txt_cs": "106",
    "d_rpt": "129",
    "locked": "0",
    "seen": "1",
    "timed": "0",
    "date_ms_part": "0",
    "deleted": "0",
    "sync_state": "0",
    "marker": "0",
    "bind_id": "0",
    "mx_status": "0",
    "need_download": "0",
    "out_time": "0",
    "preview_type": "0",
    "preview_data_ts": "0",
    "sim_id": "1",
    "block_type": "0",
    "advanced_seen": "3",
    "mx_type": "0",
    "favorite_date": "0",
    "text_only": "0",
    "__sender_address": {
        "_id": "22",
        "msg_id": "18",
        "address": "+33970580490",
        "type": "137",
        "charset": "106",
        "__display_name": "TTTïs"
    },
    "__recipient_addresses": [
        {
            "_id": "23",
            "msg_id": "18",
            "address": "+33878235890",
            "type": "151",
            "charset": "106",
            "__display_name": "oi"
        }
    ],
    "__parts": [
        {
            "_id": "40",
            "mid": "18",
            "seq": "-1",
            "ct": "application\/smil",
            "name": "smil.smil",
            "chset": "106",
            "cd": "attachment",
            "cid": "<smil.smil>",
            "cl": "smil.smil",
            "text": "<smil><head><layout><root-layout\/>\n<region id=\"Image\" fit=\"meet\" top=\"0\" left=\"0\" height=\"80%\" width=\"100%\"\/>\n<region id=\"Text\" top=\"80%\" left=\"0\" height=\"20%\" width=\"100%\"\/>\n<\/layout>\n<\/head>\n<body><par dur=\"5000ms\"><img src=\"image0000001.jpg\" region=\"Image\"\/>\n<\/par>\n<par dur=\"5000ms\"><text src=\"text0000021.txt\" region=\"Text\"\/>\n<\/par>\n<\/body>\n<\/smil>\n"
        },
        {
            "_id": "41",
            "mid": "18",
            "seq": "0",
            "ct": "image\/jpeg",
            "name": "image0000001.jpg",
            "cd": "attachment",
            "cid": "<image0000001.jpg>",
            "cl": "image0000001.jpg",
            "_data": "\/data\/user_de\/0\/com.android.providers.telephony\/app_parts\/PART_1620739770387_image0000001.jpg"
        },
        {
            "_id": "42",
            "mid": "18",
            "seq": "0",
            "ct": "text\/plain",
            "name": "text0000021.txt",
            "chset": "106",
            "cd": "attachment",
            "cid": "<text0000021.txt>",
            "cl": "text0000021.txt",
            "text": "Elle a bien ! "
        }
    ]
}
tmo1 commented 3 months ago

Okay - the problem is that the jq invocations that I provided in issue #103 were designed for the old v1 format. For the current v2 format, try the following invocations:

~$ jq 'del(.deleted)' messages.ndjson > some_dir/messages.ndjson
~$ jq 'del(. ["deleted", "sync_state"])' messages.ndjson > some_dir/messages.ndjson

Edit: corrections

DavidHenryThoreau commented 3 months ago

The two commands don't return any errors :) What's the next step ?

tmo1 commented 3 months ago

What's the next step ?

If the original export file did not contain any binary data parts (i.e., there is no data/ directory in the Zipfile), then just navigate to some_dir and run:

~$ zip messages-fixed.zip messages.njdson

If there are binary parts, then copy the data/ directory into some_dir, navigate to some_dir, and run:

~$ zip -r messages-fixed.zip messages.ndjson data/

In either case, the created messages-fixed.zip should now be able to be imported by SMS I/E.

Edit: corrections

DavidHenryThoreau commented 3 months ago

These two commands return a JSON file not a NDJSON : 

~$ jq 'del(.deleted)' messages.json > some_dir/messages.json
~$ jq 'del(. ["deleted", "sync_state"])' messages.json > some_dir/messages.json

I renamed 

mv messages.json messages.ndjson

I've zip them like you mentioned :

zip -r messages-fixed.zip messages.ndjson data/

I'm getting this error on MIUI 

can't find messages.ndjson please make sure that the provided file is a zip in the correct format
tmo1 commented 3 months ago

These two commands return a JSON file not a NDJSON :

You're right - I corrected the jq invocations above.

I renamed ...

I've zip them like you mentioned ...

Looks right.

I'm getting this error on MIUI ...

What does the Zipfile look like? (unzip -l messages-fixed.zip).

DavidHenryThoreau commented 3 months ago

I've tried 

~$ jq 'del(.deleted)' messages.ndjson > some_dir/messages.ndjson
~$ jq 'del(. ["deleted", "sync_state"])' messages.ndjson > some_dir/messages.ndjson

The messages.ndjson is formated like 

{
    "_id": "18",
    "thread_id": "14",
    "date": "1620739770",
    "date_sent": "1620730823",
    "msg_box": "1",
    "read": "1",
    "m_id": "148C2480-B248-71EB-8618-0050569C3D99",
    "ct_t": "application\/vnd.wap.multipart.related",
    "ct_l": "http:\/\/2x.orange.fr\/mmsc?XCPPKPOREREJNa",
    "m_cls": "personal",
    "m_type": "132",
    "v": "18",
    "m_size": "594830",
    "pri": "129",
    "tr_id": "TOTOTO",
    "retr_st": "128",
    "retr_txt": "1000:Ok",
    "retr_txt_cs": "106",
    "d_rpt": "129",
    "locked": "0",
    "seen": "1",
    "timed": "0",
    "date_ms_part": "0",
    "deleted": "0",
    "sync_state": "0",
    "marker": "0",
    "bind_id": "0",
    "mx_status": "0",
    "need_download": "0",
    "out_time": "0",
    "preview_type": "0",
    "preview_data_ts": "0",
    "sim_id": "1",
    "block_type": "0",
    "advanced_seen": "3",
    "mx_type": "0",
    "favorite_date": "0",
    "text_only": "0",
    "__sender_address": {
        "_id": "22",
        "msg_id": "18",
        "address": "+33970580490",
        "type": "137",
        "charset": "106",
        "__display_name": "TTTïs"
    },
    "__recipient_addresses": [
        {
            "_id": "23",
            "msg_id": "18",
            "address": "+33878235890",
            "type": "151",
            "charset": "106",
            "__display_name": "oi"
        }
    ],
    "__parts": [
        {
            "_id": "40",
            "mid": "18",
            "seq": "-1",
            "ct": "application\/smil",
            "name": "smil.smil",
            "chset": "106",
            "cd": "attachment",
            "cid": "<smil.smil>",
            "cl": "smil.smil",
            "text": "<smil><head><layout><root-layout\/>\n<region id=\"Image\" fit=\"meet\" top=\"0\" left=\"0\" height=\"80%\" width=\"100%\"\/>\n<region id=\"Text\" top=\"80%\" left=\"0\" height=\"20%\" width=\"100%\"\/>\n<\/layout>\n<\/head>\n<body><par dur=\"5000ms\"><img src=\"image0000001.jpg\" region=\"Image\"\/>\n<\/par>\n<par dur=\"5000ms\"><text src=\"text0000021.txt\" region=\"Text\"\/>\n<\/par>\n<\/body>\n<\/smil>\n"
        },
        {
            "_id": "41",
            "mid": "18",
            "seq": "0",
            "ct": "image\/jpeg",
            "name": "image0000001.jpg",
            "cd": "attachment",
            "cid": "<image0000001.jpg>",
            "cl": "image0000001.jpg",
            "_data": "\/data\/user_de\/0\/com.android.providers.telephony\/app_parts\/PART_1620739770387_image0000001.jpg"
        },
        {
            "_id": "42",
            "mid": "18",
            "seq": "0",
            "ct": "text\/plain",
            "name": "text0000021.txt",
            "chset": "106",
            "cd": "attachment",
            "cid": "<text0000021.txt>",
            "cl": "text0000021.txt",
            "text": "Elle a bien ! "
        }
    ]
}

so it's not a NDJSON ?

The 

unzip -l messages-fixed.zip

returns a data folder and a messages.ndjson

I've tried to import it to MIUI i'm getting and other values instead of sim_id it's blinking 

Org.jsonException value sim_id
DavidHenryThoreau commented 4 weeks ago

Can you reproduce that error ?

tmo1 commented 3 weeks ago

I'll take a look when I get a chance.