search

suurjaak / Skyperious

Skype chat history tool
Other
350 stars 36 forks source link

Skyperious 4.7 Online sync stopping at 4/2018 #100

Closed RikuAotsuki closed 3 years ago

RikuAotsuki commented 3 years ago

I got the program to do the online sync, but for whatever reason it decides nothing exists before April 2018, when the database should go back to April 2017.

I used Microsoft's export too, and using their (awful) parser I can see that those chats are definitely there; Skyperious' sync just isn't going back that far.

As a secondary issue, Skyperious can't create a viewable database from the export .json or .tar; It parses it, says how many messages, and then when you click "Chats" to look there's nothing there. No chats, no messages.

suurjaak commented 3 years ago

Are the messages from 2017 visible in the official Skype web interface? If not, then they're just not available for live retrieval.

Secondary issue: confirmed, this is a new bug. Will be fixed in an upcoming release.

RikuAotsuki commented 3 years ago

They're visible in the program version of Skype and in the export... and yes, the browser version has it too.

Admittedly not the place but I'm not sure where to ask: in advanced options under the export chat template, how would I get it format files as Year-Month-Day Title? Can't seem to find what variables to use.

suurjaak commented 3 years ago

Can you do the following in Skyperious:

    chat = db.live.skype.chats[live.identity_to_id(page.chat["identity"])]
    for msg in chat.getMsgs(): print msg.time

Export chat template - the format parameters are chat properties. What would the Year-Month-Day be - today's date? I suppose I can add support for them.

RikuAotsuki commented 3 years ago

I can't open a chat lacking earlier messages; the chats themselves are missing. I'll run the import again and let you know how exactly it cuts off, but there's like thirty chats just totally missing from 2017.

As for the date, I should have specified; I'm referring to the chat's creation date.

suurjaak commented 3 years ago

Can you then try the following:

for id, chat in db.live.skype.chats.recent().items():
    print (id, chat.topic if hasattr(chat, "topic") else chat.user.name)

Export chat template: you can use created_datetime or first_message_datetime, but the result in filename will have time as well, e.g. 2021-06-30 123355.

RikuAotsuki commented 3 years ago

It does, yes. In fact, I don't know how it decides retrieval order, but the missing chats are mixed in with the ones it did find, like they were skipped instead of the sync just stopping early.

suurjaak commented 3 years ago

The retrieval order is decided by Skype API, ostensibly by recency, but in practice they seem to come in rather random order.

Were there any errors printed in the log window (menu Help -> Show log window) after running online sync?

Can you paste an example from the console of such a missing chat? (You can replace all user IDs with something else, so as not to expose private information here.)

RikuAotsuki commented 3 years ago

No errors in the log, no. As for the console, this particular execution returned mostly chats that didn't sync (group names abbreviated). Also, I'm trying another new online database sync (again) and it might be registering something it didn't before, so I'll update if it just randomly decided to find everything.

for id, chat in db.live.skype.chats.recent().items(): print (id, chat.topic if hasattr(chat, "topic") else chat.user.name)

(u'19:3b8096a51ced4971a3c187cfa2d2f374@thread.skype', u'SB~') (u'19:dc9241983c0e4ee8a52945003c5e6a8b@thread.skype', u'MFD') (u'19:06d92f4ef1904102b0d8031989f839e9@thread.skype', u'LF,aS') (u'19:b11e0bbc75384c97a0f4d6839bac2cc4@thread.skype', u'BB') (u'19:3ce51f1d134d4b59af965fb52a4c9230@thread.skype', u'SS') (u'19:4aadf556f2bf424b8bf97819dec82a07@thread.skype', u'AaT') (u'19:e846d9c3653a46a2917724d331c5a0b5@thread.skype', u'AaT') (u'19:dc24f73991b842a5959df2451fe37c78@thread.skype', u'TDoKK') (u'19:0c3c5afda3764255bbd16ed80d26f1f1@thread.skype', u'AL') (u'19:a6d5451693a349368609b3efb26aac1d@thread.skype', u'NJ')

suurjaak commented 3 years ago

Can you try this:


def getter():
    chatmap = db.live.skype.chats.recent()
    while chatmap:
        for c in chatmap.values():
            yield c
        chatmap = db.live.skype.chats.recent()

chats = getter()

chat = next(chats)

print chat

(You should copy-paste the first block as a whole, and the three last statements one by one)

Repeat the last two statements until you reach a missing chat, then execute the following a few times (first time might not return anything, that is normal):

chat.getMsgs()

Does it return any messages from a missing chat? If yes, can you paste that chat here as it was printed? (It should look like SkypeGroupChat(id=...) or SkypeSingleChat(id=...))

RikuAotsuki commented 3 years ago

I'm guessing I'm doing something wrong, because it seems to just loop?

def getter(): chatmap = db.live.skype.chats.recent() while chatmap: for c in chatmap.values(): yield c chatmap = db.live.skype.chats.recent() chats = getter() chat = next(chats) print chat

chat = next(chats) Traceback (most recent call last): File "", line 1, in StopIteration print chat [SkypeGroupChat] Id: 19:70f4853da85c4b3da5d8305ff7c01dd2@thread.skype Alerts: True Topic: Group Thingy CreatorId: None UserIds: AdminIds: Open: False History: False Picture: None Active: False chat = next(chats) Traceback (most recent call last): File "", line 1, in StopIteration print chat [SkypeGroupChat] Id: 19:70f4853da85c4b3da5d8305ff7c01dd2@thread.skype Alerts: True Topic: Group Thingy CreatorId: None UserIds: AdminIds: Open: False History: False Picture: None Active: False

suurjaak commented 3 years ago

Github is really messing up the formatting here.. can you attach that last data as a file?

RikuAotsuki commented 3 years ago

Sure. Like this?

console.txt

suurjaak commented 3 years ago

Looks like the copy-paste went badly. First paste this:

def getter():
    chatmap = db.live.skype.chats.recent()
    while chatmap:
        for c in chatmap.values():
            yield c
        chatmap = db.live.skype.chats.recent()

Then this:

chat = next(chats)

Then this:

print repr(chat)

Repeat the last two until you reach a missing chat, then execute

chat.getMsgs()
RikuAotsuki commented 3 years ago

Oh, that one was my bad, I think the first result it returned was a missing chat and I didn't realize it. Here's the result:

console.txt

Edit: Switched from a direct paste to a file, just in case

suurjaak commented 3 years ago

Well, at least with this chat, messages seem to be unavailable via the online API.

What is the result for some other missing chats?

RikuAotsuki commented 3 years ago

What does that mean, exactly?

And as far as I can tell, it's stuck on that one. repeating the second and third commands just brings that chat up again and again.

suurjaak commented 3 years ago

Then something still went sideways in the copy-paste. The steps in detail:

def getter():
    chatmap = db.live.skype.chats.recent()
    while chatmap:
        for c in chatmap.values():
            yield c
        chatmap = db.live.skype.chats.recent()
chats = getter()
chat = next(chats)

print chat
chat.getMsgs()
RikuAotsuki commented 3 years ago

No idea how I could be doing it wrong, but here's what it's doing:

console.txt

suurjaak commented 3 years ago

Can you copy-paste the entire console history?

RikuAotsuki commented 3 years ago

I have no idea what's going on now; had to restart my computer the other day so the console wound up cleared, and I can't seem to get it to return ANY information now, with any combination of stuff you've mentioned. Should I just make it create a new database again and then try?

console.txt

suurjaak commented 3 years ago

That is a strange result . Sure, you can try creating a new database from scratch.

RikuAotsuki commented 3 years ago

Alright, did that. Went through the "for id, chat in db.live.skype.chats.recent().items():" bit, then the "def getter():" part. This should be everything from the last returned chats from the former to the looping problem with the latter, though it seems to be a bit different this time around. Everything earlier than that is 18 more groups of returned chats, then the initialization stuff.

console.txt

suurjaak commented 3 years ago

The reason the latter has the looping problem, is because you're doing the "for id, chat.." bit before it. The former consumes all the recent chats, so the latter gets nothing.

Please, do only the steps set down in https://github.com/suurjaak/Skyperious/issues/100#issuecomment-878911803.

RikuAotsuki commented 3 years ago

Ooooooh. Sorry, wasn't aware that the command needed to be input on a fresh database to function. Will try that and report back.

RikuAotsuki commented 3 years ago

Alrighty, so it's not returning anything, just empty brackets. I tried a second missing chat just in case and got the same result. Here's the first one:

console.txt

suurjaak commented 3 years ago

Well, if that's the case, then nothing to do, those chats are just not syncable from online.

I will keep this issue open until I publish the new program version that fixes the Skype export issue.

RikuAotsuki commented 3 years ago

Huh, I wonder what'd cause that?

Thanks for the help though. Any ETA on the update?

suurjaak commented 3 years ago

Only Microsoft knows what would cause that, and they ain't telling :)

Any ETA on the update?

I guess within a week or two.

RikuAotsuki commented 3 years ago

I guess within a week or two.

Sweet, thanks

suurjaak commented 3 years ago

The newly released Skyperious v4.8 fixes the issue of not being able to create a new database from a Skype export archive.

Can you confirm?

RikuAotsuki commented 3 years ago

Running it now; it's a few hundred thousand messages' worth so it should be done tomorrow xD

RikuAotsuki commented 3 years ago

Yup, looks like it works. Anything "missing" now is pre-database change, I think.

Speaking of which, I have the exported html files of a bunch of chats from 2017. I think I found the database Skype would've extracted them from but can't be totally sure; is there any way to convert them back to a database that can be imported into Skyperious to check?

suurjaak commented 3 years ago

The only way would be to write a parser for the HTML files, which reads in the data and inserts the conversations, contacts and messages to a blank database.

But if you have found the original Skype SQLite database, why not open that in Skyperious?

RikuAotsuki commented 3 years ago

Alrighty. I actually did open in it Skyperious; I was just wondering if there was an easy way to check the old export on the off chance I was wrong about it being the same one. Thanks though!