Email restored with incorrect time

[panmail]

I'm using GYB 1.34 in Windows 10 trying to backup from one gmail account and restore to another.

0) ran gyb-setup.bat and configured my 2 gmail accounts by providing oauth ids and secrets. 1) gyb --email xxx@gmail.com --action backup --local-folder "F:\gybBackUp" 2) gyb --email yyy@gmail.com --action restore --local-folder "F:\gybBackUp"

Backup took about 1hr for 11GB worth of gmail emails. Restore took like 8x, or 8hrs, to complete. got a few "ERROR: 400 Expected a single 'From' header. Skipping message restore, you can retry later with --fast-restore" along the way.

What I noticed is that some emails were sent, say Jan 22nd 2019 1:59pm, in the xxx@gmail.com account, has now been restored to be sent, Feb 23rd 2020 16:03, in the yyy@gmail.com account.

[jay0lee]

This is the expected behavior it some of your messages have a bad date header. GYB tells Google to set the message date based off the date header, however if Gmail servers can't properly parse the date header they have to fallback to the time the message was imported.

Jay

On Sun, Feb 23, 2020, 9:08 PM panmail notifications@github.com wrote:

I'm using GYB 1.34 in Windows 10 trying to backup from one gmail account and restore to another.

1. ran gyb-setup.bat and configured my 2 gmail accounts by providing oauth ids and secrets.
2. gyb --email xxx@gmail.com --action backup --local-folder "F:\gybBackUp"
3. gyb --email yyy@gmail.com --action restore --local-folder "F:\gybBackUp"

Backup took about 1hr for 11GB worth of gmail emails. Restore took like 8x, or 8hrs, to complete. got a few "ERROR: 400 Expected a single 'From' header. Skipping message restore, you can retry later with --fast-restore" along the way.

What I noticed is that some emails were sent, say Jan 22nd 2019 1:59pm, in the xxx@gmail.com account, has now been restored to be sent, Feb 23rd 2020 16:03, in the yyy@gmail.com account.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jay0lee/got-your-back/issues/248?email_source=notifications&email_token=ABDIZMEE3Q5SRFJOYY5W2GDREMTZXA5CNFSM4K2ABM5KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IPTODVA, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDIZMB5IZAGCA2D7VHIYD3REMTZXANCNFSM4K2ABM5A .

[panmail]

Hi Jay,

Is there any ways to identify from a gyb backup log (?? perhaps msg-db.sqlite ??) which messages are affected? or maybe a command line switch to turn on error logging?

Further more would it be possible to manually edit the downloaded .eml files so the gyb restore could stamp the affected emails with the correct date?

ATM, since there is at-least 1 message out of 32K that I could spot check as being affected, I couldn't really trust the entire restore because I need the email histories to reflect what really took place. I feel we are almost there.

You have a very unique and awesome piece of utility which I don't know why Google doesn't provide. Very much appreciate your help.

John

[jay0lee]

No way for GYB too identify as Google doesn't report whether it parsed the header correctly or not.

Can you share headers for some messages that did not get correct date?

On Mon, Feb 24, 2020, 7:44 AM panmail notifications@github.com wrote:

Hi Jay,

Is there any ways to identify from a gyb backup log (?? perhaps msg-db.sqlite ??) which messages are affected? or maybe a command line switch to turn on error logging?

Further more would it be possible to manually edit the downloaded .eml files so the gyb restore could stamp the affected emails with the correct date?

ATM, since there is at-least 1 message out of 32K that I could spot check as being affected, I couldn't really trust the entire restore because I need the email histories to reflect what really took place. I feel we are almost there.

You have a very unique and awesome piece of utility which I don't know why Google doesn't provide. Very much appreciate your help.

John

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jay0lee/got-your-back/issues/248?email_source=notifications&email_token=ABDIZMHULX3E6LCEMXJKHPTREO6JXA5CNFSM4K2ABM5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMXUVCA#issuecomment-590301832, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDIZMBY6MRXRLYXQIIZG73REO6JXANCNFSM4K2ABM5A .

[panmail]

below is the original email header and the imported email header which has yesterday's date. it should have been Jan 22, 2019, not sure why this date isn't on the original email header though, or is this the crux of the issue? within the gmail interface, the sent date is being reported correctly tho. please let me know what else I could provide?

[jay0lee]

Well there is NO Date: header on the original message that I can see so that would explain why Google had to add a Date: header itself. Based on the screenshot there's no way for Google to know that the message was from Jan 2019....

Jay Lee

On Mon, Feb 24, 2020 at 9:41 AM panmail notifications@github.com wrote:

below is the original email header and the imported email header which has yesterday's date. it should have been Jan 22, 2019, not sure why this date isn't on the original email header though. please let me know what else I could provide?

[image: image] https://user-images.githubusercontent.com/7851873/75161179-3aecb880-56d0-11ea-8296-73754c868625.png

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jay0lee/got-your-back/issues/248?email_source=notifications&email_token=ABDIZMFREJNYNMDH5LIJBC3REPMBFA5CNFSM4K2ABM5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMYBE2Q#issuecomment-590353002, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDIZMDDTJ4S4WJ6EPSQZATREPMBFANCNFSM4K2ABM5A .

[panmail]

that my thought too. do you think this is an issue with the export step because in my browser, google is reporting the correct sent date so the correct sent date is saved within google somewhere. i'll try the export again and see if the issue persists with the same email.

On Mon, Feb 24, 2020 at 8:16 AM Jay Lee notifications@github.com wrote:

Well there is NO Date: header on the original message that I can see so that would explain why Google had to add a Date: header itself. Based on the screenshot there's no way for Google to know that the message was from Jan 2019....

Jay Lee

On Mon, Feb 24, 2020 at 9:41 AM panmail notifications@github.com wrote:

below is the original email header and the imported email header which has yesterday's date. it should have been Jan 22, 2019, not sure why this date isn't on the original email header though. please let me know what else I could provide?

[image: image] < https://user-images.githubusercontent.com/7851873/75161179-3aecb880-56d0-11ea-8296-73754c868625.png

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/jay0lee/got-your-back/issues/248?email_source=notifications&email_token=ABDIZMFREJNYNMDH5LIJBC3REPMBFA5CNFSM4K2ABM5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMYBE2Q#issuecomment-590353002 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/ABDIZMDDTJ4S4WJ6EPSQZATREPMBFANCNFSM4K2ABM5A

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jay0lee/got-your-back/issues/248?email_source=notifications&email_token=AB346YL432CAZG44PNITOYLREPXHDA5CNFSM4K2ABM5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMYPOHI#issuecomment-590411549, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB346YOVTSYFFQZZXY6H5FDREPXHDANCNFSM4K2ABM5A .

[jay0lee]

There's no date header on the original message, it's not an export issue, it's that the data just isn't there. At some point the email got imported to Google without a date header, however that was done it looks like the "internal date" was set but the Date header was not.

Again, not much GYB can do to fix this.

Jay Lee

On Mon, Feb 24, 2020 at 11:55 AM panmail notifications@github.com wrote:

that my thought too. do you think this is an issue with the export step because in my browser, google is reporting the correct sent date so the correct sent date is saved within google somewhere. i'll try the export again and see if the issue persists with the same email.

On Mon, Feb 24, 2020 at 8:16 AM Jay Lee notifications@github.com wrote:

Well there is NO Date: header on the original message that I can see so that would explain why Google had to add a Date: header itself. Based on the screenshot there's no way for Google to know that the message was from Jan 2019....

Jay Lee

On Mon, Feb 24, 2020 at 9:41 AM panmail notifications@github.com wrote:

below is the original email header and the imported email header which has yesterday's date. it should have been Jan 22, 2019, not sure why this date isn't on the original email header though. please let me know what else I could provide?

[image: image] <

https://user-images.githubusercontent.com/7851873/75161179-3aecb880-56d0-11ea-8296-73754c868625.png

— You are receiving this because you commented. Reply to this email directly, view it on GitHub <

https://github.com/jay0lee/got-your-back/issues/248?email_source=notifications&email_token=ABDIZMFREJNYNMDH5LIJBC3REPMBFA5CNFSM4K2ABM5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMYBE2Q#issuecomment-590353002

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ABDIZMDDTJ4S4WJ6EPSQZATREPMBFANCNFSM4K2ABM5A

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/jay0lee/got-your-back/issues/248?email_source=notifications&email_token=AB346YL432CAZG44PNITOYLREPXHDA5CNFSM4K2ABM5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMYPOHI#issuecomment-590411549 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AB346YOVTSYFFQZZXY6H5FDREPXHDANCNFSM4K2ABM5A

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jay0lee/got-your-back/issues/248?email_source=notifications&email_token=ABDIZMGQMMO7LGUXN7WFVM3REP3XBA5CNFSM4K2ABM5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMYUZTI#issuecomment-590433485, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDIZMECTXETCJPE4J4UZSLREP3XBANCNFSM4K2ABM5A .

[panmail]

hmmmm .... that's interesting. so there's an "internal date" kept somewhere by Gmail and this doesn't get exported by GYB nor Google Takeout?

On Mon, Feb 24, 2020 at 9:18 AM Jay Lee notifications@github.com wrote:

There's no date header on the original message, it's not an export issue, it's that the data just isn't there. At some point the email got imported to Google without a date header, however that was done it looks like the "internal date" was set but the Date header was not.

Again, not much GYB can do to fix this.

Jay Lee

On Mon, Feb 24, 2020 at 11:55 AM panmail notifications@github.com wrote:

that my thought too. do you think this is an issue with the export step because in my browser, google is reporting the correct sent date so the correct sent date is saved within google somewhere. i'll try the export again and see if the issue persists with the same email.

On Mon, Feb 24, 2020 at 8:16 AM Jay Lee notifications@github.com wrote:

Well there is NO Date: header on the original message that I can see so that would explain why Google had to add a Date: header itself. Based on the screenshot there's no way for Google to know that the message was from Jan 2019....

Jay Lee

On Mon, Feb 24, 2020 at 9:41 AM panmail notifications@github.com wrote:

below is the original email header and the imported email header which has yesterday's date. it should have been Jan 22, 2019, not sure why this date isn't on the original email header though. please let me know what else I could provide?

[image: image] <

https://user-images.githubusercontent.com/7851873/75161179-3aecb880-56d0-11ea-8296-73754c868625.png

— You are receiving this because you commented. Reply to this email directly, view it on GitHub <

https://github.com/jay0lee/got-your-back/issues/248?email_source=notifications&email_token=ABDIZMFREJNYNMDH5LIJBC3REPMBFA5CNFSM4K2ABM5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMYBE2Q#issuecomment-590353002

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ABDIZMDDTJ4S4WJ6EPSQZATREPMBFANCNFSM4K2ABM5A

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <

https://github.com/jay0lee/got-your-back/issues/248?email_source=notifications&email_token=AB346YL432CAZG44PNITOYLREPXHDA5CNFSM4K2ABM5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMYPOHI#issuecomment-590411549

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AB346YOVTSYFFQZZXY6H5FDREPXHDANCNFSM4K2ABM5A

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/jay0lee/got-your-back/issues/248?email_source=notifications&email_token=ABDIZMGQMMO7LGUXN7WFVM3REP3XBA5CNFSM4K2ABM5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMYUZTI#issuecomment-590433485 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/ABDIZMECTXETCJPE4J4UZSLREP3XBANCNFSM4K2ABM5A

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jay0lee/got-your-back/issues/248?email_source=notifications&email_token=AB346YIP6G7TP3FUQHK3G33REP6NLA5CNFSM4K2ABM5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMYX6CA#issuecomment-590446344, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB346YLP7KEES25WNBOCEIDREP6NLANCNFSM4K2ABM5A .

[jay0lee]

correct, Gmail API does see the internal date.