Open wdanilo opened 1 week ago
Also, here is a code checking the same broken behavior for users.threads.list:
async test2() {
console.log(
"Performing test if 'users.threads.list' returns threads sorted by decreasing dates and historyIDs of their first messages.",
)
const count = 100
const cfg: GmailApi.Params$Resource$Users$Threads$List = {
userId: 'me',
maxResults: count,
}
const resp = await Auth.catchAllOAuth(() => this.api.users.threads.list(cfg))
if (resp.isErr()) {
console.error('GMail API query error.', resp.error)
return
}
const list = resp.value.data.threads
if (list == null) {
console.error('GMail API invalid response.')
return
}
let isHistoryIdDecreasing = true
let isDateDecreasing = true
let prevHistoryId: null | number = null
let prevDate: null | number = null
let i = 0
for (const thread of list) {
i++
const id = thread.id
if (id == null) {
console.error('Thread is missing the "id" field.')
return
}
const threadQuery = {userId: 'me', id}
const threadResp = await Auth.catchAllOAuth(() => this.api.users.threads.get(threadQuery))
if (threadResp.isErr()) {
console.error('GMail API query error.', threadResp.error)
return
}
if (threadResp.value.data.messages == null) {
console.error('Thread is missing the "messages" field.')
return
}
const firstMessage = threadResp.value.data.messages[0]
if (firstMessage == null) {
console.error('Thread is missing the first message.')
return
}
const historyIdStr = firstMessage.historyId
const dateStr = firstMessage.internalDate
if (!historyIdStr || !dateStr) {
console.error('First message is missing the "historyId" or "internalDate" field.')
return
}
const historyId = parseInt(historyIdStr, 10)
const date = parseInt(dateStr, 10)
let localIsHistoryIdDecreasing = true
let localIsDateDecreasing = true
if (prevHistoryId != null && prevHistoryId < historyId) {
isHistoryIdDecreasing = false
localIsHistoryIdDecreasing = false
}
if (prevDate != null && prevDate < date) {
isDateDecreasing = false
localIsDateDecreasing = false
}
prevHistoryId = historyId
prevDate = date
console.log(
`${i}/${count}: historyId: ${historyId}, date: ${date}, isHistoryIdDecreasing: ${localIsHistoryIdDecreasing}, isDateDecreasing: ${localIsDateDecreasing}`,
)
}
console.log({isHistoryIdDecreasing, isDateDecreasing})
}
Performing test if 'users.threads.list' returns threads sorted by decreasing dates and historyIDs of their first messages.
1/100: historyId: 15297863, date: 1719191761000, isHistoryIdDecreasing: true, isDateDecreasing: true
2/100: historyId: 15297367, date: 1719170805000, isHistoryIdDecreasing: true, isDateDecreasing: true
3/100: historyId: 15297189, date: 1719159861000, isHistoryIdDecreasing: true, isDateDecreasing: true
4/100: historyId: 15297049, date: 1719150301000, isHistoryIdDecreasing: true, isDateDecreasing: true
5/100: historyId: 15297132, date: 1719149574000, isHistoryIdDecreasing: false, isDateDecreasing: true
6/100: historyId: 15296597, date: 1719147670000, isHistoryIdDecreasing: true, isDateDecreasing: true
7/100: historyId: 15297036, date: 1719122788000, isHistoryIdDecreasing: false, isDateDecreasing: true
8/100: historyId: 15296855, date: 1719120755000, isHistoryIdDecreasing: true, isDateDecreasing: true
9/100: historyId: 15297511, date: 1719118502000, isHistoryIdDecreasing: false, isDateDecreasing: true
10/100: historyId: 15295111, date: 1719027583000, isHistoryIdDecreasing: true, isDateDecreasing: true
11/100: historyId: 15296007, date: 1719103434000, isHistoryIdDecreasing: false, isDateDecreasing: false
12/100: historyId: 15297035, date: 1719094618000, isHistoryIdDecreasing: false, isDateDecreasing: true
13/100: historyId: 15295569, date: 1719057377000, isHistoryIdDecreasing: true, isDateDecreasing: true
14/100: historyId: 15295489, date: 1719077834000, isHistoryIdDecreasing: true, isDateDecreasing: false
15/100: historyId: 15297034, date: 1719077423000, isHistoryIdDecreasing: false, isDateDecreasing: true
16/100: historyId: 15295374, date: 1719076577000, isHistoryIdDecreasing: true, isDateDecreasing: true
17/100: historyId: 15297544, date: 1719070375000, isHistoryIdDecreasing: false, isDateDecreasing: true
18/100: historyId: 15297032, date: 1719069634000, isHistoryIdDecreasing: true, isDateDecreasing: true
19/100: historyId: 15297569, date: 1719058421000, isHistoryIdDecreasing: false, isDateDecreasing: true
20/100: historyId: 15297030, date: 1719053485000, isHistoryIdDecreasing: true, isDateDecreasing: true
21/100: historyId: 15295971, date: 1719044954000, isHistoryIdDecreasing: true, isDateDecreasing: true
22/100: historyId: 15297029, date: 1719039969000, isHistoryIdDecreasing: false, isDateDecreasing: true
23/100: historyId: 15297429, date: 1719039871000, isHistoryIdDecreasing: false, isDateDecreasing: true
24/100: historyId: 15297027, date: 1719033506000, isHistoryIdDecreasing: true, isDateDecreasing: true
25/100: historyId: 15294309, date: 1719031329000, isHistoryIdDecreasing: true, isDateDecreasing: true
26/100: historyId: 15297593, date: 1719026659000, isHistoryIdDecreasing: false, isDateDecreasing: true
27/100: historyId: 15297025, date: 1719025075000, isHistoryIdDecreasing: true, isDateDecreasing: true
28/100: historyId: 15297024, date: 1719024799000, isHistoryIdDecreasing: true, isDateDecreasing: true
29/100: historyId: 15297618, date: 1719024388000, isHistoryIdDecreasing: false, isDateDecreasing: true
30/100: historyId: 15297022, date: 1719021288000, isHistoryIdDecreasing: true, isDateDecreasing: true
31/100: historyId: 15297642, date: 1719021008000, isHistoryIdDecreasing: false, isDateDecreasing: true
32/100: historyId: 15293469, date: 1719021069000, isHistoryIdDecreasing: true, isDateDecreasing: false
33/100: historyId: 15297671, date: 1719008653000, isHistoryIdDecreasing: false, isDateDecreasing: true
34/100: historyId: 15293217, date: 1719008487000, isHistoryIdDecreasing: true, isDateDecreasing: true
35/100: historyId: 15297705, date: 1719008444000, isHistoryIdDecreasing: false, isDateDecreasing: true
36/100: historyId: 15293049, date: 1719007821000, isHistoryIdDecreasing: true, isDateDecreasing: true
37/100: historyId: 15292962, date: 1719005719000, isHistoryIdDecreasing: true, isDateDecreasing: true
38/100: historyId: 15297739, date: 1719004759000, isHistoryIdDecreasing: false, isDateDecreasing: true
39/100: historyId: 15293296, date: 1719004141000, isHistoryIdDecreasing: true, isDateDecreasing: true
40/100: historyId: 15292645, date: 1719002715000, isHistoryIdDecreasing: true, isDateDecreasing: true
41/100: historyId: 15290059, date: 1718993022000, isHistoryIdDecreasing: true, isDateDecreasing: true
42/100: historyId: 15292507, date: 1718994153000, isHistoryIdDecreasing: false, isDateDecreasing: false
43/100: historyId: 15296270, date: 1719000410000, isHistoryIdDecreasing: false, isDateDecreasing: false
44/100: historyId: 15296304, date: 1719000210000, isHistoryIdDecreasing: false, isDateDecreasing: true
45/100: historyId: 15291662, date: 1718999355000, isHistoryIdDecreasing: true, isDateDecreasing: true
46/100: historyId: 15293696, date: 1719000158000, isHistoryIdDecreasing: false, isDateDecreasing: false
47/100: historyId: 15292133, date: 1719000132000, isHistoryIdDecreasing: true, isDateDecreasing: true
48/100: historyId: 15291968, date: 1718999917000, isHistoryIdDecreasing: true, isDateDecreasing: true
49/100: historyId: 15291886, date: 1718999562000, isHistoryIdDecreasing: true, isDateDecreasing: true
50/100: historyId: 15291744, date: 1718999417000, isHistoryIdDecreasing: true, isDateDecreasing: true
51/100: historyId: 15291461, date: 1718998798000, isHistoryIdDecreasing: true, isDateDecreasing: true
52/100: historyId: 15291384, date: 1718998480000, isHistoryIdDecreasing: true, isDateDecreasing: true
53/100: historyId: 15291302, date: 1718998079000, isHistoryIdDecreasing: true, isDateDecreasing: true
54/100: historyId: 15291164, date: 1718997874000, isHistoryIdDecreasing: true, isDateDecreasing: true
55/100: historyId: 15291019, date: 1718997022000, isHistoryIdDecreasing: true, isDateDecreasing: true
56/100: historyId: 15290937, date: 1718996810000, isHistoryIdDecreasing: true, isDateDecreasing: true
57/100: historyId: 15290677, date: 1718995321000, isHistoryIdDecreasing: true, isDateDecreasing: true
58/100: historyId: 15290510, date: 1718994605000, isHistoryIdDecreasing: true, isDateDecreasing: true
59/100: historyId: 15290371, date: 1718994196000, isHistoryIdDecreasing: true, isDateDecreasing: true
60/100: historyId: 15290592, date: 1718994167000, isHistoryIdDecreasing: false, isDateDecreasing: true
61/100: historyId: 15290141, date: 1718993384000, isHistoryIdDecreasing: true, isDateDecreasing: true
62/100: historyId: 15289917, date: 1718992717000, isHistoryIdDecreasing: true, isDateDecreasing: true
63/100: historyId: 15289836, date: 1718992611000, isHistoryIdDecreasing: true, isDateDecreasing: true
64/100: historyId: 15289755, date: 1718992550000, isHistoryIdDecreasing: true, isDateDecreasing: true
65/100: historyId: 15289673, date: 1718992117000, isHistoryIdDecreasing: true, isDateDecreasing: true
66/100: historyId: 15289532, date: 1718992105000, isHistoryIdDecreasing: true, isDateDecreasing: true
67/100: historyId: 15289249, date: 1718991700000, isHistoryIdDecreasing: true, isDateDecreasing: true
68/100: historyId: 15289156, date: 1718991369000, isHistoryIdDecreasing: true, isDateDecreasing: true
69/100: historyId: 15297775, date: 1718991164000, isHistoryIdDecreasing: false, isDateDecreasing: true
70/100: historyId: 15297396, date: 1718991124000, isHistoryIdDecreasing: true, isDateDecreasing: true
71/100: historyId: 15297463, date: 1718989129000, isHistoryIdDecreasing: false, isDateDecreasing: true
72/100: historyId: 15288682, date: 1718987737000, isHistoryIdDecreasing: true, isDateDecreasing: true
73/100: historyId: 15288603, date: 1718986405000, isHistoryIdDecreasing: true, isDateDecreasing: true
74/100: historyId: 15288488, date: 1718982653000, isHistoryIdDecreasing: true, isDateDecreasing: true
75/100: historyId: 15288464, date: 1718982411000, isHistoryIdDecreasing: true, isDateDecreasing: true
76/100: historyId: 15297373, date: 1718980504000, isHistoryIdDecreasing: false, isDateDecreasing: true
77/100: historyId: 15288268, date: 1718978327000, isHistoryIdDecreasing: true, isDateDecreasing: true
78/100: historyId: 15288267, date: 1718971574000, isHistoryIdDecreasing: true, isDateDecreasing: true
79/100: historyId: 15293754, date: 1718909485000, isHistoryIdDecreasing: false, isDateDecreasing: true
80/100: historyId: 15287783, date: 1718956922000, isHistoryIdDecreasing: true, isDateDecreasing: false
81/100: historyId: 15287862, date: 1718956695000, isHistoryIdDecreasing: false, isDateDecreasing: true
82/100: historyId: 15288266, date: 1718954346000, isHistoryIdDecreasing: false, isDateDecreasing: true
83/100: historyId: 15287564, date: 1718953202000, isHistoryIdDecreasing: true, isDateDecreasing: true
84/100: historyId: 15287478, date: 1718935413000, isHistoryIdDecreasing: true, isDateDecreasing: true
85/100: historyId: 15287253, date: 1718924448000, isHistoryIdDecreasing: true, isDateDecreasing: true
86/100: historyId: 15287477, date: 1718920072000, isHistoryIdDecreasing: false, isDateDecreasing: true
87/100: historyId: 15287061, date: 1718917993000, isHistoryIdDecreasing: true, isDateDecreasing: true
88/100: historyId: 15286956, date: 1718909913000, isHistoryIdDecreasing: true, isDateDecreasing: true
89/100: historyId: 15286786, date: 1718904398000, isHistoryIdDecreasing: true, isDateDecreasing: true
90/100: historyId: 15286705, date: 1718898024000, isHistoryIdDecreasing: true, isDateDecreasing: true
91/100: historyId: 15286624, date: 1718900423000, isHistoryIdDecreasing: true, isDateDecreasing: false
92/100: historyId: 15295890, date: 1718857229000, isHistoryIdDecreasing: false, isDateDecreasing: true
93/100: historyId: 15286444, date: 1718883913000, isHistoryIdDecreasing: true, isDateDecreasing: false
94/100: historyId: 15287475, date: 1718879255000, isHistoryIdDecreasing: false, isDateDecreasing: true
95/100: historyId: 15286284, date: 1718876932000, isHistoryIdDecreasing: true, isDateDecreasing: true
96/100: historyId: 15287493, date: 1718859740000, isHistoryIdDecreasing: false, isDateDecreasing: true
97/100: historyId: 15286009, date: 1718853779000, isHistoryIdDecreasing: true, isDateDecreasing: true
98/100: historyId: 15285920, date: 1718839427000, isHistoryIdDecreasing: true, isDateDecreasing: true
99/100: historyId: 15285840, date: 1718831857000, isHistoryIdDecreasing: true, isDateDecreasing: true
100/100: historyId: 15285787, date: 1718829287000, isHistoryIdDecreasing: true, isDateDecreasing: true
{ isHistoryIdDecreasing: false, isDateDecreasing: false }
Also, to make it more visual, I plotted the time I get from users.messages.list
. As you can see, there is a drop, so we got 2 messages with date way earlier than messages we get after it. X-axis is the message # from the users.messages.list
. Y-axis is its internalDate
:
And here is plot of historyId
. X-axis is the message # from the users.messages.list
. Y-axis is its historyId
. So, in contrast to what Google docs state, historyId
does not increase chronologically.
What is even worse, it appears that users.history.list
does not provide results in chronological order neither, despite Google claims so (https://developers.google.com/gmail/api/reference/rest/v1/users.history/list) (citation: Lists the history of all changes to the given mailbox. History results are returned in chronological order (increasing historyId).
). Below is a code to test it:
async test_invariants_history_list() {
console.log(
"Performing test if 'users.history.list' returns history records sorted by increasing update date.",
)
const viz = (t: boolean) => (t ? '✅' : '❌')
const count = 100
const cfg: GmailApi.Params$Resource$Users$Messages$List = {
userId: 'me',
maxResults: count,
}
const resp = await Auth.catchAllOAuth(() => this.api.users.messages.list(cfg))
if (resp.isErr()) {
console.error('GMail API query error.', resp.error)
return
}
const list = resp.value.data.messages
if (list == null) {
console.error('GMail API invalid response.')
return
}
const lastMessage = list[list.length - 1]
if (lastMessage == null) {
console.error('Last message is missing.')
return
}
const lastMessageId = lastMessage.id
if (lastMessageId == null) {
console.error('Last message is missing the "id" field.')
return
}
const lastMessageQuery = {userId: 'me', id: lastMessageId}
const lastMessageResp = await Auth.catchAllOAuth(() =>
this.api.users.messages.get(lastMessageQuery),
)
if (lastMessageResp.isErr()) {
console.error('GMail API query error.', lastMessageResp.error)
return
}
const startHistoryId = lastMessageResp.value.data.historyId
if (startHistoryId == null) {
console.error('Last message is missing the "historyId" field.')
return
}
const historyCfg: GmailApi.Params$Resource$Users$History$List = {
userId: 'me',
startHistoryId,
maxResults: 500,
}
const historyResp = await Auth.catchAllOAuth(() => this.api.users.history.list(historyCfg))
if (historyResp.isErr()) {
console.error('GMail API query error.', historyResp.error)
return
}
const historyList = historyResp.value.data.history
if (historyList == null) {
console.error('GMail API invalid response.')
return
}
let localDateIncreasing = true
let prevDate: null | number = null
const updateCount = historyList.length
let i = 0
for (const update of historyList) {
i++
if (update.messages == null) {
console.error('Update is missing the "messages" field.')
return
}
let m = 0
for (const message of update.messages) {
m++
if (message.id == null) {
console.error('Message is missing the "id" field.')
return
}
const messageQuery = {userId: 'me', id: message.id}
const messageResp = await Auth.catchAllOAuth(() =>
this.api.users.messages.get(messageQuery),
)
if (messageResp.isErr()) {
console.error('GMail API query error.', messageResp.error)
continue
}
const dateStr = messageResp.value.data.internalDate
if (!dateStr) {
console.error('Message is missing the "internalDate" field.')
return
}
const date = parseInt(dateStr, 10)
if (prevDate != null && prevDate > date) {
localDateIncreasing = false
} else {
localDateIncreasing = true
}
prevDate = date
console.log(
`${i}/${updateCount}[${m}]: date: ${date}, dateIncreasing: ${viz(localDateIncreasing)}`,
)
}
}
}
1/232[1]: date: 1718992106000, dateIncreasing: ✅
2/232[1]: date: 1718992117000, dateIncreasing: ✅
3/232[1]: date: 1718992550000, dateIncreasing: ✅
4/232[1]: date: 1718992611000, dateIncreasing: ✅
5/232[1]: date: 1718992717000, dateIncreasing: ✅
6/232[1]: date: 1718992719000, dateIncreasing: ✅
7/232[1]: date: 1718993022000, dateIncreasing: ✅
8/232[1]: date: 1718993384000, dateIncreasing: ✅
9/232[1]: date: 1718994167000, dateIncreasing: ✅
10/232[1]: date: 1718994167000, dateIncreasing: ✅
11/232[1]: date: 1718994167000, dateIncreasing: ✅
12/232[1]: date: 1718994167000, dateIncreasing: ✅
13/232[1]: date: 1718994196000, dateIncreasing: ✅
14/232[1]: date: 1718994479000, dateIncreasing: ✅
15/232[1]: date: 1718994605000, dateIncreasing: ✅
16/232[1]: date: 1718994167000, dateIncreasing: ❌
17/232[1]: date: 1718995295000, dateIncreasing: ✅
18/232[1]: date: 1718995321000, dateIncreasing: ✅
19/232[1]: date: 1718995732000, dateIncreasing: ✅
20/232[1]: date: 1718996035000, dateIncreasing: ✅
21/232[1]: date: 1718996476000, dateIncreasing: ✅
22/232[1]: date: 1718996810000, dateIncreasing: ✅
23/232[1]: date: 1718997022000, dateIncreasing: ✅
24/232[1]: date: 1718997051000, dateIncreasing: ✅
25/232[1]: date: 1718997874000, dateIncreasing: ✅
26/232[1]: date: 1718997877000, dateIncreasing: ✅
27/232[1]: date: 1718998079000, dateIncreasing: ✅
28/232[1]: date: 1718998480000, dateIncreasing: ✅
29/232[1]: date: 1718998798000, dateIncreasing: ✅
30/232[1]: date: 1718999072000, dateIncreasing: ✅
31/232[1]: date: 1718999152000, dateIncreasing: ✅
32/232[1]: date: 1718999355000, dateIncreasing: ✅
33/232[1]: date: 1718999417000, dateIncreasing: ✅
34/232[1]: date: 1718999475000, dateIncreasing: ✅
35/232[1]: date: 1718999562000, dateIncreasing: ✅
36/232[1]: date: 1718999917000, dateIncreasing: ✅
37/232[1]: date: 1719000132000, dateIncreasing: ✅
38/232[1]: date: 1719000132000, dateIncreasing: ✅
39/232[1]: date: 1719000132000, dateIncreasing: ✅
40/232[1]: date: 1719000132000, dateIncreasing: ✅
41/232[1]: date: 1719000132000, dateIncreasing: ✅
42/232[1]: date: 1719000132000, dateIncreasing: ✅
43/232[1]: date: 1719000158000, dateIncreasing: ✅
44/232[1]: date: 1719000212000, dateIncreasing: ✅
45/232[1]: date: 1719000210000, dateIncreasing: ❌
GMail API query error. {
level: 4,
message: 'OAuth call failed.',
data: {
error: GaxiosError: Requested entity was not found.
at Gaxios._request (<redacted>)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async UserRefreshClient.requestAsync (<redacted>)
at async catchAllOAuth (<redacted>)
at async Sync.test_invariants_history_list (<redacted>)
response: [Object],
config: [Object],
code: 404,
errors: [Array]
}
},
flags: {}
}
47/232[1]: date: 1719000410000, dateIncreasing: ✅
48/232[1]: date: 1718994153000, dateIncreasing: ❌
49/232[1]: date: 1719001722000, dateIncreasing: ✅
50/232[1]: date: 1719002715000, dateIncreasing: ✅
51/232[1]: date: 1719004141000, dateIncreasing: ✅
52/232[1]: date: 1719004141000, dateIncreasing: ✅
53/232[1]: date: 1719004141000, dateIncreasing: ✅
54/232[1]: date: 1719004141000, dateIncreasing: ✅
55/232[1]: date: 1719004759000, dateIncreasing: ✅
56/232[1]: date: 1719005719000, dateIncreasing: ✅
57/232[1]: date: 1719007821000, dateIncreasing: ✅
58/232[1]: date: 1719008444000, dateIncreasing: ✅
59/232[1]: date: 1719008487000, dateIncreasing: ✅
60/232[1]: date: 1719008653000, dateIncreasing: ✅
61/232[1]: date: 1719021008000, dateIncreasing: ✅
62/232[1]: date: 1719021069000, dateIncreasing: ✅
63/232[1]: date: 1719021206000, dateIncreasing: ✅
64/232[1]: date: 1719021288000, dateIncreasing: ✅
65/232[1]: date: 1719000158000, dateIncreasing: ❌
66/232[1]: date: 1719000158000, dateIncreasing: ✅
67/232[1]: date: 1719000158000, dateIncreasing: ✅
68/232[1]: date: 1718909485000, dateIncreasing: ❌
Moreover, as you can see in the above test, users.history.list
can return records, that contain invalid message ids (the GMail API query error indicates so and I can reproduce it every time).
Description
users.messages.list returns messages in a little bit random order. I also reported the issue here, but knowing Google's issue tracker, I will get a sensible reply there in a few years, so trying my luck here as well :(
Code
I'm using the following code:
Output
Summary
As you can see, the message internalDate is sometimes ... increasing. I've observed the same behavior for users.threads.list. In such a scenario:
What is described here: https://developers.google.com/gmail/api/guides/sync is not true, I can't query for the first message of users.messages.list, remember its historyId and assume this is the newest historyId, as this might NOT be the newest message.
What is described here: https://developers.google.com/gmail/api/reference/rest/v1/users.history/list (
History IDs increase chronologically but are not contiguous with random gaps in between valid IDs
) is not true, as you can see, historyIDs are sometimes increasing for older messages.