fetchMessage brining all messages instead of only missed messages

[devopsokdone]

We have upgraded SDK to 2.0.1 by overriding meta package.

Now getting messages from History with timeToken, but facing an issue with fetching messages after an end time token upto newest messages. Scenario 2 as explained in this link https://www.pubnub.com/docs/platform/messages/storage

Below is the code and logs for your reference, you can see the messages returned by history has a time token prior to end time token we have set whereas only messages only after the timeToken mentioned should be returned.

Code:

final messageHistory = await client.batch.fetchMessages( list.toSet(), end: Timetoken.fromDateTime( DateTime.fromMillisecondsSinceEpoch(lastMessage.timeToken)), reverse: true, );

Debug Logs:

lastMessageTime => 15997452027059086

messageHistory => 1 history message {content: hi, time: 1599743961077, fromUser: br-test, pn_gcm: {data: {content: hi, channel: ch_br-test_rr-test, messageId: b093a258-0d25-4f03-92cd-a438af1a796a}, pn_exceptions: [eoIyYl38TQOMJ8gNmpJ37R:APA91bGvsgjmv7w7rwZroCAl2n-vLgW_erGVioAyDzHLwDumOo7LqQR9fiMfB8U3QBIW4dh7pgI0z_KegmJrFuWci8lvWszYgELkuuc1RFqO8qVgHVXDYwMHBiAxPzDGDe-_P4umcFQN]}, type: text, id: b093a258-0d25-4f03-92cd-a438af1a796a, channel: ch_br-test_rr-test} => time token 15997439631767736 incoming => 15997439631767736 history message {content: qwerty, time: 1599744036071, fromUser: br-test, pn_gcm: {data: {content: qwerty, channel: ch_br-test_rr-test, messageId: 92f5eef8-074c-4345-aa7e-c5708cbe6363}, pn_exceptions: [eoIyYl38TQOMJ8gNmpJ37R:APA91bGvsgjmv7w7rwZroCAl2n-vLgW_erGVioAyDzHLwDumOo7LqQR9fiMfB8U3QBIW4dh7pgI0z_KegmJrFuWci8lvWszYgELkuuc1RFqO8qVgHVXDYwMHBiAxPzDGDe-_P4umcFQN]}, type: text, id: 92f5eef8-074c-4345-aa7e-c5708cbe6363, channel: ch_br-test_rr-test} => time token 15997440367618732 incoming => 15997440367618732 history message {content: qa, time: 1599744162619, fromUser: br-test, pn_gcm: {data: {content: qa, channel: ch_br-test_rr-test, messageId: fcfc2afd-6602-4e74-8a60-aac6fa325418}, pn_exceptions: [eoIyYl38TQOMJ8gNmpJ37R:APA91bGvsgjmv7w7rwZroCAl2n-vLgW_erGVioAyDzHLwDumOo7LqQR9fiMfB8U3QBIW4dh7pgI0z_KegmJrFuWci8lvWszYgELkuuc1RFqO8qVgHVXDYwMHBiAxPzDGDe-_P4umcFQN]}, type: text, id: fcfc2afd-6602-4e74-8a60-aac6fa325418, channel: ch_br-test_rr-test} => time token 15997441633913204 incoming => 15997441633913204 history message {content: was, time: 1599744175923, fromUser: br-test, pn_gcm: {data: {content: was, channel: ch_br-test_rr-test, messageId: 1e0a32d6-47f9-42da-b831-27e636931c89}, pn_exceptions: [eoIyYl38TQOMJ8gNmpJ37R:APA91bGvsgjmv7w7rwZroCAl2n-vLgW_erGVioAyDzHLwDumOo7LqQR9fiMfB8U3QBIW4dh7pgI0z_KegmJrFuWci8lvWszYgELkuuc1RFqO8qVgHVXDYwMHBiAxPzDGDe-_P4umcFQN]}, type: text, id: 1e0a32d6-47f9-42da-b831-27e636931c89, channel: ch_br-test_rr-test} => time token 15997441766302065 incoming => 15997441766302065 history message {content: qa, time: 1599744277041, fromUser: br-test, pn_gcm: {data: {content: qa, channel: ch_br-test_rr-test, messageId: c8c7fd96-dd0c-4acc-a4aa-6d3b02a502d2}, pn_exceptions: [eoIyYl38TQOMJ8gNmpJ37R:APA91bGvsgjmv7w7rwZroCAl2n-vLgW_erGVioAyDzHLwDumOo7LqQR9fiMfB8U3QBIW4dh7pgI0z_KegmJrFuWci8lvWszYgELkuuc1RFqO8qVgHVXDYwMHBiAxPzDGDe-_P4umcFQN]}, type: text, id: c8c7fd96-dd0c-4acc-a4aa-6d3b02a502d2, channel: ch_br-test_rr-test} => time token 15997442778196674 incoming => 15997442778196674 history message {content: asd, time: 1599744280757, fromUser: br-test, pn_gcm: {data: {content: asd, channel: ch_br-test_rr-test, messageId: 61eef2e5-54e5-4fec-8d88-baaeae8746aa}, pn_exceptions: [eoIyYl38TQOMJ8gNmpJ37R:APA91bGvsgjmv7w7rwZroCAl2n-vLgW_erGVioAyDzHLwDumOo7LqQR9fiMfB8U3QBIW4dh7pgI0z_KegmJrFuWci8lvWszYgELkuuc1RFqO8qVgHVXDYwMHBiAxPzDGDe-_P4umcFQN]}, type: text, id: 61eef2e5-54e5-4fec-8d88-baaeae8746aa, channel: ch_br-test_rr-test} => time token 15997442814539074 incoming => 15997442814539074 history message {content: fghj, time: 1599744285076, fromUser: br-test, pn_gcm: {data: {content: fghj, channel: ch_br-test_rr-test, messageId: 448ee5f3-428c-498f-81c9-dde41d4ecc95}, pn_exceptions: [eoIyYl38TQOMJ8gNmpJ37R:APA91bGvsgjmv7w7rwZroCAl2n-vLgW_erGVioAyDzHLwDumOo7LqQR9fiMfB8U3QBIW4dh7pgI0z_KegmJrFuWci8lvWszYgELkuuc1RFqO8qVgHVXDYwMHBiAxPzDGDe-_P4umcFQN]}, type: text, id: 448ee5f3-428c-498f-81c9-dde41d4ecc95, channel: ch_br-test_rr-test} => time token 15997442858297131 incoming => 15997442858297131 history message {content: klz, time: 1599744287938, fromUser: br-test, pn_gcm: {data: {content: klz, channel: ch_br-test_rr-test, messageId: ef00a17a-b583-4802-bc88-a5f66eb3e4fb}, pn_exceptions: [eoIyYl38TQOMJ8gNmpJ37R:APA91bGvsgjmv7w7rwZroCAl2n-vLgW_erGVioAyDzHLwDumOo7LqQR9fiMfB8U3QBIW4dh7pgI0z_KegmJrFuWci8lvWszYgELkuuc1RFqO8qVgHVXDYwMHBiAxPzDGDe-_P4umcFQN]}, type: text, id: ef00a17a-b583-4802-bc88-a5f66eb3e4fb, channel: ch_br-test_rr-test} => time token 15997442886405212 incoming => 15997442886405212 history message {content: asdfgghhh, time: 1599744309893, fromUser: br-test, pn_gcm: {data: {content: asdfgghhh, channel: ch_br-test_rr-test, messageId: 04e344c9-6a1d-4065-83b2-974feb27edd8}, pn_exceptions: [eoIyYl38TQOMJ8gNmpJ37R:APA91bGvsgjmv7w7rwZroCAl2n-vLgW_erGVioAyDzHLwDumOo7LqQR9fiMfB8U3QBIW4dh7pgI0z_KegmJrFuWci8lvWszYgELkuuc1RFqO8qVgHVXDYwMHBiAxPzDGDe-_P4umcFQN]}, type: text, id: 04e344c9-6a1d-4065-83b2-974feb27edd8, channel: ch_br-test_rr-test} => time token 15997443105914899 incoming => 15997443105914899 history message {content: wweee, time: 1599744496485, fromUser: br-test, pn_gcm: {data: {content: wweee, channel: ch_br-test_rr-test, messageId: 1646b3bc-edab-4a49-b423-d1db0e4ddcf6}, pn_exceptions: [eoIyYl38TQOMJ8gNmpJ37R:APA91bGvsgjmv7w7rwZroCAl2n-vLgW_erGVioAyDzHLwDumOo7LqQR9fiMfB8U3QBIW4dh7pgI0z_KegmJrFuWci8lvWszYgELkuuc1RFqO8qVgHVXDYwMHBiAxPzDGDe-_P4umcFQN]}, type: text, id: 1646b3bc-edab-4a49-b423-d1db0e4ddcf6, channel: ch_br-test_rr-test} => time token 15997444971171894 incoming => 15997444971171894 history message {content: aaaaa, time: 1599744715087, fromUser: br-test, pn_gcm: {data: {content: aaaaa, channel: ch_br-test_rr-test, messageId: e853794e-59a6-44b6-a626-d4ee76edde8f}, pn_exceptions: [eoIyYl38TQOMJ8gNmpJ37R:APA91bGvsgjmv7w7rwZroCAl2n-vLgW_erGVioAyDzHLwDumOo7LqQR9fiMfB8U3QBIW4dh7pgI0z_KegmJrFuWci8lvWszYgELkuuc1RFqO8qVgHVXDYwMHBiAxPzDGDe-_P4umcFQN]}, type: text, id: e853794e-59a6-44b6-a626-d4ee76edde8f, channel: ch_br-test_rr-test} => time token 15997447157732952 incoming => 15997447157732952 history message {content: acx, time: 1599744724886, fromUser: br-test, pn_gcm: {data: {content: acx, channel: ch_br-test_rr-test, messageId: 0812af69-6ebf-4873-9393-d1b75bfeaa2b}, pn_exceptions: [eoIyYl38TQOMJ8gNmpJ37R:APA91bGvsgjmv7w7rwZroCAl2n-vLgW_erGVioAyDzHLwDumOo7LqQR9fiMfB8U3QBIW4dh7pgI0z_KegmJrFuWci8lvWszYgELkuuc1RFqO8qVgHVXDYwMHBiAxPzDGDe-_P4umcFQN]}, type: text, id: 0812af69-6ebf-4873-9393-d1b75bfeaa2b, channel: ch_br-test_rr-test} => time token 15997447256655564 incoming => 15997447256655564 history message {content: asdfgh, time: 1599745178027, fromUser: br-test, pn_gcm: {data: {content: asdfgh, channel: ch_br-test_rr-test, messageId: d18b9057-52ca-4ca4-b6ed-9c4a2cda791f}, pn_exceptions: [eoIyYl38TQOMJ8gNmpJ37R:APA91bGvsgjmv7w7rwZroCAl2n-vLgW_erGVioAyDzHLwDumOo7LqQR9fiMfB8U3QBIW4dh7pgI0z_KegmJrFuWci8lvWszYgELkuuc1RFqO8qVgHVXDYwMHBiAxPzDGDe-_P4umcFQN]}, type: text, id: d18b9057-52ca-4ca4-b6ed-9c4a2cda791f, channel: ch_br-test_rr-test} => time token 15997451787186398 incoming => 15997451787186398 history message {content: qwerty, time: 1599745202023, fromUser: br-test, pn_gcm: {data: {content: qwerty, channel: ch_br-test_rr-test, messageId: 8dedd5f2-1eb7-41c1-8ad2-02455189cd4a}, pn_exceptions: [eoIyYl38TQOMJ8gNmpJ37R:APA91bGvsgjmv7w7rwZroCAl2n-vLgW_erGVioAyDzHLwDumOo7LqQR9fiMfB8U3QBIW4dh7pgI0z_KegmJrFuWci8lvWszYgELkuuc1RFqO8qVgHVXDYwMHBiAxPzDGDe-_P4umcFQN]}, type: text, id: 8dedd5f2-1eb7-41c1-8ad2-02455189cd4a, channel: ch_br-test_rr-test} => time token 15997452027059086 incoming => 15997452027059086

[devopsokdone]

Further update, now we are getting only messages after an end timeToken. The updated Code and new logs below, the updated query is - we need messages after the timeToken specified but we are getting the message with that timeToken and messages after that. How can we get messages after the timeToken specified?

Code: final messageHistory = await client.batch.fetchMessages( list.toSet(), end: Timetoken(lastMessage.timeToken), reverse: true, );

Last MessageTime When Connection closed {fromUser: bb-test, id: c5a68806-577d-44d2-80cb-05de0942b018, content: qwert, type: text, time: 1599811254157, channel: ch_aa-test_bb-test, pn_gcm: {data: {channel: ch_aa-test_bb-test, messageId: c5a68806-577d-44d2-80cb-05de0942b018, content: qwert}, pn_exceptions: []}} => timetoken 15998112549240776

Fetch Message History Log lastMessageTime 15998112549240776

history message {content: qwert, time: 1599811254157, fromUser: bb-test, pn_gcm: {data: {content: qwert, channel: ch_aa-test_bb-test, messageId: c5a68806-577d-44d2-80cb-05de0942b018}, pn_exceptions: []}, type: text, id: c5a68806-577d-44d2-80cb-05de0942b018, channel: ch_aa-test_bb-test} => time token 15998112549240776 history message {content: 123, time: 1599811328586, fromUser: aa-test, pn_gcm: {data: {content: 123, channel: ch_aa-test_bb-test, messageId: 3a7533e3-5d95-40bb-9095-916f6b065e36}, pn_exceptions: [eoxHdExDQPuGGaxT0ColZ4:APA91bE_n_r0dlimKu9Pmi-y4WXWZlG2OX4xeQmD2caYtClgQ3jcvyqSRE9aRFd1wMW_YVEL-MCeX9fBkYu0A_iL26UlrGFVW5v-DUyHW2tHjrdw2pfuZxuLmKMSqMdIUFYR1byAs5SJ]}, type: text, id: 3a7533e3-5d95-40bb-9095-916f6b065e36, channel: ch_aa-test_bb-test} => time token 15998113313590379

[are]

Hi there! Please keep in mind, that history methods will always return all messages (up to a limit), without knowing if they were previously obtained or not. Another thing that is important to remember is that end parameter is inclusive, which means that if you are using the latest message timetoken, it will match that message too. There are two solutions for this:

1. Use your internal message IDs and deduplicate on those,
2. Pass the end parameter as Timetoken(lastMessage.timeToken + 1) - so 1 more than the previous message.

[devopsokdone]

Below is what worked for us after a weeks effort.

Scenario: -t100 is the oldest TimeToken on server t000 is the TimeToken after which we need to get the messages (last message on the client) t100 is the latest TimeToken on server

Requirement: We want to retrieve messages with timeToken t001 to t100 via history, as our client went offline after message with timeToken t000

How things works as per PubNub documentation: endTimeToken will get all messages 100 to 0 (newest message on the server to timeToken provided including 0) right to left startTimeToken will get all messages -100 to -1 (oldest message on the server to timeToken Provided excluding 0) left to right

Here is the logic that worked for us Original fetch - count 25 + check if 25 messages are retrieved run once again no startTimeToken endTimeToken t000 We get message t100 to t076

Subsequent fetch - count 25 + check if 25 messages are retrieved run once again startTimeToken t076 endTimeToken t000 We get message t075 to t051

2nd subsequent fetch - count 25 + check if 25 messages are retrieved run once again startTimeToken t051 endTimeToken t000 We get message t050 to t026

3rd subsequent fetch - count 25 + check if 25 messages are retrieved run once again startTimeToken t026 endTimeToken t000 We get message t025 to t001

Also the reverse: true/false options works on each batch so one can say get me t100 to t076 or t076 to t100 (reverse will not start getting messages from t001 instead of t100).

Feel free to correct if a better way is possible.

[are]

This is mostly correct. I just want to bring your attention to the fact, that if you want to obtain history of one specific channel, you can use pubnub.channel('myChannel') to get an instance of a Channel and then you can use .messages and .history.

var myChannel = pubnub.channel("myChannel");
var history = myChannel.messages(from: Timetoken(0), to: Timetoken(100));

await history.fetch();

print(history.messages);