Add ability to paginate room history

[skylord123]

We need a node that allows for fetching room history.

[JuliaSoriaSmith]

Retweeting, this is something that would be very good. It would be nice to be able to fetch history

[skylord123]

@JuliaSoriaSmith just dropped v0.8.0 which contains a new Paginate Room node.

[JuliaSoriaSmith]

Oh this is great, I can get the last 25 events and I know it has to do with the sync, but I cannot get more than that. I have set the pageKey to the previous pageKey, so when i try to input it into another paginate node it returns the same 25 events. Im a little confused on how to use this. We have a little competition in the server to see who got the most reacts in the last 24 hours so Im trying to go through all of the messages in the last 24 hours

[skylord123]

@JuliaSoriaSmith Can you import this and set it to your room ID and see if it loops over all the events?

[{"id":"e711eca962006962","type":"matrix-paginate-room","z":"0dad84e70d3c608b","name":"","server":"3c12802bc7c793e3","roomType":"str","roomValue":"!JthAtvDowBchgDZznp:skylar.tech","paginateKeyType":"msg","paginateKeyValue":"paginationKey","paginateBackwardsType":"bool","paginateBackwardsValue":"true","pageSizeType":"num","pageSizeValue":"5","x":600,"y":3200,"wires":[["6d50340d965a49de","d479af1611467a6c"],[]]},{"id":"6d50340d965a49de","type":"debug","z":"0dad84e70d3c608b","name":"debug 124","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":790,"y":3200,"wires":[]},{"id":"5507b36fa99afced","type":"inject","z":"0dad84e70d3c608b","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":380,"y":3200,"wires":[["e711eca962006962"]]},{"id":"d479af1611467a6c","type":"function","z":"0dad84e70d3c608b","name":"Loop","func":"node.status({\n    fill: \"green\",\n    text: `${msg.start} - ${msg.end} ${msg.payload ? '' : 'DONE'}`\n});\n\nif (msg.payload) {\n    return msg;\n    setTimeout(function () {\n        node.send(msg);\n    }, 1000);\n}\n\n// return msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":570,"y":3260,"wires":[["e711eca962006962"]]},{"id":"3c12802bc7c793e3","type":"matrix-server-config","name":"@bot:skylar.tech","autoAcceptRoomInvites":false,"enableE2ee":true,"global":true}]

Just realized I didn't update the docs correctly for this node as well. Sorry about that. I'll get that fixed. This node was a bit of a late addition so there may be bugs so if you hit any let me know and I will get them resolved.

Make sure to leave pageKey undefined for the initial call to paginate room node. It will set it for you for subsequent calls.

If msg.payload is false then there are no more records to paginate in the direction you are going.

[JuliaSoriaSmith]

Ah this works perfectly :) thanks

[JuliaSoriaSmith]

Actually i spoke a little too soon,

Im having issues with retrieving the event:

[{"id":"5507b36fa99afced","type":"inject","z":"2d22625d43fa6dc7","name":"","props":[{"p":"current_time","v":"","vt":"date"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":210,"y":920,"wires":[["e711eca962006962"]]},{"id":"d479af1611467a6c","type":"function","z":"2d22625d43fa6dc7","name":"Loop","func":"node.status({\n fill: \"green\",\n text:${msg.start} - ${msg.end} ${msg.payload ? '' : 'DONE'}\n});\n\nif (msg.payload) {\n return msg;\n setTimeout(function () {\n node.send(msg);\n }, 1000);\n}\n\n// return msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":590,"y":980,"wires":[["e711eca962006962"]]},{"id":"6344f86933832496","type":"function","z":"2d22625d43fa6dc7","name":"function 11","func":"function hasMessageOlderThan24Hours(messages) {\n const currentTime = Date.now(); // Current time in milliseconds\n const time24HoursAgo = currentTime - ( 24 * 60 * 60 * 1000 ); // Time 24 hours ago in milliseconds\n\n return messages.some(message => {\n const eventTime = message.origin_server_ts;\n\n \n return eventTime < time24HoursAgo;\n });\n}\n\nconst messages = msg.payload; \nconst isOlderThan24Hours = hasMessageOlderThan24Hours(messages);\n\nif ( isOlderThan24Hours ){\n\n msg.done = true\n\n}\nelse {\n\n msg.done = false\n\n}\n\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":770,"y":980,"wires":[[]]},{"id":"e711eca962006962","type":"matrix-paginate-room","z":"2d22625d43fa6dc7","name":"","server":"1bc204dd34033aac","roomType":"str","roomValue":"!WhvydfGtSCgBTZEsIo:nits.foundation","paginateKeyType":"msg","paginateKeyValue":"paginationKey","paginateBackwardsType":"bool","paginateBackwardsValue":"true","pageSizeType":"num","pageSizeValue":"5","x":580,"y":920,"wires":[["962d5fb69f3a058c"],[]]},{"id":"962d5fb69f3a058c","type":"split","z":"2d22625d43fa6dc7","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","property":"payload","x":770,"y":920,"wires":[["16c7efb011ef944a"]]},{"id":"9c95df81a2012dd5","type":"debug","z":"2d22625d43fa6dc7","name":"debug 126","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1310,"y":920,"wires":[]},{"id":"16c7efb011ef944a","type":"switch","z":"2d22625d43fa6dc7","name":"","property":"payload","propertyType":"msg","rules":[{"t":"eq","v":"payload.event","vt":"msg"}],"checkall":"true","repair":false,"outputs":1,"x":930,"y":920,"wires":[["9c95df81a2012dd5"]]},{"id":"1bc204dd34033aac","type":"matrix-server-config","name":"Hooty","autoAcceptRoomInvites":true,"enableE2ee":true,"global":true}]

The event data is not being returned. Ultimately im trying to filter out messages older than 24 hours, and i initially had a function block, but I couldn't get any value in message.event, it was all coming out as undefined

[skylord123]

@JuliaSoriaSmith I tried to build a flow to filter based on the event time and I am having some strange issues. Debugging it further it looks like the events list in the payload is getting messed up in some sort of strange way. If you debug the msg you will see it has all the fields but if you try to access some of those fields on the event from a function node it keeps saying undefined.

What is odd is if you JSON encode then decode the object it starts working correctly. It's almost like the fields are in some sort of protected state. I'll look into this tonight and get a fix out for it. In the mean time you can get around this by doing the awful JSON encode/decode trick:

let filteredPayload = msg.payload.filter(event => {
    event = JSON.parse(JSON.stringify(event)); // hack to get object properties accessible again until a fixed release is available
    return !(event.event.origin_server_ts && currentTime - event.event.origin_server_ts > (24 * 60 * 60 * 1000));
});

[skylord123]

Update on this:

I created a new branch called upgrade based off the dev branch for handling the upgrade work. Part of this is getting rid of the deprecated libolm and replacing it with the rust crypto. I got it loaded but my existing accounts will not migrate.

25 Sep 13:16:19 - [debug] [matrix-server-config:@redacted2:skylar.tech] Downloading Rust crypto library
25 Sep 13:16:19 - [debug] [matrix-server-config:@redacted:skylar.tech] Downloading Rust crypto library
25 Sep 13:16:19 - [debug] [matrix-server-config:@redacted2:skylar.tech] Initialising Rust crypto-sdk WASM artifact
25 Sep 13:16:19 - [debug] [matrix-server-config:@redacted:skylar.tech] Initialising Rust crypto-sdk WASM artifact
25 Sep 13:16:19 - [debug] [matrix-server-config:@redacted2:skylar.tech] Opening Rust CryptoStore
25 Sep 13:16:19 - [debug] [matrix-server-config:@redacted:skylar.tech] Opening Rust CryptoStore
25 Sep 13:16:19 - [debug] [matrix-server-config:@redacted2:skylar.tech] Counting olm sessions to be migrated
25 Sep 13:16:19 - [debug] [matrix-server-config:@redacted:skylar.tech] Counting olm sessions to be migrated
25 Sep 13:16:19 - [debug] [matrix-server-config:@redacted2:skylar.tech] Counting megolm sessions to be migrated
25 Sep 13:16:19 - [debug] [matrix-server-config:@redacted:skylar.tech] Counting megolm sessions to be migrated
25 Sep 13:16:19 - [info] [matrix-server-config:@redacted2:skylar.tech] Migrating data from legacy crypto store. 21 olm sessions and 36 megolm sessions to migrate.
25 Sep 13:16:19 - [info] [matrix-server-config:@redacted2:skylar.tech] Migrating from legacy crypto to rust crypto. 0/58
25 Sep 13:16:19 - [info] [matrix-server-config:@redacted2:skylar.tech] Migrating from legacy crypto to rust crypto. 1/58
25 Sep 13:16:19 - [info] [matrix-server-config:@redacted2:skylar.tech] Migrating data from legacy crypto store. Step 2: olm sessions (21 sessions to migrate).
25 Sep 13:16:19 - [info] [matrix-server-config:@redacted:skylar.tech] Migrating data from legacy crypto store. 17 olm sessions and 24 megolm sessions to migrate.
25 Sep 13:16:19 - [info] [matrix-server-config:@redacted:skylar.tech] Migrating from legacy crypto to rust crypto. 0/42
25 Sep 13:16:19 - [info] [matrix-server-config:@redacted:skylar.tech] Migrating from legacy crypto to rust crypto. 1/42
25 Sep 13:16:19 - [info] [matrix-server-config:@redacted:skylar.tech] Migrating data from legacy crypto store. Step 2: olm sessions (17 sessions to migrate).
25 Sep 13:16:19 - [debug] [matrix-server-config:@redacted2:skylar.tech] Migrating batch of 9 olm sessions
25 Sep 13:16:19 - [debug] [matrix-server-config:@redacted:skylar.tech] Migrating batch of 12 olm sessions

This causes the client to essentially crash (not Node-RED, just the matrix nodes show disconnected)

So trying to find a solution to that issue. Haven't found anyone else encountering this so that has made it quite a bit more difficult. Element is already using this so there must be something I am doing wrong. That or maybe my local sessions are just corrupted from previous tinkering and it actually works for others. If someone wants to experiment with an existing account it could help troubleshoot this. Looks like switching back to a previous version works without any issue so this isn't a breaking change.