Open theobouwman opened 6 days ago
richvdh
commented
6 days ago @theobouwman it's worth knowing that there are two stores in matrix-js-sdk: the IStore which you have found, but the crypto stack has its own, separate, store. If you want to support end-to-end encryption, you'll also need to update the crypto stack to support a different database implementation. https://github.com/matrix-org/matrix-rust-sdk-crypto-wasm/issues/168 has some discussion about that.
theobouwman
commented
6 days ago @richvdh I know. I am first planning to implement the store without the end-to-end encryption.
But, by implementing the IStore (https://github.com/matrix-org/matrix-js-sdk/blob/develop/src/store/index.ts) should be a good start right?
And will the TimelineWindow class use the configured store as well?
richvdh
commented
6 days ago Fair enough.
I don't think TimelineWindow uses the store directly: it is a layer on top of other things that do. I'm pretty sure that if you implement IStore it will solve your problem.
theobouwman
commented
5 days ago Okay. I made a start.
Unfortunately I am getting a Caught /sync error [TypeError: undefined is not a function] on sync.
storedFilters set filter:@a09a125f-0346-4b98-a81c-5439d927b8bf:test.exploremomo.com:0 [{"definition": {"room": [Object]}, "filterId": "0", "roomFilter": [Object], "roomTimelineFilter": [Object], "userId": "@a09a125f-0346-4b98-a81c-5439d927b8bf:test.exploremomo.com"}]
LOG set filter id by name with filterId FILTER_SYNC_@a09a125f-0346-4b98-a81c-5439d927b8bf:test.exploremomo.com 0
DEBUG Sending initial sync request...
DEBUG FetchHttpApi: --> GET http://192.168.178.166:8008/_matrix/client/v3/sync?filter=xxx&timeout=xxx&_cacheBuster=xxx
DEBUG Waiting for saved sync before starting sync processing...
LOG |HTTP|GET|200| http://192.168.178.166:8000/organization/1195db93-03b2-4b00-b388-6361dec59fae/groups/8a18e426-7cad-4763-bde6-a71175d3171c
LOG |HTTP|GET|200| http://192.168.178.166:8000/organization/c15f1d2f-9127-4802-a872-ffe4066e39fb
LOG |HTTP|GET|200| http://192.168.178.166:8000/organization/c15f1d2f-9127-4802-a872-ffe4066e39fb/feature
LOG |HTTP|GET|200| http://192.168.178.166:8000/organization/c15f1d2f-9127-4802-a872-ffe4066e39fb/events/47f80386-a482-4237-b4db-beaffd58982c/attending
LOG |HTTP|GET|200| http://192.168.178.166:8000/organization/1195db93-03b2-4b00-b388-6361dec59fae/groups/8a18e426-7cad-4763-bde6-a71175d3171c/messages/a5f3edf4-e338-42dc-ab79-848f0f3f8c44/details
LOG |HTTP|GET|200| http://192.168.178.166:8000/organization/1195db93-03b2-4b00-b388-6361dec59fae/groups/8a18e426-7cad-4763-bde6-a71175d3171c/messages/cd16aebc-62ec-4df6-9845-e9ddd41cbf12/details
DEBUG FetchHttpApi: <-- GET http://192.168.178.166:8008/_matrix/client/v3/sync?filter=xxx&timeout=xxx&_cacheBuster=xxx [165ms 200]
LOG |HTTP|GET|200| http://192.168.178.166:8000/organization/f1c06a44-16fc-4791-ba8d-f5a05b235538/events/8a358d46-eb84-4b43-9ede-be7341410e94/attending
LOG |HTTP|GET|200| http://192.168.178.166:8000/organization/f1c06a44-16fc-4791-ba8d-f5a05b235538/events/8a358d46-eb84-4b43-9ede-be7341410e94/messages/c044fcf6-458d-4814-865f-8c8c1c44e257/details
LOG |HTTP|GET|200| http://192.168.178.166:8000/organization/f1c06a44-16fc-4791-ba8d-f5a05b235538/events/66a45490-c7ce-47f4-98dc-12e984820053/messages/1e804092-3650-48f9-b9d2-d7bffa221293/details
LOG |HTTP|GET|200| http://192.168.178.166:8000/organization/f1c06a44-16fc-4791-ba8d-f5a05b235538/events/66a45490-c7ce-47f4-98dc-12e984820053/messages/e181d9db-374e-4a84-990f-ecdb3684b7a4/details
LOG |HTTP|GET|200| http://192.168.178.166:8000/organization/f1c06a44-16fc-4791-ba8d-f5a05b235538/events/66a45490-c7ce-47f4-98dc-12e984820053/messages/4dc2cfd3-d4f2-4b95-8be6-657ea1d49208/details
LOG |HTTP|GET|200| http://192.168.178.166:8000/organization/f1c06a44-16fc-4791-ba8d-f5a05b235538/events/66a45490-c7ce-47f4-98dc-12e984820053/messages/784c368f-492b-4866-b592-f917101bd21c/details
LOG get user @momo-admin:test.exploremomo.com {"_events": {}, "_eventsCount": 5, "currentlyActive": true, "displayName": "@momo-admin:test.exploremomo.com", "events": {"presence": {"content": [Object], "sender": "@momo-admin:test.exploremomo.com", "type": "m.presence"}}, "lastActiveAgo": 114914, "lastPresenceTs": 1733334816197, "modified": 1733334816197, "presence": "online", "rawDisplayName": "@momo-admin:test.exploremomo.com", "userId": "@momo-admin:test.exploremomo.com"}
**ERROR Caught /sync error [TypeError: undefined is not a function]**
LOG set sync data {"account_data": {"events": [[Object]]}, "device_one_time_keys_count": {}, "device_unused_fallback_key_types": [], "next_batch": "s838_4408_36_3019_614_3_1_9_0_1", "org.matrix.msc2732.device_unused_fallback_key_types": [], "presence": {"events": [[Object], [Object]]}, "rooms": {"join": {"!AlFHBMsEClBnOKiFLy:test.exploremomo.com": [Object]}}}
LOG sync PREPARED
LOG mmkv roomids []
LOG sync SYNCING
LOG mmkv roomids []
INFO Resuming queue after resumed sync
DEBUG Attempting to send queued to-device messages
LOG wants save true
LOG wants save true
INFO store:reallySave
DEBUG FetchHttpApi: --> GET http://192.168.178.166:8008/_matrix/client/v3/sync?filter=xxx&timeout=xxx&since=xxx
DEBUG All queued to-device messages sent
LOG |HTTP|GET|200| http://192.168.178.166:8000/organization/9223c358-1135-4e8d-91ea-683c29248ac2
LOG |HTTP|GET|200| http://192.168.178.166:8000/organization/9223c358-1135-4e8d-91ea-683c29248ac2/events/e56eacca-ed33-4248-8f8c-bd6e2ee6850e/messages/8357273f-1fc8-4ff0-8abe-5b629a6fe33c/details
LOG App was opened by a URL: null
DEBUG FetchHttpApi: <-- GET http://192.168.178.166:8008/_matrix/client/v3/sync?filter=xxx&timeout=xxx&since=xxx [266ms 200]
...import { MatrixEvent, Room, User, Filter, IStateEventWithRoomId, IStartClientOpts, IEvent, ISyncResponse, MemoryStore, IStoredClientOpts, KnownMembership, RoomState, SyncAccumulator } from 'matrix-js-sdk';
import { RoomSummary } from 'matrix-js-sdk/lib/models/room-summary';
import { ToDeviceBatchWithTxnId, IndexedToDeviceBatch } from 'matrix-js-sdk/lib/models/ToDeviceMessage';
import { ISavedSync, IStore, UserCreator } from 'matrix-js-sdk/lib/store';
import { deepCopy, MapWithDefault } from 'matrix-js-sdk/lib/utils';
import { MMKVInstance, MMKVLoader } from 'react-native-mmkv-storage';
const WRITE_DELAY_MS = 1000 * 60 * 1; // once every 1 minutes
const isValidFilterId = (filterId?: string | number | null): boolean => {
const isValidStr =
typeof filterId === "string" &&
!!filterId &&
filterId !== "undefined" && // exclude these as we've serialized undefined in localStorage before
filterId !== "null";
return isValidStr || typeof filterId === "number";
}
class MatrixMMKVStore implements IStore {
private storage: MMKVInstance;
private userCreator: UserCreator | null = null;
private readonly syncAccumulator: SyncAccumulator;
accountData = new Map<string, MatrixEvent>();
private syncTs = 0;
constructor() {
this.storage = new MMKVLoader().initialize();
this.syncAccumulator = new SyncAccumulator();
}
private prefixKey(key: string): string {
return `matrix:${key}`;
}
async isNewlyCreated(): Promise<boolean> {
const flag = await this.storage.getBoolAsync(this.prefixKey('isNewlyCreated'));
if (flag === null) {
await this.storage.setBoolAsync(this.prefixKey('isNewlyCreated'), true);
return Promise.resolve(true);
}
return Promise.resolve(false);
}
getSyncToken(): string | null {
return this.storage.getString(this.prefixKey('syncToken')) as string | null;
}
setSyncToken(token: string = ''): void {
this.storage.setString(this.prefixKey('syncToken'), token);
}
storeRoom(room: Room): void {
this.storage.setMap(this.prefixKey(`room:${room.roomId}`), room);
console.log('store room', room.roomId, room)
const storedRoomIds = this.storage.getArray<string>(this.prefixKey('roomIds')) ?? []
let newRoomIds = [...storedRoomIds, room.roomId]
this.storage.setArray(this.prefixKey('roomIds'), newRoomIds)
}
setUserCreator(creator: UserCreator): void {
this.userCreator = creator;
}
getRoom(roomId: string): Room | null {
const room = this.storage.getMap<Room>(this.prefixKey(`room:${roomId}`));
console.log('get room', roomId, room)
return room
}
getRooms(): Room[] {
const roomIds = this.storage.getArray<string>(this.prefixKey('roomIds')) ?? []
console.log('mmkv roomids', roomIds)
if (roomIds.length > 0) {
const roomKeys = roomIds.map(r => `room:${r}`)
const roomData = this.storage.getMultipleItems<Room>(roomKeys, 'map');
const rooms = roomData
.map((item) => item[1]!)
.filter((room) => room);
console.log('get rooms', rooms)
return rooms
}
return []
}
removeRoom(roomId: string): void {
console.log('remove room', roomId)
this.storage.removeItem(this.prefixKey(`room:${roomId}`));
const storedRoomIds = this.storage.getArray<string>(this.prefixKey('roomIds')) ?? []
this.storage.setArray(this.prefixKey('roomIds'), storedRoomIds.filter(r => r !== roomId))
}
getRoomSummaries(): RoomSummary[] {
console.log('get room summaries')
const rooms = this.getRooms();
return rooms.filter(room => room.summary !== null).map((room) => room.summary!);
}
storeUser(user: User): void {
console.log('store user', user)
this.storage.setMap(this.prefixKey(`user:${user.userId}`), user);
}
getUser(userId: string): User | null {
const user = this.storage.getMap<User>(this.prefixKey(`user:${userId}`))
console.log('get user', userId, user)
return user
}
getUsers(): User[] {
console.log('get users')
const keys = this.storage.getAllMMKVInstanceIDs();
const userKeys = keys.filter((key) => key.startsWith('matrix:user:'));
const userData = this.storage.getMultipleItems<User>(userKeys, 'map');
console.log('users:', userData)
return userData
.map((item) => item[1]!)
.filter((user) => user);
}
scrollback(room: Room, limit: number): MatrixEvent[] {
// Placeholder: implement logic for retrieving room scrollback
console.log('scrollback', room, limit)
return [];
}
storeEvents(room: Room, events: MatrixEvent[], token: string | null, toStart: boolean): void {
// Placeholder: implement logic to store room events
console.log('store events', room)
}
storeFilter(filter: Filter): void {
if (!filter?.userId || !filter?.filterId) return;
console.log('store filter', filter)
const filterIdUserIdKey = `filter:${filter.userId}:${filter.filterId}`
// let storedFilter = this.storage.getMap<Filter>(this.prefixKey(filterIdUserIdKey))
// console.log('storedFilter', storedFilter)
this.storage.setMap(this.prefixKey(filterIdUserIdKey), filter)
// if (!storedFilter) {
// console.log('do store filter', filter)
// this.storage.setMap(this.prefixKey(filterIdUserIdKey), filter)
// }
let storedFilters = this.storage.getArray<Filter>(this.prefixKey('filters'))
console.log('storedFilters array:', storedFilters)
if (!storedFilters || storedFilters.length === 0) {
storedFilters = []
console.log('new stored filter')
}
storedFilters = storedFilters.filter(f => `filter:${f.userId}:${f.filterId}` !== filterIdUserIdKey)
storedFilters.push(filter)
console.log('storedFilters set', filterIdUserIdKey ,storedFilters)
this.storage.setArray(this.prefixKey('filters'), storedFilters)
}
getFilter(userId: string, filterId: string): Filter | null {
const filter = this.storage.getMap<object>(this.prefixKey(`filter:${userId}:${filterId}`));
if (!filter) {
console.log('get filter not found', userId, filterId)
return null
}
const f = Filter.fromJson(userId, filterId, filter)
console.log('get filter', userId, filterId, 'filterOBJ', f)
return f
}
getFilterIdByName(filterName: string): string | null {
try {
const filterId = this.storage.getString(this.prefixKey(filterName.replace('FILTER_SYNC_', '')))
if (isValidFilterId(filterId)) {
console.log('get filter id by name', filterName, filterId)
return filterId as string | null;
}
} catch {}
console.log('get filter id by name', filterName, 'null-empty')
return null;
}
setFilterIdByName(filterName: string, filterId?: string): void {
console.log('set filter id by name with filterId', filterName, filterId)
if (isValidFilterId(filterId)) {
this.storage.setString(this.prefixKey(filterName.replace('FILTER_SYNC_', '')), filterId!);
} else {
this.storage.removeItem(this.prefixKey(filterName.replace('FILTER_SYNC_', '')));
}
}
storeAccountDataEvents(events: MatrixEvent[]): void {
console.log('store account data events', events.length)
events.forEach((event) => {
this.accountData.set(event.getType(), event);
});
}
getAccountData(eventType: string): MatrixEvent | undefined {
console.log('get account', eventType)
return this.accountData.get(eventType);
}
async setSyncData(syncData: ISyncResponse): Promise<void> {
this.storage.setMap(this.prefixKey('syncData'), syncData);
console.log('set sync data', syncData)
return Promise.resolve().then(() => {
this.syncAccumulator.accumulate(syncData)
})
}
wantsSave(): boolean {
const now = Date.now();
const want = now - this.syncTs > WRITE_DELAY_MS;
console.log('wants save', want)
return want
}
save(force?: boolean): Promise<void> {
if (force || this.wantsSave()) {
return this.reallySave();
}
return Promise.resolve();
}
reallySave(): Promise<void> {
this.syncTs = Date.now(); // set now to guard against multi-writes
console.info('store:reallySave')
// work out changed users (this doesn't handle deletions but you
// can't 'delete' users as they are just presence events).
// const userTuples: [userId: string, presenceEvent: Partial<IEvent>][] = [];
// for (const u of this.getUsers()) {
// if (this.userModifiedMap[u.userId] === u.getLastModifiedTime()) continue;
// if (!u.events.presence) continue;
// userTuples.push([u.userId, u.events.presence.event]);
// // note that we've saved this version of the user
// this.userModifiedMap[u.userId] = u.getLastModifiedTime();
// }
return Promise.resolve()
// return this.backend.syncToDatabase(userTuples);
}
async startup(): Promise<void> {
// No-op for MMKV initialization
console.log('-------- store startup')
const savedSync = await this.getSavedSync()
console.log('savwd sunc startup', savedSync)
if (savedSync) {
console.log('-------- store startup sync accumulator')
return this.syncAccumulator.accumulate({
next_batch: savedSync.nextBatch,
rooms: savedSync.roomsData,
account_data: {
events: savedSync.accountData
}
}, true)
}
return Promise.resolve()
}
async getSavedSync(copy = true): Promise<ISavedSync | null> {
const res = this.storage.getMap<ISavedSync|null>(this.prefixKey('syncData'))
console.log('get saved sync', res)
return Promise.resolve(res)
// const data = this.syncAccumulator.getJSON();
// if (!data.nextBatch) return Promise.resolve(null);
// if (copy) {
// // We must deep copy the stored data so that the /sync processing code doesn't
// // corrupt the internal state of the sync accumulator (it adds non-clonable keys)
// return Promise.resolve(deepCopy(data));
// } else {
// return Promise.resolve(data);
// }
}
async getSavedSyncToken(): Promise<string | null> {
const syncData = await this.getSavedSync();
return syncData?.nextBatch || null;
}
async deleteAllData(): Promise<void> {
this.storage.clearStore();
}
async getOutOfBandMembers(roomId: string): Promise<IStateEventWithRoomId[] | null> {
console.log('get out of band members', roomId)
const res = this.storage.getArray<IStateEventWithRoomId>(this.prefixKey(`oobMembers:${roomId}`));
return Promise.resolve(res)
}
async setOutOfBandMembers(roomId: string, membershipEvents: IStateEventWithRoomId[]): Promise<void> {
console.log('set out of band members', roomId)
this.storage.setArray(`oobMembers:${roomId}`, membershipEvents);
return Promise.resolve()
}
async clearOutOfBandMembers(roomId: string): Promise<void> {
console.log('clear out of band members', roomId)
this.storage.removeItem(this.prefixKey(`oobMembers:${roomId}`));
}
async getClientOptions(): Promise<IStartClientOpts | undefined> {
return await this.storage.getMapAsync<IStartClientOpts>(this.prefixKey('clientOptions')) as IStartClientOpts | undefined;
}
async storeClientOptions(options: IStartClientOpts): Promise<void> {
await this.storage.setMap(this.prefixKey('clientOptions'), options);
}
async getPendingEvents(roomId: string): Promise<Partial<IEvent>[]> {
return await this.storage.getMapAsync<Partial<IEvent>[]>(this.prefixKey(`pendingEvents:${roomId}`)) || [];
}
async setPendingEvents(roomId: string, events: Partial<IEvent>[]): Promise<void> {
await this.storage.setArray(this.prefixKey(`pendingEvents:${roomId}`), events);
}
async saveToDeviceBatches(batch: ToDeviceBatchWithTxnId[]): Promise<void> {
await this.storage.setArrayAsync(this.prefixKey('toDeviceBatches'), batch);
}
async getOldestToDeviceBatch(): Promise<IndexedToDeviceBatch | null> {
const batches = await this.storage.getArrayAsync<IndexedToDeviceBatch>(this.prefixKey('toDeviceBatches'));
return batches?.[0] || null;
}
async removeToDeviceBatch(id: number): Promise<void> {
const batches = await this.storage.getArray<IndexedToDeviceBatch>(this.prefixKey('toDeviceBatches'));
if (batches) {
const updatedBatches = batches.filter((batch) => batch.id !== id);
await this.storage.setArrayAsync(this.prefixKey('toDeviceBatches'), updatedBatches);
}
}
async destroy(): Promise<void> {
await this.deleteAllData();
}
}
export default MatrixMMKVStore;
Hi,
We are currently using matrix js sdk in our react native app. As indexeddb is not supported in RN we currently use the memory store.
The problem is that when you open a notification from a room there is a loading state because nothin is stored on device. We are using TimelineWindow for our rooms with pagination etc https://matrix-org.github.io/matrix-js-sdk/classes/matrix.TimelineWindow.html.
So I have 2 questions:
Thanks