RealmResults allAsync and queryAsync

[ebelevics]

Problem

I'm making Todo app and here are simplified models:

@RealmModel()
class $Task {
  @PrimaryKey()
  late Uuid uuid = Uuid.v4();
  late List<$TaskSession> sessions;
}

@RealmModel()
class $TaskSession {
  @PrimaryKey()
  late Uuid uuid = Uuid.v4();

  late List<$TaskTimeInterval> timeIntervals;
}

@RealmModel(ObjectType.embeddedObject)
class $TaskTimeInterval {
  late DateTime startTime;
  late DateTime? endTime;
}

And one of the tasks is to calculate total time spent in tasks. It is fine when I have few records, but when I'm using:

return tasks.fold<Duration>(
  Duration.zero,
  (totalDuration, task) => totalDuration + calculateTaskDuration(task),
);

Duration calculateTaskDuration(Task task) {
  return task.sessions.fold<Duration>(
    Duration.zero,
    (totalDuration, session) => totalDuration + calculateSessionDuration(session),
  );
}

Duration calculateSessionDuration(TaskSession session) {
  return session.timeIntervals.fold<Duration>(
    Duration.zero,
    (totalDuration, timeInterval) => totalDuration + timeInterval.endTime!.difference(timeInterval.startTime),
  );
}

the UI gets freezed for brief portion of seconds. I am ok that it takes some time to calculate complex calculations, but so far I can't find a way how to do it async/await, to show some loading indicator.

I do understand that this approach loads all data, but there are times, when this is necessary and Iterable from RealmResults, just doesn't suit it. When RealmResults are converted to toList(), then all calculation and operations with data or close instant, but so far I can't find a way how to do it async. Maybe someone can help?

Solution

Have a possibility to use read methods as all and query async, with naming convention allAsync and queryAsync that returns Future (of List or RealmList or some other class, don't know). After loading final tasks = await realm.allAsync<Task>() you can do any calculation whatsoever, as I can operate with preloaded data. And UI will show everything accordingly with loading indicator if necessary.

Alternatives

Maybe I'm unaware of possible solutions, but what I have tried, is making heavy sync code async but unfortunately all made attempts still didn't solve UI freeze or just got error (Illegal argument in isolate message: object is unsendable). Tried Future.wait, SendPort, compute(), but with no good results.

How important is this improvement for you?

Would be a major improvement

Feature would mainly be used with

Local Database only

[desistefanova]

Hi @ebelevics, These methods all and query are really fast. They return RealmResults, but they don't load all the data. The data are actually loaded when you start to iterate and read the fields (columns) of the objects. So making these methods async won't help. It will be better if you wrap the whole calculations method into an async method and to return a Future. Then notify the UI once the future returns the value.

 Future<Duration> calculate() async {
      final tasks = realm.all<Task>();

      return tasks!.fold<Duration>(............
           //Add your calculations here
      );
    }

    calculate().then((value) => setState(() {
          _total = value;
        }));

[ebelevics]

Hmm, I'll give a try

[ebelevics]

Here is some example, which is more close to my real life scenario. I have bottom navigation, where on "Analysis" navigation I show overall global analytics of tasks, but during navigation the app freezes for some portion of second. On this example I have simulated on 50 000 tasks.

I did try Future, but from my understanding it is working on main thread (that is why there is UI freeze), I tried to create Isolate, but it returned error [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Invalid argument(s): Illegal argument in isolate message: object is unsendable - Library:'package:realm/src/results.dart' Class: RealmResults (see restrictions listed atSendPort.send()documentation for more information).

So I have no idea how to make this process smooth and working.

import 'dart:math';

import 'package:flutter/material.dart';
import 'package:realm/realm.dart';

part 'main_realm.g.dart';

///------------- Models
@RealmModel()
class $Task {
  @PrimaryKey()
  late Uuid uuid;
  late List<$Session> sessions = [];
}

@RealmModel()
class $Session {
  @PrimaryKey()
  late Uuid uuid = Uuid.v4();

  Duration get totalTime => Duration(minutes: Random().nextInt(10), seconds: Random().nextInt(59));
}

///------------- Main
late final Realm realm;

void main() {
  final config = Configuration.local([Task.schema, Session.schema]);
  realm = Realm(config);
  runApp(const MyApp());
}

///------------- Widgets

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return const MaterialApp(
      home: MyHomePage(),
    );
  }
}

class MyHomePage extends StatefulWidget {
  const MyHomePage({super.key});

  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  int _currentIndex = 0;

  void _onTabTapped(int index) {
    setState(() {
      _currentIndex = index;
    });
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: _buildPage(_currentIndex),
      bottomNavigationBar: BottomNavigationBar(
        currentIndex: _currentIndex,
        onTap: _onTabTapped,
        items: const [
          BottomNavigationBarItem(icon: Icon(Icons.home), label: 'Tasks'),
          BottomNavigationBarItem(icon: Icon(Icons.settings), label: 'Analysis'),
        ],
      ),
    );
  }

  Widget _buildPage(int index) {
    switch (index) {
      case 0:
        return const TaskPage();
      case 1:
        return const AnalysisPage();
      default:
        return Container();
    }
  }
}

///------------- Page1
class TaskPage extends StatefulWidget {
  const TaskPage({super.key});

  @override
  State<TaskPage> createState() => _TaskPageState();
}

class _TaskPageState extends State<TaskPage> {
  late RealmResults<Task> tasks;

  @override
  void initState() {
    // for (var i = 0; i <= 40000; i++) {
    //   var task = Task(Uuid.v4());
    //
    //   var sessions = <Session>[];
    //
    //   for (var j = 0; j <= Random().nextInt(4); j++) {
    //     sessions.add(Session(Uuid.v4()));
    //   }
    //
    //   task.sessions.addAll(sessions);
    //
    //   realm.write(() {
    //     print('Adding Task $i to Realm.');
    //     realm.add(task);
    //   });
    // }

    tasks = realm.all<Task>();
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        appBar: AppBar(
          title: const Text('Todo app'),
        ),
        body: Center(
          child: Column(
            children: [
              Text('\nThere are ${tasks.length} tasks\n'),
              Expanded(
                child: ListView.builder(
                  itemCount: tasks.length,
                  itemBuilder: (context, i) {
                    final task = tasks[i];

                    final textWidget = Text('Task "${task.uuid}" with ${task.sessions.length} sessions');

                    return textWidget;
                  },
                ),
              ),
            ],
          ),
        ),
      ),
    );
  }
}

///------------- Page2
class AnalysisPage extends StatefulWidget {
  const AnalysisPage({super.key});

  @override
  State<AnalysisPage> createState() => _AnalysisPageState();
}

class _AnalysisPageState extends State<AnalysisPage> {
  final timer = Stopwatch();

  var totalTaskTime = Duration.zero;
  late RealmResults<Task> tasks;

  @override
  void initState() {
    timer.start();

    tasks = realm.all<Task>();
    print("1:${timer.elapsed}");

    // compute(calculation, null);
    calculation(null).then((value) {
      print("3:${timer.elapsed}");
      totalTaskTime = value;
      setState(() {});
    });

    print("2:${timer.elapsed}");

    super.initState();
  }

  Future<Duration> calculation(dynamic data) async {
    print("4:${timer.elapsed}");
    final tasks = realm.all<Task>().toList();
    print("5:${timer.elapsed}");
    return tasks.fold<Duration>(Duration.zero, (prev, t) {
      return prev + t.sessions.fold(Duration.zero, (prev, s) => prev + s.totalTime);
    });
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        appBar: AppBar(
          title: const Text('Todo app'),
        ),
        body: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: [
            Center(child: Text('\nThere are ${tasks.length} tasks\n')),
            Center(child: Text('\nTotal spent time was $totalTaskTime \n')),
          ],
        ),
      ),
    );
  }
}

[desistefanova]

Hi @ebelevics! I think the code is ok. The only thing you can do is to remove .toList() on the RealmResults. This goes through all the items and the you iterate them again using fold. You can call final tasks = realm.all<Task>(); once and use it for the tasks.length and for the time calculation. See the example below.

class AnalysisPage extends StatefulWidget {
  const AnalysisPage({super.key});

  @override
  State<AnalysisPage> createState() => _AnalysisPageState();
}

class _AnalysisPageState extends State<AnalysisPage> {
  var totalTaskTime = Duration.zero;
  var tasksCount = 0;
  @override
  void initState() {
    super.initState();
    calculation(null).then((value) {
      totalTaskTime = value;
      setState(() {});
    });
  }

  Future<Duration> calculation(dynamic data) async {
    final tasks = realm.all<Task>();
    tasksCount = tasks.length;
    return tasks.fold<Duration>(Duration.zero, (prev, t) {
      return prev + t.sessions.fold(Duration.zero, (prev, s) => prev + s.totalTime);
    });
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        appBar: AppBar(
          title: const Text('Todo app'),
        ),
        body: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: [
            Center(child: Text('\nThere are $tasksCount tasks\n')),
            Center(child: Text('\nTotal spent time was $totalTaskTime \n')),
          ],
        ),
      ),
    );
  }
}

You can also store the calculated totalTaskTime to another field into realm in case it is not going to be edited. So that next time you read it it will be already calculated.

About Page 1, I can suggest using StreamBuilder. This will update your list automatically in case of any new changes.

///------------- Page1
class TaskPage extends StatefulWidget {
  const TaskPage({super.key});

  @override
  State<TaskPage> createState() => _TaskPageState();
}

class _TaskPageState extends State<TaskPage> {
  @override
  void initState() {
  }

  Container waitingIndicator() {
    return Container(
      color: Colors.black.withOpacity(0.2),
      child: const Center(child: CircularProgressIndicator()),
    );
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        appBar: AppBar(
          title: const Text('Todo app'),
        ),
        body: Center(
          child: StreamBuilder<RealmResultsChanges<Task>>(
              stream: realm.query<Task>("TRUEPREDICATE SORT(uuid ASC)").changes,
              builder: (context, snapshot) {
                final data = snapshot.data;
                if (data == null) return waitingIndicator();

                final results = data.results;
                return Column(children: [
                  Text('\nThere are ${results.length} tasks\n'),
                  Expanded(
                    child: ListView.builder(
                      shrinkWrap: true,
                      itemCount: results.realm.isClosed ? 0 : results.length,
                      itemBuilder: (context, i) {
                        final task = results[i];
                        final textWidget = Text('Task "${task.uuid}" with ${task.sessions.length} sessions');
                        return results[i].isValid ? textWidget : Container();
                      },
                    ),
                  ),
                ]);
              }),
        ),
      ),
    );
  }
}

[ebelevics]

I did apply your code for analysis, but this doesn't resolve UI freeze during calculation. As seen in video on pressing Analysis button, the button ink animation in not smooth, while pressing on Tasks it is. The 50000 is a bit extreme number, but it was meant for example purposes, tho when those calculations stack up in real app, I can't find clear way to make this experience smoother. Storing calculated totalTaskTime is sure an option, but I have feeling that this solution more error prone.

device-2023-07-18-101707.webm

[desistefanova]

It always needs time when you iterate between all the items. If you want to avoid storing the calculated results then perhaps you may calculate them async in advance when the first tab is populated and then to attach to listen for changes on the RealmResuls in order to re-calculate again if the data has changed. When the users open the tab the data will be ready to be displayed.

[ebelevics]

"then perhaps you may calculate them async in advance when the first tab is populated" This is exactly what I want to achieve but had no luck in code, even in this example successfully. I would be totally fine if when I click Analysis for first time, it shows loading indicator, while data are getting iterated and calculated, but so far all my attempts does freeze UI. Would be appreciated for some directions.

[nirinchev]

Not a flutter developer, so pardon the eventual inaccuracies in my reply, but you probably want to run the calculation on a different isolate. Now, the issue is that Realm objects are bound to the isolate they were created on, so you can't just move them around (guessing this is why you were getting Illegal argument in isolate message: object is unsendable). What you need to do is re-open the Realm, do the calculation and return that. Something like:

final durationInMs = await Isolate.run(() {
  final bgConfig = Configuration.local([Task.schema, Session.schema]);
  final bgRealm = Realm(bgConfig);
  return bgRealm.all<Task>().fold<Duration>(Duration.zero, (prev, t) {
    return prev + t.sessions.fold(Duration.zero, (prev, s) => prev + s.totalTime);
  }).inMilliseconds; // need to change it to int as Duration is not sendable I think
});

final duration = Duration(milliseconds: durationInMs);

[ebelevics]

@nirinchev Thank you, this indeed solved the issue and now transition is much smoother.

[ebelevics]

For those who want to get realm models as list and operate with all loaded data (for example grouping, sorting or filtering data especially when if-logic is in child models) or just separate DB model logic from App model logic (to preserve Clean Architecture principles) without UI freezing hiccups (if occurs), then thanks to @nirinchev suggestion it is possible to do it with:

  Future<List<Task>> getAllTasksAsync() async {
    final data = await Isolate.run(() {
      final bgConfig = Configuration.local([TaskLocal.schema, TaskSessionLocal.schema]);
      final bgRealm = Realm(bgConfig); // initializes DB in separate isolate, so that main UI thread doesn't freeze
      final localData = bgRealm.all<TaskLocal>(); // gets all Local data from DB
      final appData = List<Task>.from(localData.map((l) => l.asApp())); // then converts all local data to app data
      return appData; // and returns back
    });
    return data;
  }

returning RealmObjects I believe is not possible because RealmObject implements Finalizable, where SendPort.sendwhich is part of Isolate.run doesn't support sending such objects (https://api.dart.dev/stable/3.0.6/dart-isolate/SendPort/send.html).

Of course there could be some code optimizations, but the idea works, and am super glad that I can continue using Realm as DB solution for my use cases.

[ebelevics]

Just noticed a real life scenario where I needed to use allAsync function working with groupBy method. I needed to create grouped listview where I have two subgroups. So in the end I got from List<TaskRecord> ---> Map<DateTime, Map<Task, List<TaskRecord>>>. But without using isolates(Isolate.run) my UI would freeze on longer list. Also that I can't pass RealmObjects via SendPort forces me to create separate models.

I will reopen an Issue as I would like to hear some solutions to groupBy problem. For example simple chat room. You have lots of messages in chat and each message has createdAt DateTime. Now creating Map<DateTime, List<Message>> via groupBy would affect performance and UI in large lists, as every items get iterated. How would you solve this problem?

ThreadSafeReference from @nirinchev would for sure solve this issue.

[nirinchev]

The solution is the same as with a list - you can do the grouping on a background thread and pass the entire map back to the UI. Alternatively, you can load the data in batches and process more and more messages as the user scrolls on the screen, similarly to what the iOS Messages UI looks like. In any case, this is a general-purpose question for working with large datasets and not something necessarily related to Realm (passing the objects as thread safe references would not work well here if you have a million of those).

I'm going to close this as there's not much we can do on the Realm side with the current design of the database.