Cannot Access database instance inside another Isolate

[wtoalabi]

Hello everyone!

So, I am working on a project that requires users to update their database by downloading a json file.

Once I get the file, I foreach inside and insert each record to the table that needs it. Easy peasy. But the problem is each downloaded file has an average of 31,000 lines. Apparenty, I am not about to run that kind of huge operation on the main thread. So, I spawn another thread using isolates. I make the network call within the isolate, got the file but I am unable to access the db instance. Nothing just shows up. This is so confusing as I am just starting out in this whole mobile enchiladas. But here I am, stuck and I need your help. Below are the snippets of the code: Outside the isolate:

ReceivePort receivePort = ReceivePort(); isolate = await Isolate.spawn(process, {'sendPort': receivePort.sendPort, 'details': details}); ` Within the isolate

static process(Map detailMap) async { var dbClient = await DB; }

Where DB is a call to the singleton instance of the db; dbClient returns a null. Nothing shows up. If I place a print before calling the await, the print spits out its contents. But anything after the dbClient and its await call, just goes null.

==================== My second option is, could I just pre-generate the data as sql file, and attach or import it into the db when downloaded, without the need for isolates?

Any help that might point me in the direction of what I might be doing wrong is appreciated! Thanks everyone!!!!!!

[hhtokpinar]

Hi, you can try this ORM. You can transfer the records to the object class with automapping and you can queue that records with the object.save() async method. That’s all.. No need to deal with db operations. For more information https://github.com/hhtokpinar/sqfEntity

[wtoalabi]

Wow! That was fast! Yes, I will take this for a spin.

Thanks!

[hhtokpinar]

I invented the saveAll() method for you. :) you must use it..

[wtoalabi]

You are amazing! Thanks! I will check it out and get back to you!!!

[alextekartik]

@wtoalabi Sqflite already use a different thread for database operation. It does support isolate but the process is kind of complex still (see https://github.com/tekartik/sqflite/issues/169). In your case, you should use sqflite in the main isolate (i.e. you will have to communicate with your isolate).

Typically in this scenario, I do json.decode in an isolate using compute but otherwise remain in the main isolate

[wtoalabi]

Hi @alextekartik! I am just seeing this reply...I've been swamped with work recently, apologies for that! Your suggestion is what I have been doing. Within the isolate, I made the http request, get the JSON and sent each object within to the main thread to be handled by sqlflite. But during this process, I noticed jank. My spinner on the UI just hangs. Not when the http request is taking place, but when insertion is being made to the db.

[alextekartik]

Hard to tell whether sqflite is the cullprit here (i.e. can you confirm that if you comment the sqflite code, the UI does not hang). In my experiment, stuterring typically happen in heavy loop in the main isolate not async calls to sqflite.

[wtoalabi]

Thanks for this. I noticed no jank when I decoded the json within the isolate and sent it as a list of maps to the main thread. So, I guess my question is, now that the main thread has received the decoded json(List), how do I go about inserting each Map with sqflite in a more perfomant way, seeing that the List contains about 31,000 Maps inside. I am thinking of doing forEach again but wouldnt this lead back to the same issue?

What would you have done...if you have 31,000 entries within a List that you need to insert into your db?

Thanks for your help!!!

[alextekartik]

You should use batches https://github.com/tekartik/sqflite/blob/master/sqflite/README.md#batch-support and process data for example by 100 entries (or 50 or 1000 depending on your record size and processing time, there is no rule), basically your main thread processing should not block for too long.

[wtoalabi]

Thanks! So,l moved the json to the main thread then did this:

incoming.forEach((each)async{ await dbClient.insert('table', each); }); Surprisingly, aside a pause lasting for about 3 seconds (that is, the rolling stopped), the db execution began seamlessly and I was able to move around the app, while sqflite keeps doing its thing in the background. This is exactly the behaviour I was expecting. However, while the db is being installed, I got "Fatal signal 11 (SIGSEGV)" (which occurs regularly by the way) But I am curious, would this affect the background installation within the device itself when there is such an outage? If it will, can I use transaction to force a rollback if such an error occurs, to prevent half-installed table. And I just read your reply about batches...would that have prevented the initial (temporary jank)? Thanks!!!!

[alextekartik]

A batch will create a transaction and properly rollback in case of crashes. It does not solve your initial issue if it was due to a big initial processing.

Don't use forEach with asynchronous operation (although it will work). batch setup calls are synchronous, only commit is asynchronous.

[wtoalabi]

Thanks so much @alextekartik You pointed me in the right direction. What I eventually did was this: receivePort.listen((incoming) async{ var batch = dbClient.batch(); incoming.forEach((each){ batch.insert(details['key_name'], each); }); await batch.commit(noResult: true);

And it worked perfectly. But what do I know...? Am I even doing this right?

Thanks!