Bad performance with great amount of data

[Mikhail-Ivanou]

We need to load and store about 120-130 thousand records. Every record contains 11 fields (5 longs, 3 booleans and 3 Strings. I've compared performance for realm.io and SQLite. See attached googledoc https://docs.google.com/spreadsheets/d/1J4V8EVpkphDvZTETy-Xz5zQII1glwcc9PQTOxz6aBFo/edit?usp=sharing

Realm works nice at first, and then it works twice slowly than SQLite.

Tested on Adnroid 4.1 (Samsung S3)

[emanuelez]

Interesting! That's not even that much data actually! Can you provide some code to show how you implemented the inserts? Are you using one transaction per insert or do you batch them together?

[Mikhail-Ivanou]

Here is my code for caching logic.

Realm realm = Realm.getInstance(context); realm.beginTransaction(); SimpleDateFormat timeFormat = TimeFormatUtils.getBaseTimeFormat(); final List list = response.getStations(); for (int i = 0; i < list.size(); i++) { StationEpg item = list.get(i); long stationId = item.getStationId(); List

listings = item.getListings(); for (int j = 0; j < listings.size(); j++) { Listing listing = listings.get(j); ListingRealm listingRealm = realm.createObject(ListingRealm.class); updateListing(listingRealm, listing, stationId, timeFormat); } } realm.commitTransaction();

This code is called once per feed (one row in the spreadsheet) I've updated spreadsheet with one more test (only transactions time). It's ok with transactions. I'll do additional research why my test device slows down

[lsjwzh]

mark

[emanuelez]

When do you begin the transaction? That does not appear in your code

[Mikhail-Ivanou]

I've made some more tests today.

Realm works stable fast with with this code. As you can see, the main difference is only fetch query to get count of all items in database. @Override public void cache(Context context, DataSourceRequest dataSourceRequest, ListingEpgResponse response) throws Exception { Realm realm = Realm.getInstance(context); realm.beginTransaction(); SimpleDateFormat timeFormat = TimeFormatUtils.getBaseTimeFormat(); final List list = response.getStations(); Log.d("TEST_REALM", "Start prepare"); Log.startAction("TEST_REALM"); for (int i = 0; i < list.size(); i++) { StationEpg item = list.get(i); long stationId = item.getStationId(); List

listings = item.getListings(); for (int j = 0; j < listings.size(); j++) { Listing listing = listings.get(j); ListingRealm listingRealm = realm.createObject(ListingRealm.class); updateListing(listingRealm, listing, stationId, timeFormat); } } Log.d("TEST_REALM", "Finish prepare"); Log.endAction("TEST_REALM"); try { realm.commitTransaction(); } catch (Exception e) { Log.e("TEST_REALM", e); } RealmQuery query = realm.where(ListingRealm.class); RealmResults all = query.findAll(); Log.d("TEST_REALM_FETCH", "count " + all.size());

    Log.endAction("TEST_REALM_processListings");
}

[emanuelez]

Nice! If you are using version 0.71.0 you can actually use realm.where(ListingRealm.class).count() :-)

On Wed, Oct 15, 2014 at 12:53 PM, mikhail-ivanou notifications@github.com wrote:

I've made some more tests today.

Realm works stable fast with with this code. As you can see, the main difference is only fetch query to get count of all items in database. @Override https://github.com/Override public void cache(Context context, DataSourceRequest dataSourceRequest, ListingEpgResponse response) throws Exception { Realm realm = Realm.getInstance(context); realm.beginTransaction(); SimpleDateFormat timeFormat = TimeFormatUtils.getBaseTimeFormat(); final List list = response.getStations(); Log.d("TEST_REALM", "Start prepare"); Log.startAction("TEST_REALM"); for (int i = 0; i < list.size(); i++) { StationEpg item = list.get(i); long stationId = item.getStationId(); List listings = item.getListings(); for (int j = 0; j < listings.size(); j++) { Listing listing = listings.get(j); ListingRealm listingRealm = realm.createObject(ListingRealm.class); updateListing(listingRealm, listing, stationId, timeFormat); } } Log.d("TEST_REALM", "Finish prepare"); Log.endAction("TEST_REALM"); try { realm.commitTransaction(); } catch (Exception e) { Log.e("TEST_REALM", e); } RealmQuery query = realm.where(ListingRealm.class); RealmResults all = query.findAll(); Log.d("TEST_REALM_FETCH", "count " + all.size());

Log.endAction("TEST_REALM_processListings");

}

— Reply to this email directly or view it on GitHub https://github.com/realm/realm-java/issues/474#issuecomment-59188668.

Emanuele Zattin (@emanuelez https://twitter.com/emanuelez) SW Engineer & CI Specialist

[Mikhail-Ivanou]

Ok. I'll check with realm.where(ListingRealm.class).count(). But I don't need to know count in that moment.

How can I test InsertOrUpdate for Android?

[emanuelez]

There's no easy way to do it right now so you will have to do it manually performing a query and update if it returns a result or insert if the resultset is empty.

On Wed, Oct 15, 2014 at 1:24 PM, mikhail-ivanou notifications@github.com wrote:

Ok. I'll check with realm.where(ListingRealm.class).count(). But I don't need to know count in that moment.

How can I test InsertOrUpdate for Android?

— Reply to this email directly or view it on GitHub https://github.com/realm/realm-java/issues/474#issuecomment-59191517.

Emanuele Zattin (@emanuelez https://twitter.com/emanuelez) SW Engineer & CI Specialist