too slow on Android

[FunnyDevs]

I tried it on Android and is slower than sqlite. Any improvements?

[anidotnet]

For which operations do you see performance degradation? Do you have the benchmark code to share? May be I can look into it and suggest something. A vague remarks like this is not much helpful for me.

[FunnyDevs]

i make a test with these POJOs

public class PrivateConversation
{
    private String objectId;
    private User user1;
    private User user2;
    private Integer total;
    private String lastMessage;

   //...gettes and setters
}

public class User
{

    private String objectId;
    private String about;
    private String name;
    private String image;
    private Boolean isSocial;
    private String password;
    private String location;

}

public class PrivateMessage
{
    private String objectId;
    private User sender;
    private String text;
    private User recipient;
    private PrivateConversation privateConversation;
}

I saved 28 PrivateMessage. When i do a find operation, it takes 200-300ms, in sqlite 60-100 ms. I tested on a Samsung s4 mini

[anidotnet]

Can you please also provide the code you have written for - creating the database, inserting the data and find operation? Have you created any index and using it for find operation?

One thing you need to remember, when you use objects, the json serialization overhead is there. If you need improvement, you can use the NitriteCollection instead of ObjectRepository. In case of NitriteCollection there is no serialization overhead involved.

If you have enabled compression or compaction while creating the database, it may slow down the write operations. If you share the full code, I can give you better insight.

[FunnyDevs]

Creating database

db = Nitrite.builder()
                .filePath(getFilesDir()+"/test.db")
                .openOrCreate();

Insert Data privateMessageObjectRepository.insert(privateMessageList.toArray(new PrivateMessage[0]));

find data (i have created no index because the database is popolated with only 28 values) privateMessageList = privateMessageObjectRepository.find().toList();

[anidotnet]

Your code looks alright at a glance. It seems the json serialization overhead is making all the differences. Can you try with NitriteCollection by using Document directly instead of objects to check if you gain anything or not?

While using objects, I cannot bypass the serialization overhead. Anyway, the concerned logic is there in this file ObjectCursor, may be you can have a look and suggest something to improve it.

[FunnyDevs]

I try with Document, mapping only Document<--> PrivateMessage and it's very fast, it takes 10-20 ms. The Json mapping of "PrivateMessage" class is the bottleneck. Maybe using another mapping tecnique like KryoSerialization or MessagePack could improve the serialization. Inside the fetched Documents every reference of other object (User and PrivateConversation) is resolved very fast.

[anidotnet]

I'll take a look at those libraries, but also I need to verify if they have the same customization capabilities like jackson or not. Meantime you can attach your own serialization library also here and bypass jackson. All you need to do is implement NitriteMapper interface using your serialization library and set it while creating the database.

db = Nitrite.builder()
                .filePath(getFilesDir()+"/test.db")
                .nitriteMapper(myCustomMapper)
                .openOrCreate();

You can have look at this class as an example JacksonMapper. If you see performance improvement you can contribute the code here also. I'll be more than happy to accept the PR.

[anidotnet]

I just had a quick look at both Kryo and MessagePack, but what I understood is they are not json serialization libraries. Kryo does more of a binary serialization and MessagePack generates an encoded version of json, both of them are not suitable in this use case. The reason is, we need a serializer which can creates a key-value map from an object which will be used to convert an object instance to a Document, which jackson does flawlessly. Please correct me if I have missed something.

[FunnyDevs]

No,they are not json serialization libraries. There are other json libraries faster than jackson (like LoganSquare) but they require a pojo annotation process...

[anidotnet]

Annotation processing is also not favorable in this use case, user needs to have complete control over the pojo.

[anidotnet]

I validated Boon with Jackson. They seem to have the same response time for map to object conversion for a simple class. Moreover, boon does not seems to have any functionality to handle circular reference. So jackson is better suited here than boon.

Need to validate LoganSquare also. What I think is to speed up the process, I can introduce assisted serialization much like Externalizable technique in java. That way I can somewhat bypass the reflection overhead. Let me know what is your opinion.

[FunnyDevs]

Yes, i think it's the best approach. Reflection is a performance bottleneck in Android if used for a long number of classes. In fact even Jackson have a module to bypass reflection (Jackson Afterburner Module)

[anidotnet]

I have committed the changes in master. All you need to do is to implement your POJO classes from Mappable interface. One example can be found here User.java.

But remember, artifact is not published to maven yet, so you have to build it from master. Let me know if that fulfil your performance requirement or not.

[FunnyDevs]

ok,within next week-end i'll try these new changes!

[FunnyDevs]

I'm trying your changes inside my complex logic (i have used document to avoid jackson databind), but now i have always problem with "NO2.2004: store is closed " on NitriteCollection after used ObjectRepository.

[anidotnet]

If possible can you please provide the relevant code section and error stack trace? There is a very little info here. The store closed error only comes if the store is closed by the user prior calling any operation on NitriteCollection.

And to avoid databind, you don't need to use Document now directly. You can use object and ObjectRepository, but your object must implement Mappable interface.

And can you clarify what did you mean by that?

on NitriteCollection after used ObjectRepository.

[FunnyDevs]

I have mapped the entire applicativo with documents to avoid databind. Now, to try your changes, i have created a local branch of my app and changed only a section to test objectrepository ( instead change entire project) but i have this kind of problem. It Is even difficult give you a specific snippet of code. I think only that document and objectrepository together have problems. So the only solution Is trying to remap new branch with only objectrepository(2-3 hours of work)

[anidotnet]

Yes, if you use ObjectRepository, don't use Document, use objects. If you use NItriteCollection use only Document. But after my recent changes, you can discard your Document related changes and use ObjectRepository only. Just remember to implement Mappable for your POJOs.

[FunnyDevs]

i found that the store closes because throw this exception:

Caused by: java.lang.IllegalStateException: Last block not stored, possibly due to out-of-memory [1.4.196/3]

Full log

07-13 23:19:43.486 10911-10911/it.com.myapp W/System.err: java.lang.IllegalStateException: This store is closed [1.4.196/4]
07-13 23:19:43.486 10911-10911/it.com.myapp W/System.err:     at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:765)
07-13 23:19:43.486 10911-10911/it.com.myapp W/System.err:     at org.h2.mvstore.MVStore.checkOpen(MVStore.java:2409)
07-13 23:19:43.486 10911-10911/it.com.myapp W/System.err:     at org.h2.mvstore.MVStore.openMap(MVStore.java:443)
07-13 23:19:43.486 10911-10911/it.com.myapp W/System.err:     at org.h2.mvstore.MVStore.openMap(MVStore.java:427)
07-13 23:19:43.486 10911-10911/it.com.myapp W/System.err:     at org.dizitart.no2.store.NitriteMVStore.openMap(NitriteMVStore.java:87)
07-13 23:19:43.486 10911-10911/it.com.myapp W/System.err:     at org.dizitart.no2.Nitrite.getRepository(Nitrite.java:140)
07-13 23:19:43.486 10911-10911/it.com.myapp W/System.err:     at it.com.myapp.features.home.HomeInteractor.lambda$getLocalPrivateConversationList$6(HomeInteractor.java:306)
07-13 23:19:43.486 10911-10911/it.com.myapp W/System.err:     at it.com.myapp.features.home.HomeInteractor$$Lambda$8.call(Unknown Source)
07-13 23:19:43.486 10911-10911/it.com.myapp W/System.err:     at rx.internal.operators.OnSubscribeCreate.call(OnSubscribeCreate.java:72)
07-13 23:19:43.486 10911-10911/it.com.myapp W/System.err:     at rx.internal.operators.OnSubscribeCreate.call(OnSubscribeCreate.java:32)
07-13 23:19:43.486 10911-10911/it.com.myapp W/System.err:     at rx.Observable.unsafeSubscribe(Observable.java:10346)
07-13 23:19:43.486 10911-10911/it.com.myapp W/System.err:     at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
07-13 23:19:43.486 10911-10911/it.com.myapp W/System.err:     at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err:     at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err:     at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err:     at rx.Observable.unsafeSubscribe(Observable.java:10346)
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err:     at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err:     at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err:     at rx.Observable.unsafeSubscribe(Observable.java:10346)
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err:     at rx.internal.operators.OperatorSubscribeOn$SubscribeOnSubscriber.call(OperatorSubscribeOn.java:100)
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err:     at rx.android.schedulers.LooperScheduler$ScheduledAction.run(LooperScheduler.java:107)
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err:     at android.os.Handler.handleCallback(Handler.java:733)
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err:     at android.os.Handler.dispatchMessage(Handler.java:95)
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err:     at android.os.Looper.loop(Looper.java:146)
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err:     at android.app.ActivityThread.main(ActivityThread.java:5593)
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err:     at java.lang.reflect.Method.invokeNative(Native Method)
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err:     at java.lang.reflect.Method.invoke(Method.java:515)
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err:     at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1283)
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err:     at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1099)
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err:     at dalvik.system.NativeStart.main(Native Method)
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err: Caused by: java.lang.IllegalStateException: Last block not stored, possibly due to out-of-memory [1.4.196/3]
07-13 23:19:43.496 10911-10911/it.com.myapp W/System.err:     at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:765)
07-13 23:19:43.506 10911-10911/it.com.myapp W/System.err:     at org.h2.mvstore.MVStore.storeNowTry(MVStore.java:1094)
07-13 23:19:43.506 10911-10911/it.com.myapp W/System.err:     at org.h2.mvstore.MVStore.storeNow(MVStore.java:1050)
07-13 23:19:43.506 10911-10911/it.com.myapp W/System.err:     at org.h2.mvstore.MVStore.commitAndSave(MVStore.java:1039)
07-13 23:19:43.506 10911-10911/it.com.myapp W/System.err:     at org.h2.mvstore.MVStore.writeInBackground(MVStore.java:2489)
07-13 23:19:43.506 10911-10911/it.com.myapp W/System.err:     at org.h2.mvstore.MVStore$BackgroundWriterThread.run(MVStore.java:2720)

[anidotnet]

As the error says somehow your code hits out of memory error in Mvstore library. On another note, just curious if you are still mixing ObjectRepository with documents or are you using objects only?

[FunnyDevs]

No, i have refactored with only objectrepository.

[anidotnet]

Then you have to look into your code to see how an out-of-memory error is occurring. Start with some simple ObjectRepository operations to verify the performance improvement. Then add your code complexity on top of it one by one to identify the root cause of the error. There is very little information I have to help you.

[FunnyDevs]

The error occurs in a find operation. The Memory error, in my opinion, throws because i have a mappale object with inside other mappable objects

[anidotnet]

I just wrote a quick test with nested mappable objects (MappableEmployee inside MappableDepartment) but it runs fine. I think you need to pay attention to the conversion logic you have written to check if you have accidentally introduced any endless recursion. You can take help from below example.

@Data
@ToString
public class MappableEmployee implements Mappable {
    private String empId;
    private String name;
    private Date joiningDate;
    private MappableEmployee boss;

    @Override
    public Document write(NitriteMapper mapper) {
        Document document = new Document();
        document.put("empId", getEmpId());
        document.put("name", getName());
        document.put("joiningDate", getJoiningDate());

        if (getBoss() != null) {
            Document bossDoc = getBoss().write(mapper);
            document.put("boss", bossDoc);
        }
        return document;
    }

    @Override
    public void read(NitriteMapper mapper, Document document) {
        if (document != null) {
            setEmpId((String) document.get("empId"));
            setName((String) document.get("name"));
            setJoiningDate((Date) document.get("joiningDate"));

            Document bossDoc = (Document) document.get("boss");
            if (bossDoc != null) {
                MappableEmployee bossEmp = new MappableEmployee();
                bossEmp.read(mapper, bossDoc);
                setBoss(bossEmp);
            }
        }
    }
}

@Data
@ToString
public class MappableDepartment implements Mappable {
    private String name;
    private List<MappableEmployee> employeeList = new ArrayList<>();

    @Override
    public Document write(NitriteMapper mapper) {
        Document document = new Document();

        document.put("name", getName());
        List<Document> employees = new ArrayList<>();
        for (MappableEmployee employee: getEmployeeList()) {
            employees.add(employee.write(mapper));
        }
        document.put("employeeList", employees);

        return document;
    }

    @Override
    @SuppressWarnings("unchecked")
    public void read(NitriteMapper mapper, Document document) {
        if (document != null) {
            setName((String) document.get("name"));
            for (Document doc : (List<Document>) document.get("employeeList")) {
                MappableEmployee me = new MappableEmployee();
                me.read(mapper, doc);
                getEmployeeList().add(me);
            }
        }
    }
}

public class MapperTest {
    private JacksonMapper jacksonMapper;

    @Before
    public void setUp() {
        jacksonMapper = new JacksonMapper();
    }

    @Test
    public void testNested() {
        final MappableEmployee boss = new MappableEmployee();
        boss.setEmpId("1");
        boss.setName("Boss");
        boss.setJoiningDate(new Date());

        final MappableEmployee emp1 = new MappableEmployee();
        emp1.setEmpId("abcd");
        emp1.setName("Emp1");
        emp1.setJoiningDate(new Date());
        emp1.setBoss(boss);

        MappableDepartment department = new MappableDepartment();
        department.setName("Dept");
        department.setEmployeeList(new ArrayList<MappableEmployee>() {{ add(boss); add(emp1); }});

        long start = System.currentTimeMillis();
        Document document = jacksonMapper.asDocument(department);
        long diff = System.currentTimeMillis() - start;
        System.out.println(diff);

        start = System.currentTimeMillis();
        MappableDepartment dept = jacksonMapper.asObject(document, MappableDepartment.class);
        diff = System.currentTimeMillis() - start;
        System.out.println(diff);
        assertEquals(department, dept);
    }

}

[FunnyDevs]

Is possibile to use "db.getRepository" for the same class twice? Because i have to navigate between activities and the exception occurs when i call the find in second Activity

[anidotnet]

Yes you can call getRepository multiple times. But you need to remember few things

1. Multiple getRepository call for a particular type open the same underlying MVMap
2. But each getRepository call will give you new instances of ObjectRepository each time, though all of them will point to the same MVMap

So if you inadvertently close one instance, all other instances will also get closed instantly as the underlying MVMap is same, hence you will get a store closed error if you try to access one instance afterwards.

For an idiomatic usage of getRepository() please have a look at the sample android activity

[anidotnet]

If you still could not pin-point the cause, please provide me with a reproducible code to debug.

[anidotnet]

Closing this issue for inactivity. If you still facing any issue feel free to open it.