Best approach to save data locally

[kordianbruck]

When developing with meteor for the client, it implicitly uses a minimongo db on the clientside, which also stores the data in the local storage. Also this enables users to run queries (sort, filter) on those data sets.

I've been saving the data coming in currently to local HashMaps but sorting and filtering data items is really resource intensive and leads to many UI hangs. Is there any solution or suggestion on how to save this data locally and have decent performance for getting specific subsets?

(One could of course always just subscribe to the specific published feed in every activity but there is not persistence when switching)

[ocram]

Thanks for your question! This is definitely an interesting topic that needs attention.

There's already something in the works that will free you from having to parse the JSON yourself and save the results every time. This is not something that should mess up your domain logic, of course. It's really something that this library should take care of and thus there will be some enhancements.

In the beginning, an in-memory database that consists of Map instances in the application's memory -- just like what you did -- will be the only solution provided. Nevertheless, the library will allow you to provide your own database class that extends or replaces the existing behaviour. That way, you could provide a database adapter that saves to and reads from SQLite.

SQLite is also the alternative that should probably be recommended for your current problems. And if the UI hangs, you probably have to move the resource intensive operations to a separate thread.

Does that help?

[kordianbruck]

In regards of parsing the JSON: I'm currently using the gson library with underlying models - but that does not take into account the changing object structure of mongodb objects. Any ideas on how to tackle this?

There two issues with Maps: a.) They can't be searched or sorted b.) Its not too easy to use theme in a Adapter for a ListView, thus the need to copy the whole thing into a arraylist and losing any update notifications. I've used an observer pattern to counter this but it is less than ideal as the search & sort has to be done every time the dataset changes

SQLite is not an alternative in regards to losing the notification of any updates that happen. The whole idea of DDP & Meteor is of course to let the user know as soon as possible when a change occurs anywhere in the chain. Also you would need to drop the structure and data every time you restart the app. Plus object structure changes are a pain to keep in sync this way.

I'm not sure if we can work together to find a solution similar to minimongo for android. Do you have the map changes on a branch somewhere? Maybe I can contribute something?

[ocram]

Thank you!

These things are not on a public branch yet, but I can certainly do this.

First, let's agree on the problems that really exist and on the solutions that we need, though:

• Are you parsing the JSON into custom objects, "beans" or POJOs? Do you need these for structural safety and to access their member methods? Or would having all data in maps (with keys and values) help you as well? The problem is, obviously, that this library could only provide a built-in database if you can go without your custom classes and have maps only (just as in minimongo).
• For searching and sorting, the plan is to mirror most of the methods that minimongo provides and then implement the search on a map of maps in this library, i.e. clone minimongo's behaviour in the best way possible. For example, if you search for a single property that should have a specified value, the library iterates over all the entries in the map and compares the properties for you.
• No matter if we use the in-memory database or SQLite, that data store is only backing your collections, you don't lose any real-time capabilities. So you are still informed that an update happened to collection myColl where the document myDoc was added. If you know this, you can just query the database to get myDoc from myColl and see the up-to-date values. Does that make sense?

The source code for minimongo is available here, by the way: https://github.com/meteor/meteor/tree/devel/packages/minimongo

As you can see in this new commit, you will be able to pass any instance of a class implementing the new DataStore interface to the constructor. Then either keep that reference in your Activity or fetch it again using mMeteor.getDataStore().

That data store will automatically receive all data updates. The "only" thing that you have to implement in your data store is saving that data (to whatever location you want) and providing read access with queries and lookups.

To make this much more easy, we will provide a built-in solution that keeps all data in memory and will try to emulate as much of minimongo's behaviour as possible.

That being said, I'm sure we can make this work together :)

[kordianbruck]

• I'm using GSON (Checkout "Consuming http://www.javacreed.com/simple-gson-example/) to parse the JSON object into a normal java object (POJO). It makes accessing the members simpler and also gives you type safety instead of having to reparse <String, String> maps every time you need a specific value.
• That is exactly the problem though: (hash)maps are not build for iteration, search or sorting. Also this gets impractical & slow when you work on larger data sets with a couple hundred items in the maps.
• Well if we do the iterate through the map and return an arraylist approach then the arraylist does not get updated when the map changes. We would need something like a observer pattern for the maps so listview adapters can get notified when the underlying data set changes. With SQLite it gets even worse, as you would need to manage update notifications to adapters separately. Plus SQLite would do a lot of work on the flashdrive/disk.

DataStore looks good - that might help to keep all classes updated but its just a first step to really tackle this issue.

I'm not sure if replicating minimongo is the best approach as it might not really work for Java, but it would definitely be nice. I would suggest though to first tackle #48, #34 and #45 to get the project better organized.

[ocram]

Thank you!

Sure, DataStore is only the very first step, really the foundation. The other code just hasn't been committed yet because we still need to discuss how to do this best, it seems.

Well, you're using GSON, while this library uses Jackson. They should be pretty much compatible, though. Anyway, the purpose of DataStore and the upcoming database classes is that you, as a user of this library, don't have to parse the JSON anymore. You should just receive maps in your DataStore subclass.

The fact that you're using POJOs, which is great for type safety, of course, makes this problem more difficult. We probably won't be able to provide a common database interface that uses your custom POJOs. With maps, on the other hand, this will be possible.

The implementation issues (especially search, sorting) is why I've posted the link to the minimongo source code. This might be really helpful.

You said:

(hash)maps are not build for iteration, search or sorting. Also this gets impractical & slow when you work on larger data sets with a couple hundred items in the maps.

Sure, but this is what minimongo has to do as well. I think we might understate Java's performance when iterating merely hundreds of properties in a map and doing simple string comparisons. In JavaScript, your objects are not optimized for search and sorting, either. From minimongo's README:

Internally, all documents are mapped in a single JS object from _id to the document. Besides this mapping, Minimongo doesn't implement any types of secondary indexes.

Regarding your third point:

Well if we do the iterate through the map and return an arraylist approach then the arraylist does not get updated when the map changes.

Is this really a problem? You get the update notifications in the onDataXxx(...) callbacks. Say, a field named myProp has been added to document myDoc in collection myColl. Knowing that, you just query the database to get that document as an immutable Java object that you can use to update your UI. If you need to update your adapter, update it in that callback with the new value returned from the database.

That's the plan, at least. Doesn't this make sense to you? When you get the update notifications in the onDataXxx(...) callbacks, you don't really need reactive collections or observers on the lists, right?

So the plan was not to remove the onDataXxx(...) callbacks. What may be removed in the future, but will definitely be kept for compatibility reasons for now, is the parameters of those callbacks where data is received. You don't need that if you have a database. You only need to know the document that has changed, then.

Maybe @iahvector wants to discuss these ideas as well :) We have already talked about database access. Two ideas were:

• Re-using raw MongoDB queries from the Meteor client (written in JavaScript) and parsing them in this library. But maybe we will not be happy with that much inlined JavaScript code inside our Java code, without any type safety, etc. It would look like this:

  mMeteor.collection("inventory").find("{ type: 'food', price: { $lt: 9.95 } }")

• Implementing a query builder. To me, this seems to be the best solution. Something like this:

  mMeteor.collection("inventory").equals("type", "food").lowerThan("price", 9.95).find()

[kordianbruck]

Sorry for not getting back to you sooner!

Using POJOs was just useful for me, as I know how the objects look like. But yea, we will need to use maps in order to be able to support changing data object structures.

I guess I'm having my performance issues with HashMap as I still use the autopublish package in development, but that might not be an too big issue with subscriptions. I didn't know that minimongo does not optimize for performance on the client side - really interesting!

I'm still unsure on the whole callback thing. I don't really know what the best approach would be, but I'm open for any suggestions. The initial problem is: how does the adapter, that shows a subscription or subset of data, get notified if any of the objects change. Yes, we can register a callback in every adapter, but I think thats no the best approach.

In terms of API: I would prefer the query builder as JSON in strings does not come with any IDE support.

[ocram]

A first draft of the integrated database is available now:

Prerequisites

dependencies {
    compile 'com.github.delight-im:Android-DDP:e4c6f4ba0c'
}

Enabling the database

Pass an instance of Database to the constructor. Right now, the only subclass provided as a built-in database is InMemoryDatabase. So the code for the constructor becomes:

mMeteor = new Meteor(this, "ws://example.meteor.com/websocket", new InMemoryDatabase());

After that change, all database entries received from the server will automatically be parsed, updated in the built-in database and managed for you. That means no more manual JSON parsing!

So whenever you receive database notifications via DdpCallback#onDataAdded, DdpCallback#onDataChanged or DdpCallback#onDataRemoved, the data has already been updated in the database and can be retrieved from there.

Accessing the database

Database database = mMeteor.getDatabase();

Operations

On the database

// String collectionName = "myCollection";
Collection collection = mMeteor.getDatabase().getCollection(collectionName);

String[] collectionNames = mMeteor.getDatabase().getCollectionNames();

int numCollections = mMeteor.getDatabase().count();

On collections

// String documentId = "wjQvNQ6sGjzLMDyiJ";
Document document = mMeteor.getDatabase().getCollection(collectionName).getDocument(documentId);

String[] documentIds = mMeteor.getDatabase().getCollection(collectionName).getDocumentIds();

int numDocuments = mMeteor.getDatabase().getCollection(collectionName).count();

On collections (with the chainable query builder)

Any of the following method calls can be combined to select documents via complex queries:

// String fieldName = "age";
// int fieldValue = 62;
Query query = mMeteor.getDatabase().getCollection(collectionName).whereEqual(fieldName, fieldValue);

// String fieldName = "active";
// int fieldValue = false;
Query query = mMeteor.getDatabase().getCollection(collectionName).whereNotEqual(fieldName, fieldValue);

// String fieldName = "accountBalance";
// float fieldValue = 100000.00f;
Query query = mMeteor.getDatabase().getCollection(collectionName).whereLessThan(fieldName, fieldValue);

// String fieldName = "numChildren";
// long fieldValue = 3L;
Query query = mMeteor.getDatabase().getCollection(collectionName).whereLessThanOrEqual(fieldName, fieldValue);

// String fieldName = "revenue";
// double fieldValue = 0.00;
Query query = mMeteor.getDatabase().getCollection(collectionName).whereGreaterThan(fieldName, fieldValue);

// String fieldName = "age";
// int fieldValue = 21;
Query query = mMeteor.getDatabase().getCollection(collectionName).whereGreaterThanOrEqual(fieldName, fieldValue);

// String fieldName = "address";
Query query = mMeteor.getDatabase().getCollection(collectionName).whereNull(fieldName);

// String fieldName = "modifiedAt";
Query query = mMeteor.getDatabase().getCollection(collectionName).whereNotNull(fieldName);

Query query = mMeteor.getDatabase().getCollection(collectionName).find();

// int limit = 30;
Query query = mMeteor.getDatabase().getCollection(collectionName).find(limit);

// int limit = 30;
// int offset = 5;
Query query = mMeteor.getDatabase().getCollection(collectionName).find(limit, offset);

Query query = mMeteor.getDatabase().getCollection(collectionName).findOne();

Chained together, these calls may look as follows, for example:

Document document = mMeteor.getDatabase().getCollection("users").whereNotNull("lastLoginAt").whereGreaterThan("level", 3).findOne();

On documents

// String fieldName = "age";
Object field = mMeteor.getDatabase().getCollection(collectionName).getDocument(documentId).getField(fieldName);

String[] fieldNames = mMeteor.getDatabase().getCollection(collectionName).getDocument(documentId).getFieldNames();

int numFields = mMeteor.getDatabase().getCollection(collectionName).getDocument(documentId).count();

[romaluca]

For sort the collection is there anything like:

Query query = mMeteor.getDatabase().getCollection(collectionName).sort(field).find(limit);

Thanks!

[ocram]

@romaluca Unfortunately, this is not available, yet. But exactly that syntax is planned for this feature.

We first wanted to know if the database access works in general. If it does, sorting will probably be the first new feature to be added.

In the meantime, you should be able to sort entries with the Arrays.sort(T[], java.util.Comparator) method. Can you try this?

[iahvector]

Hi all, Sorry for very late reply. I was very busy lately and didn't notice the notifications.

@kordianbruck My solution to this was creating a SQLite table for each of my collections and a content provider for the database. In each activity/fragment, I use a CursorLoader to subscribe to changes in the db and display the changes reactively. In the DDP callbacks, I parse the json and add/update records in the db accordingly and notify the content provider with change. The problem with this is that it's very tedious and you have to repeat it for every app.

@mwaclawek great work on the data store. I'll try to test it as soon as I have time. Isn't there any existing NoSQL db or key-value store for Android that can be used instead of the in memory store for more efficiency and maybe built in query builder?

[kordianbruck]

@iahvector how do you handle schema/object structure changes with SQLite? I imagine it might be really complicated at some stages to hande this...

[iahvector]

I don't. You'll have to update your schema in an application update, but I think you'll have to do this with any solution so I don't this should be considered a problem (except for the amount of work required for the change).

[kordianbruck]

@iahvector not with the HashMaps that @mwaclawek implemented. Those are absolutely dynamic and don't care if a property is added or missing from a object...

[iahvector]

You won't need to update the schema, as there isn't any, but you'll have to update your code and make an app update.

[romaluca]

@mwaclawek it works! The best would be have "sort", "whereIn" and "limit" but for now i used this:

Document[] docs = mApplication.meteor.getDatabase().getCollection("comments")
                    .whereEqual("photoId", photo.getId()).find();
            Arrays.sort(docs, new Comparator<Document>() {
                @Override
                public int compare(Document o1, Document o2) {
                    return Utility.getDateMap(o1.getField("createdAt"))
                            .compareTo(Utility.getDateMap(o2.getField("createdAt")));
                }
            });

            for (int i = 0; i < (docs.length > limit - 1 ? limit - 1 : docs.length); i++) {
                adapterComments.add(new Comment(docs[i]));
            }

Do you have any suggests for do something less weight? For now i get the document's array and i put every element in a list of my objects (in this case i used Comment). I use this list for adapter in recyclerview Thanks!

[ocram]

@iahvector and @kordianbruck I didn't find any good existing solution for the data store. Maybe I've missed some. But anyway, I hope the in-memory database is working for now, as a first step. Maybe you can try it when you find some time. Apart from that, I'm open to other solutions that could probably integrated into the new structure of interfaces as well. We could have both the new InMemoryDatabase and some potential SqliteDatabase extend the Database interface. But we can't use SQLite if that means each developer has to maintain their schema in a separate file. So schema-less is probably what we have to go with.

@romaluca That's great! Is that really working in that form? That would be perfect :) For now, isn't that a good first step? It should hopefully be less code than you would have needed without the built-in database. And if we can soon replace those 7 lines for sorting with a simple sort("createdAt") call, won't that be sufficient? The limit is already there, by the way: Just pass the desired maximum number of entries to the find method.

[romaluca]

Thanks! it's a great step! i didn't see the limit :)

[ocram]

@romaluca Maybe we should remove the parameter(s) from the find method and instead add two methods limit and offset, as you suggested. I just thought it would make more sense to have these as parameters to the find method since there's also the findOne method. And findOne wouldn't make any sense if you had set a limit or offset before. But if the parameters are hard to discover, it's not a good solution, either.

[romaluca]

the method with parameter is ok the only thing is rename the parameters of sign, because now is:

find(int i)
find(int i, int i1) 

and i didn't understand that "i" was "limit" :)

[ocram]

@romaluca Oh, sorry for that! The parameter names were already limit and offset in the source code, of course. But apparently, reasonable parameter names and Javadoc were missing everywhere. The build process had to be fixed for that.

Could everybody please update their dependency to the following?

compile 'com.github.delight-im:Android-DDP:v3.1.0'

Please try and see if it's better then. Thanks for spotting this!

[romaluca]

Thanks! tonight i'll try it!

[ocram]

Closing this issue since database access is now available in the latest releases.

The current implementation is not perfect (yet), of course, but I'm sure we can improve it over time so that it fits everyone's needs. And if it doesn't work for you due to performance issues, you can implement the various interfaces, mainly DataStore and Database, to store the data somewhere else, e.g. in a SQLite database, instead of in memory.

Another way to improve performance (and prevent a freezing UI) may be to parse, store and load data from memory in separate threads, not on the main thread.

Whenever you have any questions about the database access, bug reports or feature requests, please feel free to open a new issue :)

Thanks for all your help!