Support for Null values

[mikeholler]

I want to be able to soft-delete items with a created and a deleted field, both of which being Date fields. When I tried creating an object (with no deleted flag), I received this error message:

java.lang.IllegalArgumentException: Null Date is not allowed.
            at io.realm.internal.Row.setDate(Row.java:195)
            at io.realm.RealmDataFileRealmProxy.setEnd(RealmDataFileRealmProxy.java:53)

Soft deletes like this are very common in databases, so it seems reasonable to support them. Are there any plans to support this already in the works?

[emanuelez]

Hello Michael,

thank you for trying Realm out!

We are working to add support for null objects in the core and subsequently in Cocoa and Android. In the mean time your best option is to add a boolean field for every field you want to allow nullability.

Keep the feedback coming!

On Wed, Oct 15, 2014 at 9:33 PM, Michael Holler notifications@github.com wrote:

I want to be able to soft-delete items with a created and a deleted field, both of which being Date fields. When I tried creating an object (with no deleted flag), I received this error message:

java.lang.IllegalArgumentException: Null Date is not allowed. at io.realm.internal.Row.setDate(Row.java:195) at io.realm.RealmDataFileRealmProxy.setEnd(RealmDataFileRealmProxy.java:53)

Soft deletes like this are very common in databases, so it seems reasonable to support them. Are there any plans to support this already in the works?

— Reply to this email directly or view it on GitHub https://github.com/realm/realm-java/issues/478.

Emanuele Zattin (@emanuelez https://twitter.com/emanuelez) SW Engineer & CI Specialist

[vvictor10]

I wonder why this would be categorized as an enhancement. Setting an object/attr to null should be a basic requirement of any database implementation. At the very minimum, there should be an annotation which lets you explicitly allow it to accept null values. This is a serious limitation.

[bmunkholm]

Hi @vicson2010 It's basically due to our definition of the "Enhancement" label :-) We use that for anything new that has to be implemented, and which is not a "bug". And a "bug" is behavior which is implemented, but doesn't work as we intended.

Realm was initially specifically designed to not support null for performance and API usability reasons and since you can relatively easily work around it by just adding an additional property "xyzIsNull" or similar.

But we certainly hear that users are so used to using null, and have existing cloud API's that can benefit from supporting null more directly, that we have decided to implement it. It's not trivial and therefore takes a while - but we are actively working on it now.

Cheers Brian

[vvictor10]

Thanks @bmunkholm . Appreciate the response. I am super interested in Realm for Android and would love for it to mature and compete and beat other options out there. Good luck to you guys. As part of the maturation process, I think at some point, the query API has to scale and accommodate for all the normal database/SQL constructs like sorting with multiple fields, supporting strict parent child relationships(foreign keys), primary keys. A lot of devs have good background with basic database design and querying in general and they would expect those to be available in any DB option.

[jenzz]

This issue has now become more prominent, especially with the latest release 0.77.0 which allows for easy integration with Gson & Retrofit. As @bmunkholm already mentioned, most of the web services these days omit unset fields in the response which obviously result in null values in your POJOs.

So executing the example shown in the docs using the new realm.copyToRealm() method most probably results in an IllegalArgumentException being thrown like shown above. I was trying to work around this limitation using a Gson type adapter for String.class, but Gson doesn't allow you to change the serialized form of either Strings or primitive types. That only works for your own application types.

So I think in order to integrate Realm properly with Gson & Retrofit, support for nullability is required unless anyone can give me an example of a working workaround with the current version :)

[cmelchior]

@jenzz Actually the current implement will handle that case. If you JSON data contains either "null" as value or are missing properties, they will simply be ignored and the default value for the datatype is used instead. For strings that mean "" (empty string) will be inserted instead of null.

You can see the behaviour in the unit test here: https://github.com/realm/realm-java/blob/master/realm/src/androidTest/java/io/realm/RealmJsonTest.java#L230

Also, references to other RealmObjects are allowed to be null currently.

[bmunkholm]

We should update the docs to describe that - if it doesn't already.

[bmunkholm]

@vicson2010 We are for sure not trying to copy SQL databases. We intend to make something better :-) SQL has a lot of nice queries features, but it can also become quite complex and many features are rarely used. With more features usually also comes complexity and even slowness. We will try to strike a balance between features and usability and think hard to get both. That said we have many plans for nice query features in the future - we are not at all done. We are also likely going to enhance the query syntax or even make it radically different to support those more advanced features. Don't know if you noticed that we just added multi column sort in the latest release, as this has been the most requested feature. We really try to listen to users feedback, such that we don't introduce a lot of features users don't need. Cheers

[jenzz]

@cmelchior Thanks for the clarification. I'm not sure if this is relevant, but those unit tests are only testing insertion via createObjectFromJson(). I am using copyToRealm() and seeing crashes like this:

Caused by: java.lang.IllegalArgumentException: Null String is not allowed.
01-18 18:59:05.648    4232-4232/? W/System.err﹕ at io.realm.internal.Row.setString(Row.java:217)
01-18 18:59:05.648    4232-4232/? W/System.err﹕ at io.realm.UserRealmProxy.setEmail(UserRealmProxy.java:96)
01-18 18:59:05.648    4232-4232/? W/System.err﹕ at io.realm.UserRealmProxy.copyToRealm(UserRealmProxy.java:1113)

User is the POJO extending RealmObject with a null field called email.

[cmelchior]

@jenzz Ah yes, sorry. I misunderstood your first message.

I am not sure transparently supporting Null strings for stand alone objects is the proper solution for this as we would add subtle changes to model data.

The current work around would be to do something like this:

User user = restAdapter.getUser();
if (user.getEmail() == null) {
  user.setEmail("");
  user.setEmailIsNull(true); // If it is important to check for null later
}
realm.copyToRealm(user);

Which is not exactly pretty code, but could be refactored into a static method on the User object.

That said, I can see why doing it transparently could be helpful. Also note that this error could happen for Date as well.

@bmunkholm @emanuelez @kneth What do you think?

[jenzz]

@cmelchior Thanks for the workaround.

It works very well, but can become quite tedious especially when you're dealing with larger and/or nested RealmObjects. For now, I've created static helper methods in every RealmObject class and that does the job :)

[yoavst]

http://stackoverflow.com/a/24252578/3807749 that is a good workaround for GSON:

Gson gson=GsonBuilder().registerTypeAdapterFactory(newNullStringToEmptyAdapterFactory()).create();

public static class NullStringToEmptyAdapterFactory<T> implements TypeAdapterFactory {
    public <T> TypeAdapter<T> create(Gson gson, TypeToken<T> type) {

        Class<T> rawType = (Class<T>) type.getRawType();
        if (rawType != String.class) {
            return null;
        }
        return (TypeAdapter<T>) new StringAdapter();
    }
}

public static class StringAdapter extends TypeAdapter<String> {
    public String read(JsonReader reader) throws IOException {
        if (reader.peek() == JsonToken.NULL) {
            reader.nextNull();
            return "";
        }
        return reader.nextString();
    }
    public void write(JsonWriter writer, String value) throws IOException {
        if (value == null) {
            writer.nullValue();
            return;
        }
        writer.value(value);
    }
}

[falkorichter]

Seriously? no null values? I´ll five your project one more change on Android before dropping it.

[bmunkholm]

@falkorichter : Not sure if you saw all the comments - but it's currently being developed and will soon be ready.

[falkorichter]

@bmunkholm +1 :) I´ll wait a little longer then :)

[degill]

Hey there, I will just hook into this conversation.

Reading previous comments I expected the following JSON to return an empty String or even ignoring the whole "name" value when adding it to Realm and executing "getName()" on the added object, but it actually saves the object with "null" (as a valid String object containg the word "null"):

{
    "id": 8147,
    "name": null
}

String json = "{\"id\": 8147,\"name\": null}";
Realm realm = Realm.getInstance(this);
realm.beginTransaction();
realm.createObjectFromJson(Person.class, json);
realm.commitTransaction();
Person first = realm.where(Person.class).findFirst();
Log.d("Activity", first.getName()); //outputs "null", as a String object. String filled with characters "null"

Am I understanding something wrong here?

[vvictor10]

@bmunkholm Any timeline on when this could be out?

[emanuelez]

@vvictor10 the support for null strings is currently under review in the core engine so it's coming soon!

[vvictor10]

@emanuelez Thanks for responding. I haven't been following your releases closely. I am looking forward to this one and the primary key feature. Do you guys have a scheduled release calendar like every 2 weeks or is it ad-hoc based on feature set/fixes etc?

[mmvicsonvictor]

@bmunkholm @emanuelez Any idea if this is going to make it in the next release for Java? Or when it could possibly make it?

[jenzz]

+1 :+1:

[kneth]

We cannot set a date for this feature. Our core (the actual database) has to support null values, and the core team is working hard to finalize it at the moment.

[kaushikmukunda]

+1

[vvictor10]

Hey guys, do we have any updates on this particular enhancement?

[bmunkholm]

We expect to release NULL support gradually, type by type, starting with support for strings. We really try to stay away of promising dates, as we prioritise support and bug-fixing before new features, making it difficult to predict. That said, the core team is putting the final touches on NULL support for strings this week. So I hope the bindings can start integrating it next week. Just to give you an idea that we are getting closer and closer - finally. So we are counting in small weeks now - without promising any specific dates :-)

[bmunkholm]

NULL for numbers of all kinds (including date) is further out though. That might be another month or two or in that range - assuming no priority changes.

[AndreasBackx]

What will be the types supported in the beginning? Only Strings or also primitive types?

[kneth]

Strings will be first. Boxed types will be next (integer and dates).

[freezy]

So it didn't make it into 0.80.2 I see?

[mmvicsonvictor]

Any idea when this would make it in? Thanks.

[kneth]

DO NOT USE THIS IN PRODUCTION

I have created a jar file with partial null support. You can download it as http://static.realm.io/downloads/java/realm-0.80.3-null-support-alpha1.jar, and you have to manually copy the jar file to your app's libs folder and sync your Android Studio project (see also http://realm.io/docs/java/latest/#installation).

This is a early release of the null support: please take notice of the "alpha1" in the file name :-) As Realm Core still works on supporting null, it hasn't been fully tested, some feature might be missing, and performance might not be optimal.

With this release, string and byte arrays are nullable by default. If you wish to mark a string or byte array not nullable, you can use the annotation @NotNullable. You do not have to specify that a primary key is not-nullable; not nullable is implied by being a primary key.

Types like boolean, integer, float, double and dates are still being worked on by the Realm Core team, and fields of these types are always not-nullable. We still have some work on the JSON and migration parts of Realm.

The internal file format of Realm files are changed. Using this release will irreversibly change your Realm files!

You are invited to try out this early release. Comments, bug reports, suggestions, etc. are welcomed, and please report them in #1098.

DO NOT USE THIS IN PRODUCTION

[jorgehsrocha]

Yes, this can be a big problem while sync my data with Ream. But, i'll waiting for next commits about case.

Thank you, guys. You are awesome.

[AndreasBackx]

Great to be seeing support for null values. Glad I've kept some breathing room to make Realm objects out of my data classes. Guess I can work around the non nullable fields with some default values without having to add extra fields.

Any idea how long it could take for these features to be fully tested and stable for production? Just to keep in the back of my head so I know whether I can start working with null values and have it stable by the time my app releases.

[kneth]

@AndreasBackx I can't give you a time frame. Before including null support in our releases, we wish to support integers too. Currently the Realm Core team is working on that.

[alekseichuk]

+1

[kneth]

I have uploaded a few version of Realm with partial null support (string and binary only). You find it at http://static.realm.io/downloads/java/realm-0.81-null-support-beta1.jar. Please try it out, and give us feedback and bug reports at #1098.

As this release will change your Realm files irreversibly, try this release on test system :-)

[kneth]

@mikeholler @AndreasBackx @jhrocha @mmvicsonvictor @freezy @vvictor10 @kaushikmukunda @falkorichter @jenzz

Any chance you guys have had the chance to try out the beta? We would be grateful for some feedback!

[AndreasBackx]

@kneth I'd love to be able to use Realm again in the future, but #909 is a bit of a pain and it puzzles me why it's not allowed. Thus I have not been able to test out the beta.

[vvictor10]

@kneth Have not been working on Realm for a bit now. Getting back to it now though. Is the null support still in beta? Which version should we be trying out?

[kneth]

@vvictor10 I merged master into the branch yesterday, so I have uploaded a new jar file. You can download here: http://static.realm.io/downloads/java/realm-0.81-null-support-beta2.jar.

[vvictor10]

Thanks @kneth Quick question unrelated to the null issue. I saw this on the 0.81 change log:

• Introduced RealmModules for working with custom schemas in libraries and apps.

Does this mean, we can have Realm models defined across modules in our android project and would the annotation processor take care of combining the ValidationList?

[vvictor10]

I guess that approach still would not work. As The dex complains about the DefaultRealmModule.

com.android.dex.DexException: Multiple dex files define Lio/realm/DefaultRealmModule;

Is there anyway we can create a different DefaultRealmModule and other classes, specific to each module. Perhaps RealmModule etc?

[cmelchior]

Hi @vvictor10 Yes, RealmModules replace the ValidationList file and allow you to use Realm across libraries as well as making different schemas for different Realm files. You can read more here: https://realm.io/docs/java/latest/#schemas

[vvictor10]

@kneth Any idea when this will be merged to master?

[bmunkholm]

We hope to do that very soon now - hopefully within few weeks. There is nothing technical to prevent a release, so you should be able to safely use the beta. We might choose to release background queries first, and follow with null support (still only for strings and binary).

[vvictor10]

@bmunkholm Thanks for that update. Is there any documentation around how the background queries feature is going to work? We have a publish-subscribe pattern in some of our apps, using the otto and a wrapper for async request processing. I want to see how we can use the async queries to work with the existing design or even possibly replace it. Thanks.

[emanuelez]

@vvictor10 It's currently under review and can be followed here: https://github.com/realm/realm-java/pull/1214

[mmvicsonvictor]

@emanuelez @bmunkholm Any dates for this to get into master?

[bmunkholm]

We try to not give dates at all since our first priority is support which is really impossible to predict. That said, I would assume it can still take a couple of weeks from now.

[mmvicsonvictor]

@bmunkholm @emanuelez See that the PR is still open. Any ideas on when this would get into master? Thanks.