Null pointer exception indexed attribute has null value.

[GoogleCodeExporter]

What steps will reproduce the problem?
1.Create IndexedCollection for a POJO type and add index on any attribute.
2.Add a object of that POJO with indexed attribute null.
3. Works with empty string.

What is the expected output? What do you see instead?
Should behave as with empty string even with complex type is null there.
Otherwise user have to check and set empty string for null.

What version of the product are you using? On what operating system?
0.9.1-all

Please provide any additional information below.

Original issue reported on code.google.com by SylentPr...@gmail.com on 25 Oct 2012 at 2:47

[GoogleCodeExporter]

This is an enhancement request not a defect.

Original comment by SylentPr...@gmail.com on 25 Oct 2012 at 2:48

[GoogleCodeExporter]

This is a good question. And quite a tricky one. On one hand, handling nulls 
automatically would be helpful in many cases. On the other hand, handling nulls 
automatically can have a big impact on performance.

The easy way to handle potential nulls in your data right now, is to check for 
null in your implementation of attributes. Essentially you need to COALESCE 
nulls to a value of your choosing, as you would do in SQL.

For example, here's a null-safe NAME attribute for a Car:

    public static final Attribute<Car, String> NAME = new SimpleAttribute<Car, String>("name") {
        public String getValue(Car car) {
            String name = car.name;
            return name == null ? "" : name;
        }
    };

You would only do this on fields which you know might contain nulls, for 
performance reasons (see below).

Currently CQEngine follows the convention of the Java Collections Framework, by 
throwing NullPointerException if null is supplied but is not expected. Some 
collections in the JDK support nulls, some do not. For example 
HashMap.put(null, null) will work, but ConcurrentHashMap.put(null, null) will 
throw NullPointerException.

Checking for nulls can have an impact on performance in code which operates in 
tight loops, and so I think the choice of null handling strategy needs to be a 
user's decision.

In CQEngine, if attributes checked for nulls by default, it would have a big 
impact on the performance of queries on attributes for which you haven't added 
any indexes (CQEngine would need to read attributes and check for null values 
in every object). For attributes on which you have added indexes, it wouldn't 
make such a difference however (i.e. only when objects are added to the index 
for the first time). Also, when indexing primitive fields, of course there can 
never be an issue with nulls so it wouldn't make sense to check for nulls on 
attributes on those fields.

Personally I don't like the extra lines of code required to implement the 
null-safe attribute above. I will think about it and maybe look into adding an 
additional type of attribute that would simplify it. For example there could be 
a SimpleNullSafeAttribute or something. In the meantime the example above will 
work for fields which might contain nulls. 

Thanks for the report :)

Original comment by ni...@npgall.com on 25 Oct 2012 at 11:04

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

Original comment by ni...@npgall.com on 29 Oct 2012 at 1:50

• Changed state: Started

[GoogleCodeExporter]

Fixed in 1.0.0.

CQEngine now has more explicit support for attributes to indicate that "this 
attribute does not apply to this object". CQEngine 1.0.0 supports new 
"Nullable" attributes:

 - http://cqengine.googlecode.com/svn/cqengine/javadoc/apidocs/com/googlecode/cqengine/attribute/SimpleNullableAttribute.html
 - http://cqengine.googlecode.com/svn/cqengine/javadoc/apidocs/com/googlecode/cqengine/attribute/MultiValueNullableAttribute.html

In CQEngine 0.9.1, this was actually possible using MultiValueAttribute, where 
if the attribute did not apply to a given object, MultiValueAttribute could 
return an empty list (i.e. the empty set) as the value for that object. 
CQEngine would then not add the object to any indexes built on that attribute. 
So the engine supported this behaviour, but it wasn't very clear how to use it 
before.

Here's a null-safe NAME attribute for a car based on the new 
SimpleNullableAttribute in CQEngine 1.0.0:

    public static final Attribute<Car, String> NAME = new SimpleNullableAttribute<Car, String>("name") {
        public String getValue(Car car) { return car.name; }
    };

So the code is simple again. 1.0.0 is available in the Downloads tab, and is 
queued for sync with Maven central, it should appear there in a few hours.

Original comment by ni...@npgall.com on 29 Oct 2012 at 9:59

• Changed state: Fixed

[GoogleCodeExporter]

This still doesn't work for me, using CQEngine:
<dependency>
    <groupId>com.googlecode.cqengine</groupId>
    <artifactId>cqengine</artifactId>
    <version>1.2.6</version>
</dependency>

Property:
public static final Attribute<ConsumerLoad, String> ADDRESS2 = new 
SimpleNullableAttribute<ConsumerLoad, String>("ADDRESS2") {
    public String getValue(ConsumerLoad consumerload) {
        String address2 = consumerload.address2;
        return address2 == null ? "" : address2;
    }
};

Error and dumps:
and(equal(ConsumerLoad.ADDRESS1, 5651 N ARLINGTON BLVD), 
equal(ConsumerLoad.ADDRESS2, null))
-Update failed, load id 200061 : null
java.lang.NullPointerException
    at com.googlecode.cqengine.query.simple.Equal.calcHashCode(Equal.java:79)
    at com.googlecode.cqengine.query.simple.SimpleQuery.hashCode(SimpleQuery.java:102)
    at java.util.AbstractList.hashCode(Unknown Source)
    at com.googlecode.cqengine.query.logical.And.calcHashCode(And.java:76)
    at com.googlecode.cqengine.query.logical.LogicalQuery.hashCode(LogicalQuery.java:110)
    at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
    at com.googlecode.cqengine.engine.impl.QueryEngineImpl.retrieveRecursive(QueryEngineImpl.java:301)
    at com.googlecode.cqengine.engine.impl.QueryEngineImpl.retrieve(QueryEngineImpl.java:241)
    at com.googlecode.cqengine.collection.impl.ConcurrentIndexedCollection.retrieve(ConcurrentIndexedCollection.java:79)
    at com.palmcoastdata.cdb.update.ConsumerUpdate.update(ConsumerUpdate.java:166)

Original comment by clint...@gmail.com on 5 Feb 2014 at 3:36

[GoogleCodeExporter]

public static final Attribute<ConsumerLoad, String> ADDRESS2 = new 
SimpleNullableAttribute<ConsumerLoad, String>("ADDRESS2") {
    public String getValue(ConsumerLoad consumerload) { return consumerload.address2; }
};

..Also does not work.

Original comment by clint...@gmail.com on 5 Feb 2014 at 3:50

[GoogleCodeExporter]

Hi clint317,

You need to rephrase your query using 'has' instead: 
http://cqengine.googlecode.com/svn/cqengine/javadoc/apidocs/com/googlecode/cqeng
ine/query/simple/Has.html

Replace: equal(ConsumerLoad.ADDRESS2, null)
With:    not(has(ConsumerLoad.ADDRESS2))

..and it should work.

'has' in a CQEngine query is equivalent to 'IS NOT NULL' in an SQL query. It 
means "does the object have a value for this attribute".

So 'not(has)' is the same as SQL 'IS NULL'.

Null values won't be stored in CQEngine indexes by default, so see the JavaDoc 
for 'has' above, for guidance on accelerating these queries. That is, you can 
add a StandingQueryIndex to accelerate it (if you wish).

Original comment by ni...@npgall.com on 5 Feb 2014 at 5:21

[GoogleCodeExporter]

I don't want to assert that it has a value or doesn't have a value. I want to 
assert that the property's value is equal to a given value.

select * where ConsumerLoad.ADDRESS2 = 'value';
--'value' may be null or defined
select * where ConsumerLoad.ADDRESS2 is null;

equal(ConsumerLoad.ADDRESS2, '123 Main St') - works
equal(ConsumerLoad.ADDRESS2, null) does not work

has(ConsumerLoad.ADDRESS2) - only test if it HAS a value and not if the value 
matches something

Original comment by clint...@gmail.com on 6 Feb 2014 at 1:36

[GoogleCodeExporter]

Null is not a value. Null is the absence of a value.

Your SQL statement..

select * where ConsumerLoad.ADDRESS2 = 'value';

..will not work as you expect if you substitute null for 'value'. It will not 
match any rows. In SQL you should never see or use "WHERE column = null" 
because it evaluates to false. See 
http://stackoverflow.com/questions/3777230/is-there-any-difference-between-is-nu
ll-and-null

SQL has a special syntax for testing if a column does not have a value: "IS 
NULL". CQEngine has a special syntax also: has, and not(has).

So you need to use 'has' if you want to test for null in a regular field (or 
test for no values in a multi-valued field), or use 'equal' if you want to test 
equality of non-null values. It is fairly consistent with the design of SQL.

Original comment by ni...@npgall.com on 6 Feb 2014 at 2:30

[GoogleCodeExporter]

Ok, so what your saying is that 'equal()' is EXACTLY like SQL limitations on 
null. I know how to use SQL and I'm aware you can't use x=null in a select. I 
was under the impression that SimpleNullableAttribute meant I could use 
'equal()' to test a value if the value might be null.

In other words there is no way to use CQEngine to do something like:
equal(ConsumerLoad.ADDRESS2, null);
Unless you want to use 'has()', or 'not(has())' to simply test for nulls and 
only for nulls.

And if you use:
equal(ConsumerLoad.ADDRESS2, variable);
Equal() will throw an exception if variable is null even if the property is 
SimpleNullableAttribute.

I can't even use:
in(ConsumerLoad.ADDRESS1, null, address.getAddress1());
If I make the property a MultiValueNullableAttribute.

In fact the ONLY way (least possible lines of code) to test a property against 
a value that might or not be null is basically converting nulls to empty 
strings:
public static final Attribute<ConsumerLoad, String> ADDRESS2 = new 
SimpleNullableAttribute<ConsumerLoad, String>("ADDRESS2") {
    public String getValue(ConsumerLoad consumerload) { return consumerload.address2==null?"":consumerload.address2; }
};
And:
equal(ConsumerLoad.ADDRESS2, 
address.getAddress2()==null?"":address.getAddress2());

Ok, got it. Thank you.

Original comment by clint...@gmail.com on 6 Feb 2014 at 7:30

[GoogleCodeExporter]

May I remind you that CQEngine is free of charge! So if you want things 
improved, just supply a patch. Politely please! Take a look at the QueryFactory 
class for examples of how queries are created.

If you want a combination of equal() and has(), just add a method like this to 
your application:

    public static <O, A> Query<O> nullSafeEqual(Attribute<O, A> attribute, A attributeValue) {
        return attributeValue == null ? not(has(attribute)) : equal(attribute, attributeValue);
    }

Original comment by ni...@npgall.com on 6 Feb 2014 at 9:44