Benchmarking more cases

roookeee commented 5 years ago

The OrderFactory currently only returns one case which could be seen as the most "happiest" case: nothing is null which introduces several issues:

all the Mappers don't actually have to be implemented / defined in a null-safe manner. It's uncertain right now whether every implemented mapper is actually null safe (the tests don't cover that either). Each mappers null-safe declaration could also be way more verbose than it is right know which would skew the first impression how succinct / readable each mapping framework actually is in a nullable scenario
the performance characteristics of each mapper could be drastically different for "unhappy" cases e.g. every field that should be mapped is null

I would propose the OrderManager to return a List<Order> which contains a happy, unhappy and middle case which is iterated for each mapper. Each Mapper will suffer in its final score if it handles one of those cases poorly. Maybe use an Order[] instead of a List<Order> to avoid skewing the results by having to instantiate an Iterator<Order> in each benchmark call. An extended for-loop on an array desugars into an indexed based loop that won't impact the results as much

roookeee commented 5 years ago

I would volunteer for a PR if this issue is accepted

arey commented 5 years ago

The 2 source and target data structures are similar. There is no upper case conversion or lastname + firstname to name conversion. Thus nullable fields should not be a problem. Nullable relationship or empty collections could be. Before going further, I would like some precisions. Just a few samples could be fine.

roookeee commented 5 years ago

Looking at the manual implementation everything is indeed null checked but the input values for the benchmark are never null. One could conclude that 1) the manual implementation does unnecessary null checks and may lose some performance compared to the mapping frameworks or 2) that the input of the benchmark does not reflect the nullability of some fields.

What I am proposing: use Order values in the benchmark that actually have null values. As I see it most if not all mappers included in this repository do a null check implicitly (without further configuration) BUT some mappers might need additional configuration to add a null check (see my just added datus implementation). I could have omitted the null checks as I know that the current benchmark values are never null which would improve datus performance - cheating yes, but there is no unit test that shows that all mappers can handle null fields of the input value and the benchmark doesn't reflect null values in the fields too. I would propose the following unit-test and benchmark cases:

//case 1: everything is present
Order order1 = new Order();
Customer customer = new Customer();
order1.setCustomer(customer);
customer.setName("Joe Smith");
Address billingAddress = new Address();
customer.setBillingAddress(billingAddress);
billingAddress.setStreet("1234 Market Street");
billingAddress.setCity("San Fran");
Address shippingAddress = new Address();
customer.setShippingAddress(shippingAddress);
shippingAddress.setStreet("1234 West Townsend");
shippingAddress.setCity("Boston");
List<Product> products = new ArrayList<Product>(2);
order1.setProducts(products);
products.add(new Product("socks"));
products.add(new Product("shoes"));

//case 2: everything is null
Order order2 = new Order();

//case 3: some fields are null (both addresses)
Order order3 = new Order();
Customer customer3 = new Customer();
order3.setCustomer(customer3);
customer3.setName("Joe Smith");
customer3.setShippingAddress(shippingAddress);
shippingAddress.setStreet("1234 West Townsend");
shippingAddress.setCity("Boston");
List<Product> products3 = new ArrayList<Product>(2);
order3.setProducts(products3);
products3.add(new Product("socks"));
products3.add(new Product("shoes"));

arey commented 5 years ago

I've done a test with the partial Order (case 3). Only the Jmapper test fails and requires some additional configuration :

com.googlecode.jmapper.exceptions.JMapperException: com.googlecode.jmapper.exceptions.NestedBeanNullException: the field street is null, error occured in mapping phase between the fields: OrderDTO.shippingStreetAddress and Order.customer

    at com.googlecode.jmapper.config.JmapperLog.error(JmapperLog.java:46)
    at com.googlecode.jmapper.JMapper.getDestination(JMapper.java:100)
    at com.javaetmoi.benchmark.mapping.mapper.jmapper.JMapperMapper.map(JMapperMapper.java:35)
    at com.javaetmoi.benchmark.mapping.mapper.dozer.AbstractMapperTest.map(AbstractMapperTest.java:16)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runners.Suite.runChild(Suite.java:128)
    at org.junit.runners.Suite.runChild(Suite.java:27)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
    at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: com.googlecode.jmapper.exceptions.NestedBeanNullException: the field street is null, error occured in mapping phase between the fields: OrderDTO.shippingStreetAddress and Order.customer
    at com.googlecode.jmapper.config.Error.nestedBeanNull(Error.java:127)
    at comjavaetmoibenchmarkmappingmodeldtoOrderDTOcomjavaetmoibenchmarkmappingmodelentityOrder741697001.nullVSouAllAll(comjavaetmoibenchmarkmappingmodeldtoOrderDTOcomjavaetmoibenchmarkmappingmodelentityOrder741697001.java)
    at com.googlecode.jmapper.JMapper.getDestination(JMapper.java:98)
    ... 32 more

I'm not sure to add those kind of test. I'll prefer to enhance the model with more advanced features: enum mapping, transcoding, String to numeric conversion ...

roookeee commented 5 years ago

Your call :) Maybe just make a comment in the readme that you expect or don't expect new mappers to handle null, a unit test would do too. Thanks for your time. If you change your mind give me a heads up, will do a PR then

arey commented 5 years ago

Thank you. I've added some unit tests and complete the readme.md.

arey / java-object-mapper-benchmark

Benchmarking more cases #17