Divide the class repository (Reflector) for different SqlSession Config

[alexey-su]

How to divide storage classes (org.apache.ibatis.reflection.Reflector) for each OSGi bundle?

There is a set of components. Each contains a description of the classes of documents to one version.

Document version	Count *.java files	Size (MBytes)	Count *.class files	Size (MBytes)	jar file size (bytes)
1.0	353	~4.9	807	~4.0	876216
4.1	358	~5.0	826	~4.1	896596
4.2	375	~5.2	853	~4.2	896596
4.3	436	~6.7	1072	~5.3	1173399
4.4	483	~8.0	1227	~6.1	1353393
4.5	483	~8.0	1227	~6.2	1360002
4.6	485	~8.0	1230	~6.2	1363847
5.0	508	~8.9	1349	~6.8	1506754
5.1	514	~8.6	1355	~6.8	1506568

The system documents get different versions. Depending on the version, rises OSGi service and OSGi component the structure of the documents required version. One service + One component model.

To process a single document there is no need to prepare everything at once SqlSession Mapper. Loading Mapper interface occurs on demand.

locker.lock();
try {
    // lazy load Reflector
    Reflector.forClass(dataClass);

    // lazy load mapper
    Config config = sqlSession.getConfig();
    if(config.hasMapper(mapperClass) == null)
       config.addMapper(mapperClass);

    return config.getMapper(mapperClass);
} finaly {
    locker.unlock();
}

Periodically rid of OSGi service. ServiceTracker open / close. See complete event service activity. JVisualVM shows purification classes in the HeapSpece.

Reflection class continues to hold classes document structures.

Disabling cache slows down the process of processing multiple documents in different threads.

I think we should add a local cache for Reflection in the base class Config.

[emacarron]

Sorry @alexey-su I did not get that. What is a document?

[alexey-su]

To optimize work with objects, MyBatis uses ObjectWrapper and ObjectWrapperFactory.

To work with the fields of objects used class MetaObject. This is essentially the meta data about the class and the object itself. Meta data about the class - it MetaClass. Cache these objects (MetaClass), stored in a

public class Reflector {
    private static final ConcurrentMap<Class<?>, Reflector> REFLECTOR_MAP = new ConcurrentHashMap<Class<?>, Reflector>();

Information on all classes are stored in one place. The classes themselves are in different OSGi bundle. If OSGi bundle is not used, GC can unload classes of PERMGEN. Storage REFLECTOR_MAP not cleared and GC can not unload classes.

In my situation JVisualVM shows how unloaded all classes of OSGi services PERMGEN. Classes of the big jar files are not preempted.

I propose to move REFLECTOR_MAP in Configuration, where there are other objects

  protected ObjectFactory objectFactory = new DefaultObjectFactory();
  protected ObjectWrapperFactory objectWrapperFactory = new DefaultObjectWrapperFactory();
  // this way
  protected ConcurrentMap<Class<?>, Reflector> reflectorMap = Reflector.createDefaultReflectorMap();

PS.

About the task at hand.

A bit confusing you. In the problem is not one big document. Many documents and all gathered in a large jar file. The structure of the documents is described as a set of nested xsd schemas. Java objects are generated by JAXB.

Different versions of documents enter the system. It is necessary to lay out the information in the database. Create a new layer in the form of Java objects (entity), is not entirely correct. This will create a set of classes, comparable to the amount of processed classes from XSD files.

Therefore, each set Mapper interface interacts with the objects of one version of the document collection. The task is written in OSGi, to solve the problem of large load of classes.

Service is calculated on the treatment of most documents one version. Each received a query contains only one document. Therefore, to prepare Configuration for all types of documents are not necessary. Uses lazy loading XML describing SQL queries (Mapper).

This is just one of the tasks of document processing. Next comes the analysis of historical data.

[emacarron]

Thanks for the explanation @alexey-su

So.. you speak about Documents because you are using JAXB but the problem is with classes not with documents.

So lets go with that point.

Classes are indeed held in an static property of the Reflector class. Why this is property unloaded in a "classic" JVM but not in an OSGI container?

[alexey-su]

I will not tell you his theory overflow PERMGEN. This is a well written to me. "Classloader-Related Memory Issues" in book http://javabook.compuware.com/content/memory/problem-patterns/class-loader-issues.aspx Parts "Classloader Leaks" and "Classloader Cannot Be Garbage-Collected".

Perhaps the authors of the book is somewhat wrong. I find it difficult to test the theory practice. So far I only see PERMGEN. And I have to increase the size of the memory to 512M and above.

[emacarron]

Thanks for the link!

Let me share what I understood.

Classes are objects and need memory. They are garbage collected when their classloader is no longer referenced. This happens when the application gets unloaded. And this may happen:

• When restarting an app in a normal jvm
• When unloading the app bundle in an OSGI container

According to the link the classloader cannot be garbage colledted if any of its loaded classes is:

• In a thread local variable
• In a cache

Let's go to MyBatis.

• You are using it in an OSGI container.
• You have .jar files that holds JAXB classes for a document
• You have several .jar files, one for each document version

And I suppose that the problem you have is that when you unload one jar, it is not really unloaded because MyBatis keeps references to:

• The bean classes (metadata)
• Also to the mappers (MapperRegistry does not remove mappers)

Is that OK?

[alexey-su]

A simple solution is to periodically release the dynamic OSGi Service. "OSGi Alliance. OSGi compendum v5.0" see http://www.osgi.org/Specifications/HomePage Part "112 Declarative Services Specification".

When the service is not needed, the memory is freed. Including ClassLoader. Static object Reflector.REFLECTOR_MAP live very long. You need to unload the OSGi bundle mybatis-3.2.x.jar. MapperObject, MapperClass, MapperRegistry will be deleted when unloaded Configuration

I propose to remove static cache Reflector.REFLECTOR_MAP. Use this cache in the Configuration.

To support the old functionality, you can use the static cache. For OSGi will use the local cache.

[emacarron]

Let's suppose we move that static variable to Configuration.

The Configuration variable will still hold the target classes forever. How will be they unloaded?

[alexey-su]

Why leave static variable? I suggested

protected ObjectFactory objectFactory = new DefaultObjectFactory();
protected ObjectWrapperFactory objectWrapperFactory = new DefaultObjectWrapperFactory();
// this way
protected ConcurrentMap<Class<?>, Reflector> reflectorMap = Reflector.createDefaultReflectorMap();

public void setReflectorMap(ConcurrentMap<Class<?>, Reflector>) { ... }
public ConcurrentMap<Class<?>, Reflector> getReflectorMap() { ... }

You can call it more beautiful.

public interface ReflectorFactory { ... }

And in the class Configuration

protected ReflectorFactory = new DefaultReflectorFactory();

public void setReflectorFactory(ReflectorFactory reflectorFactory) { ... }
public ReflectorFactory getReflectorFactory() { ... }

Parameter is not static. You can do any implementation. The main goal - is to abandon the static objects in the library.

[emacarron]

HI @alexey-su

Removing static variables is a good idea indeed but I still do not understand how that will fix the OSGI problem.

[alexey-su]

The problem in OSGi solve itself.

Book "OSGi Core Revision 5". Chapter 5, part 5.6 Service Factory. When I need a service, I'll take it

MySerivece getService (BundleContext context, SeriveceReference ref) {
  return context.getService (ref);
}

When the service is not needed, I let him go

context.ungetService (ref);

Book "OSGi Compenduim". Chapter 112 Declarative Services Specification. Part 112.2.3 Delayed Component. And part 112.5 Component Life Cycle. Component publish but not initialized.

I use Declarative Services. I am requesting service (getService). Create new object. The service connects to the database service (DataSource). More precisely, to XADataSource wrapper. Apache Aries RecoverableDataSource implementation. Welcom to MyBatis Guice test BaseDB from issuer 29. I release my service (ungetService). Service releases database service. Service object happily falls into the clutches of the Garbage Collector. My service interface is described in another bundle. GC cleans all the objects and classes in OSGi Bundle service implementation.

[emacarron]

Hi @alexey-su can you please send this as a Pull Request?