Image rendering throws exception with java.lang.OutOfMemoryError

[jithumurugan]

Hello,

I have uploaded the below given dataset into my container based omero server and could visualize and properly work with it via omero-web client https://downloads.openmicroscopy.org/images/Leica-SCN/openslide/Leica-1/Leica-1.scn

The problem occurs when I try to download this data or render it using omero-py based BlitzGateway functions (gateway::renderImage), I get the following exception:

OutOfMemory

``` _WARNING:omero.gateway:InternalException on to renderCompressed((object #0 (::omero::romio::PlaneDef) { slice = 0 x = 0 y = 0 z = 0 t = 0 region = stride = 0 }, ), {}) Traceback (most recent call last): File "C:\Python38\lib\site-packages\omero\gateway\__init__.py", line 4793, in __call__ return self.f(*args, **kwargs) File "C:\Python38\lib\site-packages\omero_api_RenderingEngine_ice.py", line 913, in renderCompressed return _M_omero.api.RenderingEngine._op_renderCompressed.invoke(self, ((_def, ), _ctx)) omero.InternalException: exception ::omero::InternalException { serverStackTrace = ome.conditions.InternalException: Wrapped Exception: (java.lang.OutOfMemoryError): Java heap space at ome.io.bioformats.BfPixelBuffer.getPlane(BfPixelBuffer.java:208) at omeis.providers.re.data.PlaneFactory.createPlane(PlaneFactory.java:208) at omeis.providers.re.HSBStrategy.getWavelengthData(HSBStrategy.java:100) at omeis.providers.re.HSBStrategy.makeRenderingTasks(HSBStrategy.java:231) at omeis.providers.re.HSBStrategy.render(HSBStrategy.java:328) at omeis.providers.re.HSBStrategy.renderAsPackedInt(HSBStrategy.java:292) at omeis.providers.re.Renderer.renderAsPackedInt(Renderer.java:558) at ome.services.RenderingBean.renderAsPackedInt(RenderingBean.java:512) at ome.services.RenderingBean.renderCompressed(RenderingBean.java:542) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy106.renderCompressed(Unknown Source) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) serverExceptionClass = ome.conditions.InternalException message = Wrapped Exception: (java.lang.OutOfMemoryError): Java heap space }_ I can see the same error been thrown inside the server side /opt/omero/server/OMERO.server/var/log/Blitz-0.log also .................. _2021-03-29 13:22:03,952 INFO [ ome.services.util.ServiceHandler] (l.Server-3) Meth: interface omeis.providers.re.RenderingEngine.renderCompressed 2021-03-29 13:22:03,952 INFO [ ome.services.util.ServiceHandler] (l.Server-3) Args: [Type: XY, z=0, t=0, renderShapes=false, shapeIds=[]] 2021-03-29 13:22:03,953 INFO [ omeis.providers.re.Renderer] (l.Server-3) Using: 'omeis.providers.re.HSBStrategy' rendering strategy. 2021-03-29 13:24:00,001 INFO [ ome.services.blitz.fire.SessionManagerI] (1-thread-4) Performing requestHeartbeats 2021-03-29 13:26:34,770 INFO [ org.perf4j.TimingLogger] (l.Server-3) start[1617024123952] time[270818] tag[omero.call.exception] 2021-03-29 13:26:34,774 ERROR [ ome.services.util.ServiceHandler] (l.Server-3) java.lang.Error: Wrapped Exception: (java.lang.OutOfMemoryError): Java heap space java.lang.OutOfMemoryError: Java heap space at ome.io.bioformats.BfPixelBuffer.getPlane(BfPixelBuffer.java:208) ~[omero-romio.jar:5.6.2] at omeis.providers.re.data.PlaneFactory.createPlane(PlaneFactory.java:208) ~[omero-renderer.jar:5.5.7] at omeis.providers.re.HSBStrategy.getWavelengthData(HSBStrategy.java:100) ~[omero-renderer.jar:5.5.7] at omeis.providers.re.HSBStrategy.makeRenderingTasks(HSBStrategy.java:231) ~[omero-renderer.jar:5.5.7] at omeis.providers.re.HSBStrategy.render(HSBStrategy.java:328) ~[omero-renderer.jar:5.5.7] at omeis.providers.re.HSBStrategy.renderAsPackedInt(HSBStrategy.java:292) ~[omero-renderer.jar:5.5.7] at omeis.providers.re.Renderer.renderAsPackedInt(Renderer.java:558) ~[omero-renderer.jar:5.5.7] at ome.services.RenderingBean.renderAsPackedInt(RenderingBean.java:512) ~[omero-server.jar:5.6.1] at ome.services.RenderingBean.renderCompressed(RenderingBean.java:542) ~[omero-server.jar:5.6.1] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na] at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na] at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop.jar:4.3.14.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop.jar:4.3.14.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop.jar:4.3.14.RELEASE] at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) ~[omero-server.jar:5.6.1] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop.jar:4.3.14.RELEASE] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop.jar:4.3.14.RELEASE] at com.sun.proxy.$Proxy106.renderCompressed(Unknown Source) ~[na:na] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na] at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na] at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop.jar:4.3.14.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop.jar:4.3.14.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop.jar:4.3.14.RELEASE] at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93) ~[omero-server.jar:5.6.1] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop.jar:4.3.14.RELEASE] at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43) ~[omero-blitz.jar:5.5.8] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop.jar:4.3.14.RELEASE] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop.jar:4.3.14.RELEASE] 2021-03-29 13:26:34,775 WARN [ ome.services.util.ServiceHandler] (l.Server-3) Unknown exception thrown. java.lang.OutOfMemoryError: Java heap space at ome.io.bioformats.BfPixelBuffer.getPlane(BfPixelBuffer.java:208) ~[omero-romio.jar:5.6.2] at omeis.providers.re.data.PlaneFactory.createPlane(PlaneFactory.java:208) ~[omero-renderer.jar:5.5.7] at omeis.providers.re.HSBStrategy.getWavelengthData(HSBStrategy.java:100) ~[omero-renderer.jar:5.5.7] at omeis.providers.re.HSBStrategy.makeRenderingTasks(HSBStrategy.java:231) ~[omero-renderer.jar:5.5.7] at omeis.providers.re.HSBStrategy.render(HSBStrategy.java:328) ~[omero-renderer.jar:5.5.7] at omeis.providers.re.HSBStrategy.renderAsPackedInt(HSBStrategy.java:292) ~[omero-renderer.jar:5.5.7] at omeis.providers.re.Renderer.renderAsPackedInt(Renderer.java:558) ~[omero-renderer.jar:5.5.7] at ome.services.RenderingBean.renderAsPackedInt(RenderingBean.java:512) ~[omero-server.jar:5.6.1] at ome.services.RenderingBean.renderCompressed(RenderingBean.java:542) ~[omero-server.jar:5.6.1] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na] at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na] at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop.jar:4.3.14.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop.jar:4.3.14.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop.jar:4.3.14.RELEASE] at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) ~[omero-server.jar:5.6.1] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop.jar:4.3.14.RELEASE] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop.jar:4.3.14.RELEASE] at com.sun.proxy.$Proxy106.renderCompressed(Unknown Source) ~[na:na] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na] at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na] at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop.jar:4.3.14.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop.jar:4.3.14.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop.jar:4.3.14.RELEASE] at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93) ~[omero-server.jar:5.6.1] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop.jar:4.3.14.RELEASE] at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43) ~[omero-blitz.jar:5.5.8] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop.jar:4.3.14.RELEASE] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop.jar:4.3.14.RELEASE] 2021-03-29 13:26:34,776 INFO [ ome.services.util.ServiceHandler] (l.Server-3) Excp: java.lang.OutOfMemoryError: Java heap space 2021-03-29 13:26:34,776 ERROR [ ome.services.util.ServiceHandler] (l.Server-3) Method interface omeis.providers.re.RenderingEngine.renderCompressed invocation took 270818 2021-03-29 13:27:00,001 INFO [ ome.services.blitz.fire.SessionManagerI] (1-thread-5) Performing requestHeartbeats_ ```

I think, I have to adjust the heapsize of server side JVM, which all config has to be adapted? Would it be better to have enough size defined for the docker images already?

Also could you let me know which is the python gateway function to be used to download the 'Leica-1.scn' image locally to my directory from omero-server?

[joshmoore]

I think, I have to adjust the heapsize of server side JVM ...

This does look to be the problem. However, OMERO tries to auto-detect the appropriate memory size from that which is available. (It doesn't even know that it's in a container) How much memory does your host server have?

See https://docs.openmicroscopy.org/omero/5.6/sysadmins/server-performance.html#memory-configuration for more information.

Also could you let me know which is the python gateway function ...

@jithumurugan, can we move general questions to http://image.sc/tags/omero please, and only deal with true bugs here?

Thanks! ~Josh

[jithumurugan]

Hello Josh, Thanks for the response, I had to change the JVM setting via omero configuration params in docker.compose.yml, the default setting CONFIG_omero_jvmcfg_percent 22 was not enough sufficient. I have 125G RAM in my host with 14G free at the time. I had to increase it to 50% to get rid of the out of memory error, see the configs below:

......................................... omeroserver: image: "openmicroscopy/omero-server:5.6" environment: CONFIG_omero_db_host: database CONFIG_omero_db_user: omero CONFIG_omero_db_pass: omero CONFIG_omero_db_name: omero CONFIG_omero_jvmcfg_percent: 50 CONFIG_Ice_MessageSizeMax: 266000000 ROOTPASS: omero

............................................

But after the above changes, I do not get any OME instead I am getting different sort of error:

_WARNING:omero.gateway:MemoryLimitException on <class 'omero.gateway.OmeroGatewaySafeCallWrapper'> to <8fc3481d-92b4-409f-b7d1-f101f897486comero.api.RenderingEngine> renderCompressed((object #0 (::omero::romio::PlaneDef) { slice = 0 x = 0 y = 0 z = 0 t = 0 region = stride = 0 }, <ServiceOptsDict: {'omero.client.uuid': '16c11a5b-861d-416d-867c-a7c437aa92fc', 'omero.event': 'Internal', 'omero.session.uuid': 'c859ef56-c9fa-4025-9313-2a683708789c'}>), {}) Traceback (most recent call last): File "C:\Python38\lib\site-packages\omero\gateway__init.py", line 4793, in call__ return self.f(*args, **kwargs) File "C:\Python38\lib\site-packages\omero_api_RenderingEngine_ice.py", line 913, in renderCompressed return _M_omero.api.RenderingEngine._op_renderCompressed.invoke(self, ((_def, ), ctx)) Ice.MemoryLimitException: exception ::Ice::MemoryLimitException { reason = requested 328860651 bytes, maximum allowed is 256000000 bytes (see Ice.MessageSizeMax) }

I tried to give as maximum as possible, e.g. something like CONFIG_Ice_MessageSizeMax: 5000000000. Strangely I see that these values are not taken by omero (see error below), I see some these setting is ignored and some default value is been taken. Any idea, why this is happening? Am I doing something wrong here?

I know these are more configuration issues, but in my opinion we have to define a proper set of working config parameters in docker-compose.yml file, so that it would be easy for the users further? I am very sure that the cases like https://downloads.openmicroscopy.org/images/Leica-SCN/openslide/Leica-1/Leica-1.scn (36832 * 38432 pixels) will be very often be uploaded by users and have to be retrieved easily without such errors. so keeping this in mind, should not we define the needed config parameters ahead instead of users debugging and fixing them after deploying the containers?

[jithumurugan]

Sorry Josh, I am new here and getting used to the different possibilities, I sure will post the general queries in the community further, http://image.sc/tags/omero Thanks for letting me know about this,

Regards Jithu

[joshmoore]

@jithumurugan, no apologies need. There are just far for people watching image.sc in order to help you out. :+1:

Regarding your MessageSizeMax, I don't think this is a configuration issue. Eventually image sizes become large enough that it's no longer possible to load all data in one call. (We have one image that's 980K x 320K!) You will need to move to a tile-by-tile loading of your data. Let's discuss on image.sc. (If there is a bug in the Python code, then that will be an issue or PR against https://github.com/ome/omero-py/blob/3f68db664839a8f383baff6759d5ef0569863690/src/omero/gateway/__init__.py#L9588)

All the best, ~Josh

[jithumurugan]

Thanks again Josh, I think the tile wise approach should be used, I saw one conversation already in the community, hope this is the solution what I am looking for: https://forum.image.sc/t/imagej-plugin-and-big-2d-image-ice-connectionlostexception/36043

Yes you are right too, may be the render_image should somehow intelligently limit the possible image dimensions and prompt the user to go for tiled rendering when the image dimensions are huge (y)