search

BaderLab / EnrichmentMapApp

The EnrichmentMap Cytoscape App allows you to visualize the results of gene-set enrichment as a network.
http://apps.cytoscape.org/apps/enrichmentmap
GNU Lesser General Public License v2.1
31 stars 12 forks source link

EM session not restoring after save #482

Closed risserlin closed 2 years ago

risserlin commented 2 years ago

Issue is occuring during save though.
Large EM with over 300 datasets. Save the session. close cytoscape and reload session. Network is there but none of the EM informations is there. Cytoscape 3.9.1 EM - 3.3.3

Error from log file: 2022-03-24 12:38:24,781 [Task-Thread-10-Factory-0x3fb24cb7] ERROR org.cytoscape.application.userlog - Unexpected exception while handling listener: org.cytoscape.session.events.SessionAboutToBeSavedListener java.lang.reflect.InvocationTargetException: null at jdk.internal.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.cytoscape.event.internal.CyListenerAdapter.fireEvent(CyListenerAdapter.java:120) [!/:?] at org.cytoscape.event.internal.CyEventHelperImpl.fireEvent(CyEventHelperImpl.java:81) [!/:?] at org.cytoscape.session.internal.CySessionManagerImpl.getCurrentSession(CySessionManagerImpl.java:115) [!/:?] at org.cytoscape.task.internal.session.SaveSessionAsTask.run(SaveSessionAsTask.java:75) [!/:?] at org.cytoscape.work.internal.task.JDialogTaskManager$TaskRunnable.innerRun(JDialogTaskManager.java:321) [!/:?] at org.cytoscape.work.internal.task.JDialogTaskManager$TaskRunnable.run(JDialogTaskManager.java:352) [!/:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: java.lang.OutOfMemoryError: UTF16 String size is 1207959550, should be less than 1073741823 at java.lang.StringUTF16.newBytesFor(StringUTF16.java:49) ~[?:?] at java.lang.AbstractStringBuilder.inflate(AbstractStringBuilder.java:228) ~[?:?] at java.lang.AbstractStringBuilder.appendChars(AbstractStringBuilder.java:1701) ~[?:?] at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:634) ~[?:?] at java.lang.StringBuffer.append(StringBuffer.java:392) ~[?:?] at java.io.StringWriter.write(StringWriter.java:122) ~[?:?] at com.google.gson.stream.JsonWriter.string(JsonWriter.java:591) ~[?:?] at com.google.gson.stream.JsonWriter.writeDeferredName(JsonWriter.java:402) ~[?:?] at com.google.gson.stream.JsonWriter.beginObject(JsonWriter.java:307) ~[?:?] at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:238) ~[?:?] at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:69) ~[?:?] at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:208) ~[?:?] at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:145) ~[?:?] at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:69) ~[?:?] at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:125) ~[?:?] at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:243) ~[?:?] at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:69) ~[?:?] at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:208) ~[?:?] at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:145) ~[?:?] at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:69) ~[?:?] at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:125) ~[?:?] at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:243) ~[?:?] at com.google.gson.Gson.toJson(Gson.java:669) ~[?:?] at com.google.gson.Gson.toJson(Gson.java:648) ~[?:?] at com.google.gson.Gson.toJson(Gson.java:603) ~[?:?] at com.google.gson.Gson.toJson(Gson.java:583) ~[?:?] at org.baderlab.csplugins.enrichmentmap.model.io.ModelSerializer.serialize(ModelSerializer.java:60) ~[?:?] at org.baderlab.csplugins.enrichmentmap.model.io.ModelSerializer.serialize(ModelSerializer.java:44) ~[?:?] at org.baderlab.csplugins.enrichmentmap.model.io.SessionModelIO.lambda$saveModel$0(SessionModelIO.java:91) ~[?:?] at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684) ~[?:?] at org.baderlab.csplugins.enrichmentmap.model.io.SessionModelIO.saveModel(SessionModelIO.java:88) ~[?:?] at org.baderlab.csplugins.enrichmentmap.model.io.SessionListener.save(SessionListener.java:64) ~[?:?] at org.baderlab.csplugins.enrichmentmap.model.io.SessionListener.handleEvent(SessionListener.java:47) ~[?:?] ... 14 more

mikekucera commented 2 years ago

Caused by: java.lang.OutOfMemoryError: UTF16 String size is 1207959550, should be less than 1073741823

EM stores its data in the session in a hidden table. It looks like the datasets combined have too much data to fit into a cytoscape table cell. I can fix this but it would take a while and require a new session format. Is there any way you can work around this for now?

Mike.

On Thu, Mar 24, 2022 at 12:44 PM Ruth Isserlin @.***> wrote:

Issue is occuring during save though. Large EM with over 300 datasets. Save the session. close cytoscape and reload session. Network is there but none of the EM informations is there. Cytoscape 3.9.1 EM - 3.3.3

Error from log file: 2022-03-24 12:38:24,781 [Task-Thread-10-Factory-0x3fb24cb7] ERROR org.cytoscape.application.userlog - Unexpected exception while handling listener: org.cytoscape.session.events.SessionAboutToBeSavedListener java.lang.reflect.InvocationTargetException: null at jdk.internal.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.cytoscape.event.internal.CyListenerAdapter.fireEvent(CyListenerAdapter.java:120) [!/:?] at org.cytoscape.event.internal.CyEventHelperImpl.fireEvent(CyEventHelperImpl.java:81) [!/:?] at org.cytoscape.session.internal.CySessionManagerImpl.getCurrentSession(CySessionManagerImpl.java:115) [!/:?] at org.cytoscape.task.internal.session.SaveSessionAsTask.run(SaveSessionAsTask.java:75) [!/:?] at org.cytoscape.work.internal.task.JDialogTaskManager$TaskRunnable.innerRun(JDialogTaskManager.java:321) [!/:?] at org.cytoscape.work.internal.task.JDialogTaskManager$TaskRunnable.run(JDialogTaskManager.java:352) [!/:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: java.lang.OutOfMemoryError: UTF16 String size is 1207959550, should be less than 1073741823 at java.lang.StringUTF16.newBytesFor(StringUTF16.java:49) ~[?:?] at java.lang.AbstractStringBuilder.inflate(AbstractStringBuilder.java:228) ~[?:?] at java.lang.AbstractStringBuilder.appendChars(AbstractStringBuilder.java:1701) ~[?:?] at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:634) ~[?:?] at java.lang.StringBuffer.append(StringBuffer.java:392) ~[?:?] at java.io.StringWriter.write(StringWriter.java:122) ~[?:?] at com.google.gson.stream.JsonWriter.string(JsonWriter.java:591) ~[?:?] at com.google.gson.stream.JsonWriter.writeDeferredName(JsonWriter.java:402) ~[?:?] at com.google.gson.stream.JsonWriter.beginObject(JsonWriter.java:307) ~[?:?] at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:238) ~[?:?] at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:69) ~[?:?] at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:208) ~[?:?] at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:145) ~[?:?] at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:69) ~[?:?] at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:125) ~[?:?] at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:243) ~[?:?] at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:69) ~[?:?] at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:208) ~[?:?] at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:145) ~[?:?] at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:69) ~[?:?] at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:125) ~[?:?] at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:243) ~[?:?] at com.google.gson.Gson.toJson(Gson.java:669) ~[?:?] at com.google.gson.Gson.toJson(Gson.java:648) ~[?:?] at com.google.gson.Gson.toJson(Gson.java:603) ~[?:?] at com.google.gson.Gson.toJson(Gson.java:583) ~[?:?] at org.baderlab.csplugins.enrichmentmap.model.io.ModelSerializer.serialize(ModelSerializer.java:60) ~[?:?] at org.baderlab.csplugins.enrichmentmap.model.io.ModelSerializer.serialize(ModelSerializer.java:44) ~[?:?] at org.baderlab.csplugins.enrichmentmap.model.io.SessionModelIO.lambda$saveModel$0(SessionModelIO.java:91) ~[?:?] at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684) ~[?:?] at org.baderlab.csplugins.enrichmentmap.model.io.SessionModelIO.saveModel(SessionModelIO.java:88) ~[?:?] at org.baderlab.csplugins.enrichmentmap.model.io.SessionListener.save(SessionListener.java:64) ~[?:?] at org.baderlab.csplugins.enrichmentmap.model.io.SessionListener.handleEvent(SessionListener.java:47) ~[?:?] ... 14 more

— Reply to this email directly, view it on GitHub https://github.com/BaderLab/EnrichmentMapApp/issues/482, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABI2JAGVUSEF76VXZMGRRX3VBSLYFANCNFSM5RRVO6KA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

risserlin commented 2 years ago

How would I go about working around this? Cut the data into smaller pieces? I started saving just the starting network (no post analysis edges) and it won't even do that. 301 datasets, 827 nodes and >12000 edges. Is it the datasets that is causing the problem?

mikekucera commented 2 years ago

It’s the number of datasets that’s too much, the size of the network isn’t a problem. Do you also have expression data?

I’m any case I think this is a big that will have to be fixed in EM.

Can you send me your data files and I can try to reproduce the problem?

On Thu, Mar 24, 2022 at 2:07 PM Ruth Isserlin @.***> wrote:

How would I go about working around this? Cut the data into smaller pieces? I started saving just the starting network (no post analysis edges) and it won't even do that. 301 datasets, 827 nodes and >12000 edges. Is it the datasets that is causing the problem?

— Reply to this email directly, view it on GitHub https://github.com/BaderLab/EnrichmentMapApp/issues/482#issuecomment-1077900898, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABI2JAHSV3NPWCKDR7XUQVTVBSVPZANCNFSM5RRVO6KA . You are receiving this because you commented.Message ID: @.***>

risserlin commented 2 years ago

I have an expression file and a rank file associated with each dataset. I could try and use one expression file instead of the individual dataset files if that will help. I'll send you the data.

mikekucera commented 2 years ago

Yes I think using one expression file would probably help. The expression data is the largest part of the EM data.

On Thu, Mar 24, 2022 at 2:17 PM Ruth Isserlin @.***> wrote:

I have an expression file and a rank file associated with each dataset. I could try and use one expression file instead of the individual dataset files if that will help. I'll send you the data.

— Reply to this email directly, view it on GitHub https://github.com/BaderLab/EnrichmentMapApp/issues/482#issuecomment-1077912151, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABI2JAGEULK7ZX5CV5AUQRTVBSWSFANCNFSM5RRVO6KA . You are receiving this because you commented.Message ID: @.***>

risserlin commented 2 years ago

I'll try that.

risserlin commented 2 years ago

Unfortunately using one common expression file or using no expression file did not fix the issue. I sent you a link to the data.

mikekucera commented 2 years ago

The JSON data for this network is indeed huge (1.9GB). Normally the EM data model is serialized to a JSON String and stored in a CyTable. The problem in this case is that the data is so large it won't fit into a String object, and it throws an OutOfMemoryError during JSON serialization.

I'm working on a new strategy for saving the data that will handle the OutOfMemoryError. The strategy is to catch the OutOfMemoryError if it happens and then save the JSON to a separate file inside the session instead. The caveat is that this will require a new version of EM, and saving the data in the new format will not be backwards compatible with previous versions of EM. But it will only effect sessions that trigger an OutOFMemoryError when saving like this one.

risserlin commented 2 years ago

Is there a hack around this in the mean time? Where is the data running out of room? If I reduce the number of datasets or split them up into different networks would that help?

risserlin commented 2 years ago

great. Thanks!

On Mar 26, 2022, at 10:04 AM, Mike Kucera @.**@.>> wrote:

The JSON data for this network is indeed huge (1.9GB). Normally the EM data model is serialized to a JSON String and stored in a CyTable. The problem in this case is that the data is so large it won't fit into a String object, and it throws an OutOfMemoryError during JSON serialization.

I'm working on a new strategy for saving the data that will handle the OutOfMemoryError. The strategy is to catch the OutOfMemoryError if it happens and then save the JSON to a separate file inside the session instead. The caveat is that this will require a new version of EM, and saving the data in the new format will not be backwards compatible with previous versions of EM. But it will only effect sessions that trigger an OutOFMemoryError when saving like this one.

— Reply to this email directly, view it on GitHubhttps://github.com/BaderLab/EnrichmentMapApp/issues/482#issuecomment-1079701056, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AA5KFS7ASMYPUMY765UFCHTVB4KPPANCNFSM5RRVO6KA. You are receiving this because you authored the thread.Message ID: @.***>