Silenced out of memory error

[fanavarro]

Hi all, the last week I created an issue because I was not able to generate the owl ontology file for the last version of Snomed (#37). The generated owl showed an incorrect hierarchy of concepts, where most of them were outside the Snomed root concept. This was provoked because there was an out of memory error, which was silenced in the process.

I am doing experiments with the different versions of snomed along the time, and I repeated the process by assigning 4GB of memory. Then, I noticed that the owl files generated with 2GB of memory were, in general, different to the files generated with 4GB. For example, the following figure shows the snomed 2019-01-31 owl file generated with 2Gb and with 2GB:

As you can see, annotations are created incorrectly, but they could be legit so I did not detect the error.

Would it be possible to show this out of memory error, at least in the log?

[kaicode]

Hi @fanavarro, Thanks for reporting this. This is a bug which needs to be fixed.

The workaround using the logs is to check you have a line containing the string "reference set members read from sct2_sRefset_OWLExpressionSnapshot". If this string is not present in the log output then the OWL axioms have not been completely read and the ontology file will be incomplete.

We will try to implement better exception handling for this in the next couple of weeks. Pull requests welcome.

Kind regards, Kai

[grahamponting]

Hi all, I am getting the "out of memory" error using using the 2.9.0 file, detaiIs included. Is there any timeline for a fix. ?

Many thanks, Graham Ponting error track 13Aug2020.txt

[kaicode]

Hi @grahamponting,

Your log shows this command is being used: java -jar snomed-owl-toolkit-2.9.0-executable.jar -rf2-to-owl -rf2-snapshot-archives SnomedCT_InternationalRF2_PRODUCTION_20200309T120000Z.zip. Please give 4g of memory as shown in the example in the README.

The corrected version of the command in your log would be: java -Xms4g -jar snomed-owl-toolkit-2.9.0-executable.jar -rf2-to-owl -rf2-snapshot-archives SnomedCT_InternationalRF2_PRODUCTION_20200309T120000Z.zip.

It is not possible for Java to use more memory than it's given. It is also not possible for this program to use less memory. I'm pleased that the program reported the error effectively. I'll close this ticket now.

Kind regards, Kai Kewley

[grahamponting]

Thanks Kai, When I try and run with the updated memory, I get the following error;

C:\Users\Graham Ponting\Downloads\Downloads\owl TEST>java -Xms4g -jar snomed-owl-toolkit-2.9.0-executable.jar -rf2-to-owl -rf2-snapshot-archives SnomedCT_ InternationalRF2_PRODUCTION_20200309T120000Z.zip Invalid initial heap size: -Xms4g The specified size exceeds the maximum representable size. Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit.

Any ideas. ? My PC has 16G of Ram and there was little else running at the time so I presume it’s to do with the java config. ?

Many many thanks for your help. Graham Ponting

From: Kai Kewley notifications@github.com Sent: 13 August 2020 16:39 To: IHTSDO/snomed-owl-toolkit snomed-owl-toolkit@noreply.github.com Cc: grahamponting grahamaponting@hotmail.com; Mention mention@noreply.github.com Subject: Re: [IHTSDO/snomed-owl-toolkit] Silenced out of memory error (#38)

Hi @grahampontinghttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgrahamponting&data=02%7C01%7C%7C258cbcd47c0f4e26ea8208d83f9ef65c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637329299247091923&sdata=FLxLA6CGrXvPdQm3ti1Y1Z27WPxfNRc3Q0rdhdVUCeA%3D&reserved=0,

Your log shows this command is being used: java -jar snomed-owl-toolkit-2.9.0-executable.jar -rf2-to-owl -rf2-snapshot-archives SnomedCT_InternationalRF2_PRODUCTION_20200309T120000Z.zip. Please give 4g of memory as shown in the example in the READMEhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FIHTSDO%2Fsnomed-owl-toolkit%23snomed-rf2-classification&data=02%7C01%7C%7C258cbcd47c0f4e26ea8208d83f9ef65c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637329299247101798&sdata=RJ89W4Azl0gA6sRNS1sP%2BlL1fElh2rUT676YJiUhWHQ%3D&reserved=0.

The corrected version of the command in your log would be: java -Xms4g -jar snomed-owl-toolkit-2.9.0-executable.jar -rf2-to-owl -rf2-snapshot-archives SnomedCT_InternationalRF2_PRODUCTION_20200309T120000Z.zip.

It is not possible for Java to use more memory than it's given. It is also not possible for this program to use less memory. I'm pleased that the program reported the error effectively. I'll close this ticket now.

Kind regards, Kai Kewley

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FIHTSDO%2Fsnomed-owl-toolkit%2Fissues%2F38%23issuecomment-673551206&data=02%7C01%7C%7C258cbcd47c0f4e26ea8208d83f9ef65c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637329299247111903&sdata=PekVFBNBYz7h%2BCtlIsFTwvX1e6z7T1giO00t3zG2uPo%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJSWI6VFTR4IXH7M7JFJPO3SAQCIHANCNFSM4KWO4IOA&data=02%7C01%7C%7C258cbcd47c0f4e26ea8208d83f9ef65c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637329299247121883&sdata=lo7IJ4R%2BFI%2F2fCFiohXHvKtobw30H1XIOYS1FOKr7L8%3D&reserved=0.

[CoderMChu]

Hi @grahamponting,

Can you check the Java version installed on your PC please using the following command in a command line? java -d64

If you get an error when running above command then you will not be able to use 4g due to "On most modern 32-bit Windows systems the maximum heap size will range from 1.4G to 1.6G".

See more details via the link here Why can't I get a larger heap with the 32-bit JVM?

If you have got 32 bit JRE then please install the 64 bit. Hopefully this will solve your issue.

Kind regards,

Michael

[grahamponting]

Hi Michael,

Many thanks, I have upgraded to 64 bit Java and that means I can now produce an OWL file which I have imported into Protégé, although the Elk reasoner seems unstable.

Thanks very much for your help.

From: Michael Chu notifications@github.com Sent: 14 August 2020 11:18 To: IHTSDO/snomed-owl-toolkit snomed-owl-toolkit@noreply.github.com Cc: grahamponting grahamaponting@hotmail.com; Mention mention@noreply.github.com Subject: Re: [IHTSDO/snomed-owl-toolkit] Silenced out of memory error (#38)

Hi @grahampontinghttps://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgrahamponting&data=02%7C01%7C%7C6b1c9cb15fa84c197fcd08d8403b5193%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637329970791385470&sdata=zUV22AX177mGc2t3glvQflQigVC2QAS6sKhFh34nuEw%3D&reserved=0,

Can you check the Java version installed on your PC please using the following command in a command line? java -d64

If you get an error when running above command then you will not be able to use 4g due to "On most modern 32-bit Windows systems the maximum heap size will range from 1.4G to 1.6G".

See more details via the link here Why can't I get a larger heap with the 32-bit JVM?https://eur06.safelinks.protection.outlook.com/?url=https:%2F%2Fwww.oracle.com%2Fjava%2Ftechnologies%2Fhotspotfaq.html%23:~:text%3DThe%2520maximum%2520theoretical%2520heap%2520limit%2C1.4G%2520to%25201.6G.&data=02%7C01%7C%7C6b1c9cb15fa84c197fcd08d8403b5193%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637329970791395465&sdata=sLhcptKxmJjwueGvgF8mUSRGh8JWneG5C29i08JR%2BVM%3D&reserved=0

If you have got 32 bit JRE then please install the 64 bit. Hopefully this will solve your issue.

Kind regards,

Michael

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FIHTSDO%2Fsnomed-owl-toolkit%2Fissues%2F38%23issuecomment-674004126&data=02%7C01%7C%7C6b1c9cb15fa84c197fcd08d8403b5193%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637329970791395465&sdata=zVG5G9NWD5P59P%2Ft1sT2m6vbJOGCzjWcegby9OFP6xQ%3D&reserved=0, or unsubscribehttps://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJSWI6W45SKMKL7DIM55DDDSAUFNLANCNFSM4KWO4IOA&data=02%7C01%7C%7C6b1c9cb15fa84c197fcd08d8403b5193%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637329970791395465&sdata=RSQrnForL74rhzVjxW8jew8DzrfkfMb%2FfCJrhL9zqYM%3D&reserved=0.