Missing of KR matrices and malfunction of addNorm utility

[Yonghao-Holden]

Are you sure this is a bug? If unsure, please post your question/situation to the forum first: aidenlab.org/forum.html

Describe the bug Hello,

I used the code below to dump observed KR matrix from .hic files, however, some .hic files don't have KR-normalized matrixes at a certain resolution. For example, one .hic file has KR normalization matrixes for chr11_25kb, chr11_50kb, chr11_100kb, chr12_25kb, chr12_100kb, but don't have the one for chr12_50kb. Although it worked perfectly with VC, we need KR-normalized matrix for our pipeline.

java -jar juicer_tools.jar dump observed KR sample.hic 12 12 BP 50000

The only message I got after running this code is as below: WARN [2021-01-05T19:16:10,229] [Globals.java:138] [main] Development mode is enabled

No error message or any message. This is the only message that popped out.

I searched in the forum and found this post (https://groups.google.com/g/3d-genomics/c/vsUgDt21CF4/m/-lIByy3HBgAJ). It says it may because the KR normalization didn't converge, so no output was generated. And it suggests us to increase the iteration time for function computeKR in juicebox.tools.utils.original.NormalizationCalculations.java., so that the matrix can converge, following by running addNorm function to re-create KR vectors for the .hic file.

Before editing the code, I tried the addNorm function to see if it's working properly. However, when I ran the code below, I lost all normalized matrices under different normalization (KR, VC, VC_SQRT), different resolution, and different chromosomes (I tested 1-22 and chr1-chr22). Although the size of the .hic file is the same after running addNorm.

java -jar juicer_tools.jar addNorm sample.hic -j 10

java -jar juicer_tools.jar addNorm sample.hic -j 10 -w 500

This is the version of Juicer I'm using right now: Juicer Tools Version 1.22.01

So there are two main issues happening:

1. Some KR-normalized matrices are missing at a certain resolution for certain chromosomes.

2. After running addNorm, all normalization vectors are missing in the .hic file.

I will greatly appreciate it if you guys can give us some suggestions to fix those issues!

Thank you ahead!

Holden

To Reproduce Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. macOS Mojave, Windows 10, etc]
• Version [e.g. 1.6]

Additional context Add any other context about the problem here.

[yanchunzhang]

I also got the same error. The matrix had no value after addNorm.

[cbravo93]

I have also encountered the same issue. In my case Im using HiC files from the latest release of ENCODE (processed with juicer 2.09 and juicertools 2.13.05). All HiC files seem to have been affected by this bug, and addNorm causes all values to be gone.

[sa501428]

Can you clarify which normalizations were present before/after running addnorm? Was there an error encountered during the run?

On Mon, Jan 24, 2022, 7:00 AM cbravo @.***> wrote:

I have also encountered the same issue. In my case Im using HiC files from the latest release of ENCODE (processed with juicer 2.09 and juicertools 2.13.05). All HiC files seem to have been affected by this bug, and addNorm causes all values to be gone.

— Reply to this email directly, view it on GitHub https://github.com/aidenlab/juicer/issues/201#issuecomment-1020073881, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRT23PRX26KY4RN4V2I3C3UXVENRANCNFSM4V6SGPIA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you are subscribed to this thread.Message ID: @.***>

[cbravo93]

All default normalization work except for KR (NONE, VC, VC_SQRT and SCALE). Then after finding this issue I tested running myself addNorm (using ENCFF685BLG as example). This is the code I ran:

java -jar ${juicer_tools_path}juicer_tools.2.13.05.jar dump observed VC test.hic 'chr1' 'chr1' BP 500000 test_VC_1.txt
head test_VC_1.txt
java -jar ${juicer_tools_path}juicer_tools.2.13.05.jar dump observed KR test.hic 'chr1' 'chr1' BP 500000 test_KR_1.txt
head test_KR_1.txt
java -jar ${juicer_tools_path}juicer_tools.2.13.05.jar addNorm test.hic -j 8
java -jar ${juicer_tools_path}juicer_tools.2.13.05.jar dump observed KR test.hic 'chr1' 'chr1' BP 500000 test_KR_2.txt
head test_KR_2.txt
java -jar ${juicer_tools_path}juicer_tools.2.13.05.jar dump observed VC test.hic 'chr1' 'chr1' BP 500000 test_VC_2.txt
head test_VC_2.txt

KR is empty. I can also provide the ids for the other hic files I am using, they come from the Deeply Profiled Cell Lines collection.

[sa501428]

So this isn't a bug; KR has been deprecated in Juicer 2 in favor of SCALE, which converges more reliably and runs faster.

[sa501428]

Also I'd recommend Straw over Dump; can read the Hi-C data directly into your language of choice.