search

aidenlab / 3d-dna

3D de novo assembly (3D DNA) pipeline
MIT License
207 stars 55 forks source link

error with run-asm-pipeline.sh #100

Open Huangtianyu-caas opened 3 years ago

Huangtianyu-caas commented 3 years ago

Hi,

I was running ./run-asm-pipeline.sh and generated below error:

  `...detecting misjoins in round 0 assembly:
  :) -p flag was triggered. Running with GNU Parallel support parameter set to false.
  :) -c flag was triggered, starting calculations with 5% saturation level
  :) -w flag was triggered, performing cursory search for mismatches at 25000 resolution
  :) -d flag was triggered, depletion score will be averaged across a region bounded by 125000 superdiagonal
  :) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold
  :) -n flag was triggered, performing mismatch region thinning at 1000 resolution
  ...Dumping 25000 resolution matrix
  Could not read hic file: null
  :( Juicebox dump is empty! Perhaps something is wrong with the hic file or the requested resolution is too high. Exiting!
  :) -w flag was triggered, performing cursory search for repeat at 25000 resolution
  :) -t flag was triggered, flagging regions with coverage higher than 2
  ...Dumping 25000 resolution coverage track
  Could not read hic file: null
  :( Juicebox coverage dump is empty! Perhaps something is wrong with the hic file or the requested resolution is too high. Exiting!
  mv: cannot stat `depletion_score_wide.wig': No such file or directory
  mv: cannot stat `depletion_score_narrow.wig': No such file or directory
  mv: cannot stat `mismatch_wide.bed': No such file or directory
  mv: cannot stat `mismatch_narrow.bed': No such file or directory
  mv: cannot stat `coverage_wide.wig': No such file or directory
  mv: cannot stat `repeats_wide.bed': No such file or directory
  cat: mismatch_narrow.at.step.0.bed: No such file or directory
  cat: repeats_wide.at.step.0.bed: No such file or directory
  awk: cmd. line:1: fatal: cannot open file `purged2.0_asm.superscaf_track.txt' for reading (No such file or directory)
  awk: /public/home/huangtianyu/software/3d-dna/edit/overlay-edits.awk:20: fatal: cannot open file `purged2.0_asm.scaffold_track.txt' for reading (No such file or directory)
  :) No more input edits to be done. Moving to polishing!
  :( No resolved files are found. Please rerun the pipeline to include the scaffold segment. Exiting!`

This error has already been reported, but did not come out a proper resolution.

Could you please give me some suggestion about this issue?

My best, Tianyu

dudcha commented 3 years ago

Hi Tianyu,

The messages you show suggest that the .hic files didn't build. Look for messages upstream from what you share from Java that might give you a clue why.

Best, Olga

Huangtianyu-caas commented 3 years ago

Hi Olga,

Below is the ls -la output of the aligned folder: drwxr-xr-x 3 huangtianyu wangguirong_group 8192 Dec 24 02:58 . drwxr-xr-x 6 huangtianyu wangguirong_group 8192 Jan 5 12:05 .. -rw-r--r-- 1 huangtianyu wangguirong_group 21958556468 Dec 23 09:21 abnormal.sam -rw-r--r-- 1 huangtianyu wangguirong_group 1 Dec 23 09:35 collisions_nodups.txt -rw-r--r-- 1 huangtianyu wangguirong_group 1 Dec 23 09:35 collisions.txt -rw-r--r-- 1 huangtianyu wangguirong_group 11667002029 Dec 23 08:20 dups.txt -rw-r--r-- 1 huangtianyu wangguirong_group 512 Dec 22 20:11 header drwxr-xr-x 2 huangtianyu wangguirong_group 8192 Dec 24 03:01 inter_30_contact_domains -rw-r--r-- 1 huangtianyu wangguirong_group 305298605 Dec 24 02:58 inter_30.hic -rw-r--r-- 1 huangtianyu wangguirong_group 13529 Dec 23 21:45 inter_30_hists.m -rw-r--r-- 1 huangtianyu wangguirong_group 1905 Dec 23 21:45 inter_30.txt -rw-r--r-- 1 huangtianyu wangguirong_group 373813916 Dec 23 18:57 inter.hic -rw-r--r-- 1 huangtianyu wangguirong_group 14271 Dec 23 09:19 inter_hists.m -rw-r--r-- 1 huangtianyu wangguirong_group 1905 Dec 23 09:19 inter.txt -rw-r--r-- 1 huangtianyu wangguirong_group 53247617281 Dec 23 08:20 merged_nodups.txt -rw-r--r-- 1 huangtianyu wangguirong_group 65283703843 Dec 23 07:30 merged_sort.txt -rw-r--r-- 1 huangtianyu wangguirong_group 369084533 Dec 23 08:20 opt_dups.txt -rw-r--r-- 1 huangtianyu wangguirong_group 2738803700 Dec 23 09:21 unmapped.sam

It seems that .hic files were successfully built. Actually, when running run-asm-pipeline.sh, only 'merged_nodups.txt' and conference genome file were required, do you mean there may be some error with my 'merged_nodups.txt' file?

My best, Tianyu

dudcha commented 3 years ago

Hi Tianyu,

The aligned folder and hic files generated by Juicer are irrelevant here. The file that is not being found by 3D-DNA is the one after assembly. Has a form of .0.hic, e.g.

Best, Olga

Huangtianyu-caas commented 3 years ago

Hi Olga,

Thanks for your explanation, could you please inform me where should be the right place of .0.hic file (like the name of the folder that contains 0.hic file)?

Best,

dudcha commented 3 years ago

The folder in which you are running 3d-dna. -Olga

On Jan 4, 2021, at 10:25 PM, Huangtianyu-caas notifications@github.com wrote:

Hi Olga,

Thanks for your explanation, could you please inform me where should be the right place of .0.hic file (like the name of the folder that contains 0.hic file)?

Best,

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/aidenlab/3d-dna/issues/100#issuecomment-754385292, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACLAMG4THJZOXHSTZ4XXXNLSYKIFBANCNFSM4VR2H2OQ.

Huangtianyu-caas commented 3 years ago

Hi Tianyu,

The messages you show suggest that the .hic files didn't build. Look for messages upstream from what you share from Java that might give you a clue why.

Best, Olga

Oh, these are the whole messages: `version: 180922 :| WARNING: GNU Parallel version 20150322 or later not installed. We highly recommend to install it to increase performance. Starting pipeline without parallelization! awk: /public/home/huangtianyu/software/3d-dna/utils/generate-sorted-cprops-file.awk:15: fatal: 3 is invalid as number of arguments for asorti /public/home/huangtianyu/software/3d-dna/run-asm-pipeline.sh: line 634: 1 + / 2100000000 : syntax error: operand expected (error token is "/ 2100000000 ") /public/home/huangtianyu/software/3d-dna/run-asm-pipeline.sh: line 635: [: -ne: unary operator expected ############### Starting iterating scaffolding with editing: ...starting round 0 of scaffolding: :) -p flag was triggered. Running LIGer with GNU Parallel support parameter set to false. :) -s flag was triggered, starting calculations with 15000 threshold starting contig/scaffold size :) -q flag was triggered, starting calculations with 1 threshold mapping quality :( Cprops file not found. Please double-check your input!!

*****************************************************
This is a wrapper for a Hi-C Limitless Iterative Greedy genome assembly (LIGer) algorithm, version date: Dec 7, 2016.

Usage: ./run-liger-scaffolder.sh [-h] [-s minimal_scaffold_size] [-t link_threshold] [-q mapq] path_to_cprops_file path_to_merge_nodups_file

ARGUMENTS:
path_to_cprops_file     Path to (prefiltered) cprops file listing contigs for which LIGer scaffolding will be attempted
path_to_merge_nodups_file  Path to merge_nodups Juicer output file

OPTIONS:
-h                      Shows this help
-s size                                 Set minimal contig/scaffold size to use as input
-q mapq                 Set threshold for Hi-C reads mapping quality (default is 1)
-p true/false                   Use GNU Parallel to speed up calculations (default is true)
-t link_threshod        Set threshold for joining links [not working yet, uses default]

Uses scrape-mnd.awk, generate-unsorted-confidence-table.awk, confidence-to-assembly.awk, scaffolds-to-original-notation.awk and drop-smallest-dubious-element.awk that should be in the same folder as the wrapper script.

In the current version ordering and orienting contigs at each iteration is based on the data from  whole inter-contig contact matrices. At each iteraction the contigs and corresponding matrices are updated, the idea being that some low-confidence scaffolds links may benefit from more data provided by the already-joined contigs and get resolved.

Note that in the current version the input is expected in the cprops format.
*****************************************************

...visualizing round 0 results:
:) -p flag was triggered. Running with GNU Parallel support parameter set to false.
:) -q flag was triggered, starting calculations for 1 threshold mapping quality
:) -i flag was triggered, building mapq without
:) -c flag was triggered, will remove temporary files after completion
:( Not sure how to parse your input or input files not found at intended locations. Exiting!

*****************************************************
Visualizing draft genomes in juicebox: 18 July 2016

USAGE: ./run-asm-visualizer.sh [options] <path_to_cprops> <path_to_asm> <path_to_contig_mnd>

DESCRIPTION:
This is a script to visualize draft assemblies (represented in input by their cprops and asm files) from pairwise contact data represented by Juicer merged_nodups.txt file. The script will produce hic files for viewing in Juicebox as well as scaffold annotation files. Metadata can be attached to the map by passing -i and -g flags with paths to stats and graph files.

ARGUMENTS:
path_to_cprops                          Specify cprops file path.
path_to_asm                                     Specify asm file.
path_to_contig_mnd                      Specify path to mnd file describing pairwise Hi-C contacts between assembly contigs.

OPTIONS:
-q mapq                                         Build map for a specific mapq threshold (default is 1).
-p true/false                           Use GNU Parallel to speed up computation (default is true).
-z zoom                                         Build map with hardcoded zoom level. By default this is calculated based on the cprops file and applied only to genomes >= 2.1 Gb.

**unprompted**
#-l gap_file                            Path to gap bed file - necessary to build contig annotation track.
-m path_to_asm_mnd                      Path to mnd already lifted from input to assembly chromosome: used to skip the remapping step.
-n                                                      Skip normalization.
-r                                                      Build for specific resolutions (default is -r 2500000,1000000,500000,250000,100000,50000,25000,10000,5000,1000)
-c                                                      Clean up when done (default: no cleanup.)
-i                                                      Ignore mapq suffix.
-h                                                      Shows this help
*****************************************************

...detecting misjoins in round 0 assembly:
:) -p flag was triggered. Running with GNU Parallel support parameter set to false.
:) -c flag was triggered, starting calculations with 5% saturation level
:) -w flag was triggered, performing cursory search for mismatches at 25000 resolution
:) -d flag was triggered, depletion score will be averaged across a region bounded by 125000 superdiagonal
:) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold
:) -n flag was triggered, performing mismatch region thinning at 1000 resolution
...Dumping 25000 resolution matrix
Could not read hic file: null
:( Juicebox dump is empty! Perhaps something is wrong with the hic file or the requested resolution is too high. Exiting!
:) -w flag was triggered, performing cursory search for repeat at 25000 resolution
:) -t flag was triggered, flagging regions with coverage higher than 2
...Dumping 25000 resolution coverage track
Could not read hic file: null
:( Juicebox coverage dump is empty! Perhaps something is wrong with the hic file or the requested resolution is too high. Exiting!
mv: cannot stat `depletion_score_wide.wig': No such file or directory
mv: cannot stat `depletion_score_narrow.wig': No such file or directory
mv: cannot stat `mismatch_wide.bed': No such file or directory
mv: cannot stat `mismatch_narrow.bed': No such file or directory
mv: cannot stat `coverage_wide.wig': No such file or directory
mv: cannot stat `repeats_wide.bed': No such file or directory
cat: mismatch_narrow.at.step.0.bed: No such file or directory
cat: repeats_wide.at.step.0.bed: No such file or directory
awk: cmd. line:1: fatal: cannot open file `purged2.0_asm.superscaf_track.txt' for reading (No such file or directory)
awk: /public/home/huangtianyu/software/3d-dna/edit/overlay-edits.awk:20: fatal: cannot open file `purged2.0_asm.scaffold_track.txt' for reading (No such file or directory)
:) No more input edits to be done. Moving to polishing!
:( No resolved files are found. Please rerun the pipeline to include the scaffold segment. Exiting!`
dudcha commented 3 years ago

Make sure you make run-asm-pipeline executable or execute directly with bash. I highly recommend installing GNU parallel. In theory we support the non-parallel version, but it is much less tested since we ourselves don't use it. -Olga

Huangtianyu-caas commented 3 years ago

GNU parallel was installed, and has been detected by 3d-dna. Also I use bash ~/software/3d-dna/run-asm-pipeline.sh to run, but still generated this error:

        `version: 180922
        awk: /public/home/huangtianyu/software/3d-dna/utils/generate-sorted-cprops-file.awk:15: fatal: 3 is invalid as number of arguments for asorti
        /public/home/huangtianyu/software/3d-dna/run-asm-pipeline.sh: line 634: 1 +  / 2100000000 : syntax error: operand expected (error token is "/ 2100000000 ")
        /public/home/huangtianyu/software/3d-dna/run-asm-pipeline.sh: line 635: [: -ne: unary operator expected
        /public/home/huangtianyu/software/3d-dna/run-asm-pipeline.sh: line 656: [: purged2.0.cprops: binary operator expected
        ###############
        Starting iterating scaffolding with editing:
        ...starting round 0 of scaffolding:
        :) -p flag was triggered. Running LIGer with GNU Parallel support parameter set to true.
        :) -s flag was triggered, starting calculations with 15000 threshold starting contig/scaffold size
        :) -q flag was triggered, starting calculations with 1 threshold mapping quality
        :( Cprops file not found. Please double-check your input!!

        *****************************************************
        This is a wrapper for a Hi-C Limitless Iterative Greedy genome assembly (LIGer) algorithm, version date: Dec 7, 2016.

        Usage: ./run-liger-scaffolder.sh [-h] [-s minimal_scaffold_size] [-t link_threshold] [-q mapq] path_to_cprops_file path_to_merge_nodups_file

        ARGUMENTS:
        path_to_cprops_file     Path to (prefiltered) cprops file listing contigs for which LIGer scaffolding will be attempted
        path_to_merge_nodups_file  Path to merge_nodups Juicer output file

        OPTIONS:
        -h                      Shows this help
        -s size                                 Set minimal contig/scaffold size to use as input
        -q mapq                 Set threshold for Hi-C reads mapping quality (default is 1)
        -p true/false                   Use GNU Parallel to speed up calculations (default is true)
        -t link_threshod        Set threshold for joining links [not working yet, uses default]

        Uses scrape-mnd.awk, generate-unsorted-confidence-table.awk, confidence-to-assembly.awk, scaffolds-to-original-notation.awk and drop-smallest-dubious-element.awk that should be in the same folder as the wrapper script.

        In the current version ordering and orienting contigs at each iteration is based on the data from  whole inter-contig contact matrices. At each iteraction the contigs and corresponding matrices are updated, the idea being that some low-confidence scaffolds links may benefit from more data provided by the already-joined contigs and get resolved.

        Note that in the current version the input is expected in the cprops format.
        *****************************************************

        ...visualizing round 0 results:
        :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
        :) -q flag was triggered, starting calculations for 1 threshold mapping quality
        :) -i flag was triggered, building mapq without
        :) -c flag was triggered, will remove temporary files after completion
        :( Not sure how to parse your input or input files not found at intended locations. Exiting!

        *****************************************************
        Visualizing draft genomes in juicebox: 18 July 2016

        USAGE: ./run-asm-visualizer.sh [options] <path_to_cprops> <path_to_asm> <path_to_contig_mnd>

        DESCRIPTION:
        This is a script to visualize draft assemblies (represented in input by their cprops and asm files) from pairwise contact data represented by Juicer merged_nodups.txt file. The script will produce hic files for viewing in Juicebox as well as scaffold annotation files. Metadata can be attached to the map by passing -i and -g flags with paths to stats and graph files.

        ARGUMENTS:
        path_to_cprops                          Specify cprops file path.
        path_to_asm                                     Specify asm file.
        path_to_contig_mnd                      Specify path to mnd file describing pairwise Hi-C contacts between assembly contigs.

        OPTIONS:
        -q mapq                                         Build map for a specific mapq threshold (default is 1).
        -p true/false                           Use GNU Parallel to speed up computation (default is true).
        -z zoom                                         Build map with hardcoded zoom level. By default this is calculated based on the cprops file and applied only to genomes >= 2.1 Gb.

        **unprompted**
        #-l gap_file                            Path to gap bed file - necessary to build contig annotation track.
        -m path_to_asm_mnd                      Path to mnd already lifted from input to assembly chromosome: used to skip the remapping step.
        -n                                                      Skip normalization.
        -r                                                      Build for specific resolutions (default is -r 2500000,1000000,500000,250000,100000,50000,25000,10000,5000,1000)
        -c                                                      Clean up when done (default: no cleanup.)
        -i                                                      Ignore mapq suffix.
        -h                                                      Shows this help
        *****************************************************

        ...detecting misjoins in round 0 assembly:
        :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
        :) -c flag was triggered, starting calculations with 5% saturation level
        :) -w flag was triggered, performing cursory search for mismatches at 25000 resolution
        :) -d flag was triggered, depletion score will be averaged across a region bounded by 125000 superdiagonal
        :) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold
        :) -n flag was triggered, performing mismatch region thinning at 1000 resolution
        ...Dumping 25000 resolution matrix
        Could not read hic file: null
        :( Juicebox dump is empty! Perhaps something is wrong with the hic file or the requested resolution is too high. Exiting!
        :) -w flag was triggered, performing cursory search for repeat at 25000 resolution
        :) -t flag was triggered, flagging regions with coverage higher than 2
        ...Dumping 25000 resolution coverage track
        Could not read hic file: null
        :( Juicebox coverage dump is empty! Perhaps something is wrong with the hic file or the requested resolution is too high. Exiting!
        mv: cannot stat `depletion_score_wide.wig': No such file or directory
        mv: cannot stat `depletion_score_narrow.wig': No such file or directory
        mv: cannot stat `mismatch_wide.bed': No such file or directory
        mv: cannot stat `mismatch_narrow.bed': No such file or directory
        mv: cannot stat `coverage_wide.wig': No such file or directory
        mv: cannot stat `repeats_wide.bed': No such file or directory
        cat: mismatch_narrow.at.step.0.bed: No such file or directory
        cat: repeats_wide.at.step.0.bed: No such file or directory
        awk: cmd. line:1: fatal: cannot open file `purged2.0_asm.superscaf_track.txt' for reading (No such file or directory)
        awk: /public/home/huangtianyu/software/3d-dna/edit/overlay-edits.awk:20: fatal: cannot open file `purged2.0_asm.scaffold_track.txt' for reading (No such file or directory)
        :) No more input edits to be done. Moving to polishing!
        :( No resolved files are found. Please rerun the pipeline to include the scaffold segment. Exiting!`
dudcha commented 3 years ago

See this: https://github.com/aidenlab/3d-dna/issues/25. Also check the awk version. -Olga

On Jan 4, 2021, at 11:36 PM, Huangtianyu-caas notifications@github.com wrote:

GNU parallel was installed, and has been detected by 3d-dna. Also I use bash ~/software/3d-dna/run-asm-pipeline.sh to run, but still generated this error: `version: 180922 awk: /public/home/huangtianyu/software/3d-dna/utils/generate-sorted-cprops-file.awk:15: fatal: 3 is invalid as number of arguments for asorti /public/home/huangtianyu/software/3d-dna/run-asm-pipeline.sh: line 634: 1 + / 2100000000 : syntax error: operand expected (error token is "/ 2100000000 ") /public/home/huangtianyu/software/3d-dna/run-asm-pipeline.sh: line 635: [: -ne: unary operator expected /public/home/huangtianyu/software/3d-dna/run-asm-pipeline.sh: line 656: [: purged2.0.cprops: binary operator expected ############### Starting iterating scaffolding with editing: ...starting round 0 of scaffolding: :) -p flag was triggered. Running LIGer with GNU Parallel support parameter set to true. :) -s flag was triggered, starting calculations with 15000 threshold starting contig/scaffold size :) -q flag was triggered, starting calculations with 1 threshold mapping quality :( Cprops file not found. Please double-check your input!!

    *****************************************************
    This is a wrapper for a Hi-C Limitless Iterative Greedy genome assembly (LIGer) algorithm, version date: Dec 7, 2016.

    Usage: ./run-liger-scaffolder.sh [-h] [-s minimal_scaffold_size] [-t link_threshold] [-q mapq] path_to_cprops_file path_to_merge_nodups_file

    ARGUMENTS:
    path_to_cprops_file     Path to (prefiltered) cprops file listing contigs for which LIGer scaffolding will be attempted
    path_to_merge_nodups_file  Path to merge_nodups Juicer output file

    OPTIONS:
    -h                      Shows this help
    -s size                                 Set minimal contig/scaffold size to use as input
    -q mapq                 Set threshold for Hi-C reads mapping quality (default is 1)
    -p true/false                   Use GNU Parallel to speed up calculations (default is true)
    -t link_threshod        Set threshold for joining links [not working yet, uses default]

    Uses scrape-mnd.awk, generate-unsorted-confidence-table.awk, confidence-to-assembly.awk, scaffolds-to-original-notation.awk and drop-smallest-dubious-element.awk that should be in the same folder as the wrapper script.

    In the current version ordering and orienting contigs at each iteration is based on the data from  whole inter-contig contact matrices. At each iteraction the contigs and corresponding matrices are updated, the idea being that some low-confidence scaffolds links may benefit from more data provided by the already-joined contigs and get resolved.

    Note that in the current version the input is expected in the cprops format.
    *****************************************************

    ...visualizing round 0 results:
    :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
    :) -q flag was triggered, starting calculations for 1 threshold mapping quality
    :) -i flag was triggered, building mapq without
    :) -c flag was triggered, will remove temporary files after completion
    :( Not sure how to parse your input or input files not found at intended locations. Exiting!

    *****************************************************
    Visualizing draft genomes in juicebox: 18 July 2016

    USAGE: ./run-asm-visualizer.sh [options] <path_to_cprops> <path_to_asm> <path_to_contig_mnd>

    DESCRIPTION:
    This is a script to visualize draft assemblies (represented in input by their cprops and asm files) from pairwise contact data represented by Juicer merged_nodups.txt file. The script will produce hic files for viewing in Juicebox as well as scaffold annotation files. Metadata can be attached to the map by passing -i and -g flags with paths to stats and graph files.

    ARGUMENTS:
    path_to_cprops                          Specify cprops file path.
    path_to_asm                                     Specify asm file.
    path_to_contig_mnd                      Specify path to mnd file describing pairwise Hi-C contacts between assembly contigs.

    OPTIONS:
    -q mapq                                         Build map for a specific mapq threshold (default is 1).
    -p true/false                           Use GNU Parallel to speed up computation (default is true).
    -z zoom                                         Build map with hardcoded zoom level. By default this is calculated based on the cprops file and applied only to genomes >= 2.1 Gb.

    **unprompted**
    #-l gap_file                            Path to gap bed file - necessary to build contig annotation track.
    -m path_to_asm_mnd                      Path to mnd already lifted from input to assembly chromosome: used to skip the remapping step.
    -n                                                      Skip normalization.
    -r                                                      Build for specific resolutions (default is -r 2500000,1000000,500000,250000,100000,50000,25000,10000,5000,1000)
    -c                                                      Clean up when done (default: no cleanup.)
    -i                                                      Ignore mapq suffix.
    -h                                                      Shows this help
    *****************************************************

    ...detecting misjoins in round 0 assembly:
    :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
    :) -c flag was triggered, starting calculations with 5% saturation level
    :) -w flag was triggered, performing cursory search for mismatches at 25000 resolution
    :) -d flag was triggered, depletion score will be averaged across a region bounded by 125000 superdiagonal
    :) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold
    :) -n flag was triggered, performing mismatch region thinning at 1000 resolution
    ...Dumping 25000 resolution matrix
    Could not read hic file: null
    :( Juicebox dump is empty! Perhaps something is wrong with the hic file or the requested resolution is too high. Exiting!
    :) -w flag was triggered, performing cursory search for repeat at 25000 resolution
    :) -t flag was triggered, flagging regions with coverage higher than 2
    ...Dumping 25000 resolution coverage track
    Could not read hic file: null
    :( Juicebox coverage dump is empty! Perhaps something is wrong with the hic file or the requested resolution is too high. Exiting!
    mv: cannot stat `depletion_score_wide.wig': No such file or directory
    mv: cannot stat `depletion_score_narrow.wig': No such file or directory
    mv: cannot stat `mismatch_wide.bed': No such file or directory
    mv: cannot stat `mismatch_narrow.bed': No such file or directory
    mv: cannot stat `coverage_wide.wig': No such file or directory
    mv: cannot stat `repeats_wide.bed': No such file or directory
    cat: mismatch_narrow.at.step.0.bed: No such file or directory
    cat: repeats_wide.at.step.0.bed: No such file or directory
    awk: cmd. line:1: fatal: cannot open file `purged2.0_asm.superscaf_track.txt' for reading (No such file or directory)
    awk: /public/home/huangtianyu/software/3d-dna/edit/overlay-edits.awk:20: fatal: cannot open file `purged2.0_asm.scaffold_track.txt' for reading (No such file or directory)
    :) No more input edits to be done. Moving to polishing!
    :( No resolved files are found. Please rerun the pipeline to include the scaffold segment. Exiting!`

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/aidenlab/3d-dna/issues/100#issuecomment-754408688, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACLAMG6R4WY6SRG6FUGT3ULSYKQNDANCNFSM4VR2H2OQ.

Huangtianyu-caas commented 3 years ago

Hi Olga,

Thanks for your suggestion, I found the error was caused by the wrong version of gawk.

After updating the gawk, the pipeline started to work, but it seems that the pipeline did not successfully finished. Since the log file is too long, I only present the last part of it:

  `Finished preprocess
  HiC file version: 8

  Calculating norms for zoom BP_2500000
  Calculating norms for zoom BP_1000000
  Calculating norms for zoom BP_500000
  Calculating norms for zoom BP_250000
  Calculating norms for zoom BP_100000
  Calculating norms for zoom BP_50000
  Calculating norms for zoom BP_25000
  Calculating norms for zoom BP_10000
  Calculating norms for zoom BP_5000
  Calculating norms for zoom BP_1000
  Writing expected
  Writing norms
  Finished writing norms
  ###############
  Starting split:
  :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
  ...-q flag was triggered, will ignore all reads with mapping quality less then 1 for polishing
  ...-j flag was triggered, will use Juicebox map purged2.polished.hic
  ...-a flag was triggered, will use scaffold annotation file purged2.polished_asm.scaffold_track.txt
  ...-b flag was triggered, will use superscaffold annotation file purged2.polished_asm.superscaf_track.txt
  ...-s flag was triggered, will ignore all scaffolds shorter than 100000 for polishing
  :) -c flag was triggered, starting calculations with 5% saturation level
  :) -w flag was triggered, performing cursory search for mismatches at 100000 resolution
  :) -d flag was triggered, depletion score will be averaged across a region bounded by 3000000 superdiagonal
  :) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold
  :) -n flag was triggered, performing mismatch region thinning at 1000 resolution
  :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
  :) -c flag was triggered, starting calculations with 5% saturation level
  :) -w flag was triggered, performing cursory search for mismatches at 100000 resolution
  :) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold
  :) -d flag was triggered, depletion score will be averaged across a region bounded by 3000000 superdiagonal
  :) -n flag was triggered, performing mismatch region thinning at 1000 resolution
  ...Dumping 100000 resolution matrix
  HiC file version: 8
  ...Estimating necessary saturation level for requested misassembly sensitivity
  sort: unrecognized option '--parallel=24'
  Try `sort --help' for more information.
  ...Coarse resolution saturation level =
  ...Analyzing near-diagonal mismatches
  ...Filtering mismatch calls without better alternative: Not functional yet, skipping
  ...Thinning mismatch region boundaries.
  ...Dumping 1000 resolution matrix
  HiC file version: 8
  ...Estimating necessary saturation level for requested misassembly sensitivity
  sort: unrecognized option '--parallel=24'
  Try `sort --help' for more information.
  ...Fine resolution saturation level =
  ...Analyzing near-diagonal mismatches
  :( Assembly file contains multiple entries of the same contig. Exiting!
  :( Assembly file contains multiple entries of the same contig. Exiting!
  ...-p flag was triggered. Running with GNU Parallel support parameter set to true.
  ...-r flag was triggered, output will be labeled as *.split.*
  ...applying edits to cprops file
  ...applying edits to mnd file
  :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
  :) -q flag was triggered, starting calculations for 1 threshold mapping quality
  :) -i flag was triggered, building mapq without
  :) -c flag was triggered, will remove temporary files after completion
  ...Remapping contact data from the original contig set to assembly
  ...Building track files
  ...Building the hic file
  Not including fragment map
  Start preprocess
  Writing header
  Writing body
  ..
  Writing footer

  Finished preprocess
  HiC file version: 8

  Calculating norms for zoom BP_2500000
  Calculating norms for zoom BP_1000000
  Calculating norms for zoom BP_500000
  Calculating norms for zoom BP_250000
  Calculating norms for zoom BP_100000
  Calculating norms for zoom BP_50000
  Calculating norms for zoom BP_25000
  Calculating norms for zoom BP_10000
  Calculating norms for zoom BP_5000
  Calculating norms for zoom BP_1000
  Writing expected
  Writing norms
  Finished writing norms
  ###############
  Starting sealing:
  ... -i flag was triggered, will attempt to place back only debris contigs/scaffolds and those less than 15000
  :| Warning: no explicit bundle size was listed. Will use the same one as listed for false positive size threshold: this is the most typical scenario.
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
  :) -q flag was triggered, starting calculations for 1 threshold mapping quality
  :) -i flag was triggered, building mapq without
  :) -c flag was triggered, will remove temporary files after completion
  ...Remapping contact data from the original contig set to assembly
  ...Building track files
  ...Building the hic file
  Not including fragment map
  Start preprocess
  Writing header
  Writing body
  ..
  Writing footer

  Finished preprocess
  HiC file version: 8

  Calculating norms for zoom BP_2500000
  Calculating norms for zoom BP_1000000
  Calculating norms for zoom BP_500000
  Calculating norms for zoom BP_250000
  Calculating norms for zoom BP_100000
  Calculating norms for zoom BP_50000
  Calculating norms for zoom BP_25000
  Calculating norms for zoom BP_10000
  Calculating norms for zoom BP_5000
  Calculating norms for zoom BP_1000
  Writing expected
  Writing norms
  Finished writing norms
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  ###############
  Finilizing output:
  ... -s flag was triggered, treating all contigs/scaffolds shorter than 15000 as unattempted.
  ... -l flag was triggered. Output will appear with headers of the form purged2_hic_scaffold_#.
  ... -g flag was triggered, making gap size between scaffolded draft sequences to be equal to 500.
  Analyzing the merged assembly
  ...trimming N overhangs
  ...adding gaps`

Could you please inform me if this report is right? I finally got a .final.fasta file which is almostly twice large of my reference genome, is that a wrong result?

My best, Tianyu

dudcha commented 3 years ago

Hi Tianyu,

Check your coreutils: "sort: unrecognized option '--parallel=24’". See in readme: GNU coreutils sort >=8.11 Best, Olga

On Jan 7, 2021, at 9:22 PM, Huangtianyu-caas notifications@github.com wrote:

Hi Olga,

Thanks for your suggestion, I found the error was caused by the wrong version of gawk.

After updating the gawk, the pipeline started to work, but it seems that the pipeline did not successfully finished. Since the log file is too long, I only present the last part of it:

`Finished preprocess HiC file version: 8

Calculating norms for zoom BP_2500000 Calculating norms for zoom BP_1000000 Calculating norms for zoom BP_500000 Calculating norms for zoom BP_250000 Calculating norms for zoom BP_100000 Calculating norms for zoom BP_50000 Calculating norms for zoom BP_25000 Calculating norms for zoom BP_10000 Calculating norms for zoom BP_5000 Calculating norms for zoom BP_1000 Writing expected Writing norms Finished writing norms ############### Starting split: :) -p flag was triggered. Running with GNU Parallel support parameter set to true. ...-q flag was triggered, will ignore all reads with mapping quality less then 1 for polishing ...-j flag was triggered, will use Juicebox map purged2.polished.hic ...-a flag was triggered, will use scaffold annotation file purged2.polished_asm.scaffold_track.txt ...-b flag was triggered, will use superscaffold annotation file purged2.polished_asm.superscaf_track.txt ...-s flag was triggered, will ignore all scaffolds shorter than 100000 for polishing :) -c flag was triggered, starting calculations with 5% saturation level :) -w flag was triggered, performing cursory search for mismatches at 100000 resolution :) -d flag was triggered, depletion score will be averaged across a region bounded by 3000000 superdiagonal :) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold :) -n flag was triggered, performing mismatch region thinning at 1000 resolution :) -p flag was triggered. Running with GNU Parallel support parameter set to true. :) -c flag was triggered, starting calculations with 5% saturation level :) -w flag was triggered, performing cursory search for mismatches at 100000 resolution :) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold :) -d flag was triggered, depletion score will be averaged across a region bounded by 3000000 superdiagonal :) -n flag was triggered, performing mismatch region thinning at 1000 resolution ...Dumping 100000 resolution matrix HiC file version: 8 ...Estimating necessary saturation level for requested misassembly sensitivity sort: unrecognized option '--parallel=24' Try sort --help' for more information. ...Coarse resolution saturation level = ...Analyzing near-diagonal mismatches ...Filtering mismatch calls without better alternative: Not functional yet, skipping ...Thinning mismatch region boundaries. ...Dumping 1000 resolution matrix HiC file version: 8 ...Estimating necessary saturation level for requested misassembly sensitivity sort: unrecognized option '--parallel=24' Trysort --help' for more information. ...Fine resolution saturation level = ...Analyzing near-diagonal mismatches :( Assembly file contains multiple entries of the same contig. Exiting! :( Assembly file contains multiple entries of the same contig. Exiting! ...-p flag was triggered. Running with GNU Parallel support parameter set to true. ...-r flag was triggered, output will be labeled as .split. ...applying edits to cprops file ...applying edits to mnd file :) -p flag was triggered. Running with GNU Parallel support parameter set to true. :) -q flag was triggered, starting calculations for 1 threshold mapping quality :) -i flag was triggered, building mapq without :) -c flag was triggered, will remove temporary files after completion ...Remapping contact data from the original contig set to assembly ...Building track files ...Building the hic file Not including fragment map Start preprocess Writing header Writing body .. Writing footer

Finished preprocess HiC file version: 8

Calculating norms for zoom BP_2500000 Calculating norms for zoom BP_1000000 Calculating norms for zoom BP_500000 Calculating norms for zoom BP_250000 Calculating norms for zoom BP_100000 Calculating norms for zoom BP_50000 Calculating norms for zoom BP_25000 Calculating norms for zoom BP_10000 Calculating norms for zoom BP_5000 Calculating norms for zoom BP1000 Writing expected Writing norms Finished writing norms ############### Starting sealing: ... -i flag was triggered, will attempt to place back only debris contigs/scaffolds and those less than 15000 :| Warning: no explicit bundle size was listed. Will use the same one as listed for false positive size threshold: this is the most typical scenario. :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :) -p flag was triggered. Running with GNU Parallel support parameter set to true. :) -q flag was triggered, starting calculations for 1 threshold mapping quality :) -i flag was triggered, building mapq without :) -c flag was triggered, will remove temporary files after completion ...Remapping contact data from the original contig set to assembly ...Building track files ...Building the hic file Not including fragment map Start preprocess Writing header Writing body .. Writing footer

Finished preprocess HiC file version: 8

Calculating norms for zoom BP_2500000 Calculating norms for zoom BP_1000000 Calculating norms for zoom BP_500000 Calculating norms for zoom BP_250000 Calculating norms for zoom BP_100000 Calculating norms for zoom BP_50000 Calculating norms for zoom BP_25000 Calculating norms for zoom BP_10000 Calculating norms for zoom BP_5000 Calculating norms for zoom BP1000 Writing expected Writing norms Finished writing norms :| Warning: No input for label1 was provided. Default for label1 is ":::fragment" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" ############### Finilizing output: ... -s flag was triggered, treating all contigs/scaffolds shorter than 15000 as unattempted. ... -l flag was triggered. Output will appear with headers of the form purged2_hicscaffold#. ... -g flag was triggered, making gap size between scaffolded draft sequences to be equal to 500. Analyzing the merged assembly ...trimming N overhangs ...adding gaps` Could you please inform me if this report is right? I finally got a .final.fasta file which is almostly twice large of my reference genome, is that a wrong result?

My best, Tianyu

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/aidenlab/3d-dna/issues/100#issuecomment-756522573, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACLAMG3N3LJAPMGXFVPC4I3SYZ3AVANCNFSM4VR2H2OQ.

Huangtianyu-caas commented 3 years ago

Hi Olga,

I have updated GNU coreutils sort to 8.25, however the error still there, below error presented over and over again while the pipeline is still on going:

  `...Remapping contact data from the original contig set to assembly
  ...Building track files
  ...Building the hic file
  Not including fragment map
  Start preprocess
  Writing header
  Writing body
  ..
  Writing footer

  Finished preprocess
  HiC file version: 8

  Calculating norms for zoom BP_2500000
  Calculating norms for zoom BP_1000000
  Calculating norms for zoom BP_500000
  Calculating norms for zoom BP_250000
  Calculating norms for zoom BP_100000
  Calculating norms for zoom BP_50000
  Calculating norms for zoom BP_25000
  Calculating norms for zoom BP_10000
  Calculating norms for zoom BP_5000
  Calculating norms for zoom BP_1000/public/home/huangtianyu/software/3d-dna/visualize/juicebox_tools.sh: line 3: 14278 Killed                  java -Xms49152m -Xmx49152m -jar `dirname $0`/juicebox_tools.jar $*
  mv: cannot stat 'purged2.resolved.polish.mismatch_narrow.bed': No such file or directory
  mv: cannot stat 'purged2.resolved.polish.depletion_score_narrow.wig': No such file or directory
  mv: cannot stat 'purged2.resolved.polish.mismatch_wide.bed': No such file or directory
  mv: cannot stat 'purged2.resolved.polish.depletion_score_wide.wig': No such file or directory
  ###############
  Starting split:
  :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
  ...-q flag was triggered, will ignore all reads with mapping quality less then 1 for polishing
  ...-j flag was triggered, will use Juicebox map purged2.polished.hic
  ...-a flag was triggered, will use scaffold annotation file purged2.polished_asm.scaffold_track.txt
  ...-b flag was triggered, will use superscaffold annotation file purged2.polished_asm.superscaf_track.txt
  ...-s flag was triggered, will ignore all scaffolds shorter than 100000 for polishing
  :) -c flag was triggered, starting calculations with 5% saturation level
  :) -w flag was triggered, performing cursory search for mismatches at 100000 resolution
  :) -d flag was triggered, depletion score will be averaged across a region bounded by 3000000 superdiagonal
  :) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold
  :) -n flag was triggered, performing mismatch region thinning at 1000 resolution
  :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
  :) -c flag was triggered, starting calculations with 5% saturation level
  :) -w flag was triggered, performing cursory search for mismatches at 100000 resolution
  :) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold
  :) -d flag was triggered, depletion score will be averaged across a region bounded by 3000000 superdiagonal
  :) -n flag was triggered, performing mismatch region thinning at 1000 resolution
  ...Dumping 100000 resolution matrix
  HiC file version: 8
  Unable to dump
  java.io.IOException: Normalization missing for: assembly_assembly_BP_100000
          at juicebox.data.DatasetReaderV2.readNormalizedBlock(DatasetReaderV2.java:719)
          at juicebox.data.MatrixZoomData.dump(MatrixZoomData.java:675)
          at juicebox.tools.clt.old.Dump.dumpMatrix(Dump.java:282)
          at juicebox.tools.clt.old.Dump.run(Dump.java:505)
          at juicebox.tools.HiCTools.main(HiCTools.java:86)
  ...Estimating necessary saturation level for requested misassembly sensitivity
  ...Coarse resolution saturation level =
  ...Analyzing near-diagonal mismatches
  ...Filtering mismatch calls without better alternative: Not functional yet, skipping
  ...Thinning mismatch region boundaries.
  ...Dumping 1000 resolution matrix
  HiC file version: 8
  ...Estimating necessary saturation level for requested misassembly sensitivity
  ...Fine resolution saturation level = 2
  ...Analyzing near-diagonal mismatches
  ...-p flag was triggered. Running with GNU Parallel support parameter set to true.
  ...-r flag was triggered, output will be labeled as *.split.*
  ...applying edits to cprops file
  ...applying edits to mnd file
  :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
  :) -q flag was triggered, starting calculations for 1 threshold mapping quality
  :) -i flag was triggered, building mapq without
  :) -c flag was triggered, will remove temporary files after completion
  ...Remapping contact data from the original contig set to assembly
  ...Building track files
  ...Building the hic file
  Not including fragment map
  Start preprocess
  Writing header
  Writing body
  ./public/home/huangtianyu/software/3d-dna/visualize/juicebox_tools.sh: line 3: 59351 Killed                  java -Xms49152m -Xmx49152m -jar `dirname $0`/juicebox_tools.jar $*
  ###############
  Starting sealing:
  ... -i flag was triggered, will attempt to place back only debris contigs/scaffolds and those less than 15000
  :| Warning: no explicit bundle size was listed. Will use the same one as listed for false positive size threshold: this is the most typical scenario.
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"
  :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
  :) -q flag was triggered, starting calculations for 1 threshold mapping quality
  :) -i flag was triggered, building mapq without
  :) -c flag was triggered, will remove temporary files after completion
  ...Remapping contact data from the original contig set to assembly
  ...Building track files
  ...Building the hic file
  Not including fragment map
  Start preprocess
  Writing header
  Writing body
  ./public/home/huangtianyu/software/3d-dna/visualize/juicebox_tools.sh: line 3:  4748 Killed                  java -Xms49152m -Xmx49152m -jar `dirname $0`/juicebox_tools.jar $*
  :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_"
  :| Warning: No input for label2 was provided. Default for label2 is ":::debris"`

My best, Tianyu

dudcha commented 3 years ago

Here’s your error

juicebox_tools.sh: line 3: 14278 Killed java -Xms49152m -Xmx49152m -jar dirname $0/juicebox_tools.jar $* Make sure you have enough memory Olga

On Jan 8, 2021, at 11:08 PM, Huangtianyu-caas notifications@github.com wrote:

juicebox_tools.sh: line 3: 14278 Killed java -Xms49152m -Xmx49152m -jar dirname $0/juicebox_tools.jar $*

Huangtianyu-caas commented 3 years ago

Hi Olga,

I ran the pipeline with enough memory, it still continue reporting errors, this time I ran the pipeline on a PBS system, so the errors were divided into two parts.

The one is the normal results reports: `version: 180922 ############### Starting iterating scaffolding with editing: ...starting round 0 of scaffolding: :) -p flag was triggered. Running LIGer with GNU Parallel support parameter set to true. :) -s flag was triggered, starting calculations with 15000 threshold starting contig/scaffold size :) -q flag was triggered, starting calculations with 1 threshold mapping quality ...Using cprops file: purged2.0.cprops ...Using merged_nodups file: purged2.mnd.0.txt ...Scaffolding all scaffolds and contigs greater or equal to 15000 bp. ...Starting iteration # 1 ...Starting iteration # 2 ...Starting iteration # 3 ...Starting iteration # 4 ...Starting iteration # 5 ...Starting iteration # 6 ...Starting iteration # 7 ...Starting iteration # 8 ...Starting iteration # 9 ...Starting iteration # 10 ...Starting iteration # 11 ...Starting iteration # 12 ...Starting iteration # 13 ...Starting iteration # 14 ...Starting iteration # 15 ...Starting iteration # 16 ...Starting iteration # 17 ...Starting iteration # 18 ...Starting iteration # 19 ...Starting iteration # 20 ...Starting iteration # 21 ...Starting iteration # 22 ...Starting iteration # 23 ...Starting iteration # 24 :) DONE! ...visualizing round 0 results: :) -p flag was triggered. Running with GNU Parallel support parameter set to true. :) -q flag was triggered, starting calculations for 1 threshold mapping quality :) -i flag was triggered, building mapq without :) -c flag was triggered, will remove temporary files after completion ...Remapping contact data from the original contig set to assembly ...Building track files ...Building the hic file Not including fragment map Start preprocess Writing header Writing body .. Writing footer

    Finished preprocess
    HiC file version: 8

    Calculating norms for zoom BP_2500000
    Calculating norms for zoom BP_1000000
    Calculating norms for zoom BP_500000
    Calculating norms for zoom BP_250000
    Calculating norms for zoom BP_100000
    Calculating norms for zoom BP_50000
    Calculating norms for zoom BP_25000
    Calculating norms for zoom BP_10000
    Calculating norms for zoom BP_5000
    Calculating norms for zoom BP_1000
    Writing expected
    Writing norms
    Finished writing norms
    ...detecting misjoins in round 0 assembly:
    :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
    :) -c flag was triggered, starting calculations with 5% saturation level
    :) -w flag was triggered, performing cursory search for mismatches at 25000 resolution
    :) -d flag was triggered, depletion score will be averaged across a region bounded by 125000 superdiagonal
    :) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold
    :) -n flag was triggered, performing mismatch region thinning at 1000 resolution
    ...Dumping 25000 resolution matrix
    HiC file version: 8
    ...Estimating necessary saturation level for requested misassembly sensitivity
    ...Coarse resolution saturation level = 7.17577
    ...Analyzing near-diagonal mismatches
    ...Filtering mismatch calls without better alternative: Not functional yet, skipping
    ...Thinning mismatch region boundaries.
    ...Dumping 1000 resolution matrix
    HiC file version: 8
    ...Estimating necessary saturation level for requested misassembly sensitivity
    ...Fine resolution saturation level = 2
    ...Analyzing near-diagonal mismatches
    :) -w flag was triggered, performing cursory search for repeat at 25000 resolution
    :) -t flag was triggered, flagging regions with coverage higher than 2
    ...Dumping 25000 resolution coverage track
    HiC file version: 8
    ...-p flag was triggered. Running with GNU Parallel support parameter set to true.
    ...-r flag was triggered, output will be labeled as *.1.*
    ...applying edits to cprops file
    ...applying edits to mnd file
    ...starting round 1 of scaffolding:
    :) -p flag was triggered. Running LIGer with GNU Parallel support parameter set to true.
    :) -s flag was triggered, starting calculations with 15000 threshold starting contig/scaffold size
    :) -q flag was triggered, starting calculations with 1 threshold mapping quality
    ...Using cprops file: purged2.1.cprops
    ...Using merged_nodups file: purged2.mnd.1.txt
    ...Scaffolding all scaffolds and contigs greater or equal to 15000 bp.
    ...Starting iteration # 1
    ...Starting iteration # 2
    ...Starting iteration # 3
    ...Starting iteration # 4
    ...Starting iteration # 5
    ...Starting iteration # 6
    ...Starting iteration # 7
    ...Starting iteration # 8
    ...Starting iteration # 9
    ...Starting iteration # 10
    ...Starting iteration # 11
    ...Starting iteration # 12
    ...Starting iteration # 13
    ...Starting iteration # 14
    ...Starting iteration # 15
    ...Starting iteration # 16
    ...Starting iteration # 17
    ...Starting iteration # 18
    ...Starting iteration # 19
    ...Starting iteration # 20
    ...Starting iteration # 21
    ...Starting iteration # 22
    ...Starting iteration # 23
    ...Starting iteration # 24
    ...Starting iteration # 25
    ...Starting iteration # 26
    ...Starting iteration # 27
    ...Starting iteration # 28
    ...Starting iteration # 29
    ...Starting iteration # 30
    :) DONE!
    ...visualizing round 1 results:
    :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
    :) -q flag was triggered, starting calculations for 1 threshold mapping quality
    :) -i flag was triggered, building mapq without
    :) -c flag was triggered, will remove temporary files after completion
    ...Remapping contact data from the original contig set to assembly
    ...Building track files
    ...Building the hic file
    Not including fragment map
    Start preprocess
    Writing header
    Writing body
    ..
    Writing footer

    Finished preprocess
    HiC file version: 8

    Calculating norms for zoom BP_2500000
    Calculating norms for zoom BP_1000000
    Calculating norms for zoom BP_500000
    Calculating norms for zoom BP_250000
    Calculating norms for zoom BP_100000
    Calculating norms for zoom BP_50000
    Calculating norms for zoom BP_25000
    Calculating norms for zoom BP_10000
    Calculating norms for zoom BP_5000
    Calculating norms for zoom BP_1000
    Writing expected
    Writing norms
    Finished writing norms
    ...detecting misjoins in round 1 assembly:
    :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
    :) -c flag was triggered, starting calculations with 5% saturation level
    :) -w flag was triggered, performing cursory search for mismatches at 25000 resolution
    :) -d flag was triggered, depletion score will be averaged across a region bounded by 125000 superdiagonal
    :) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold
    :) -n flag was triggered, performing mismatch region thinning at 1000 resolution
    ...Dumping 25000 resolution matrix
    HiC file version: 8
    ...Estimating necessary saturation level for requested misassembly sensitivity
    ...Coarse resolution saturation level = 7.16204
    ...Analyzing near-diagonal mismatches
    ...Filtering mismatch calls without better alternative: Not functional yet, skipping
    ...Thinning mismatch region boundaries.
    ...Dumping 1000 resolution matrix
    HiC file version: 8
    ...Estimating necessary saturation level for requested misassembly sensitivity
    ...Fine resolution saturation level = 2
    ...Analyzing near-diagonal mismatches
    :) -w flag was triggered, performing cursory search for repeat at 25000 resolution
    :) -t flag was triggered, flagging regions with coverage higher than 2
    ...Dumping 25000 resolution coverage track
    HiC file version: 8
    ...-p flag was triggered. Running with GNU Parallel support parameter set to true.
    ...-r flag was triggered, output will be labeled as *.2.*
    ...applying edits to cprops file
    ...applying edits to mnd file
    ...starting round 2 of scaffolding:
    :) -p flag was triggered. Running LIGer with GNU Parallel support parameter set to true.
    :) -s flag was triggered, starting calculations with 15000 threshold starting contig/scaffold size
    :) -q flag was triggered, starting calculations with 1 threshold mapping quality
    ...Using cprops file: purged2.2.cprops
    ...Using merged_nodups file: purged2.mnd.2.txt
    ...Scaffolding all scaffolds and contigs greater or equal to 15000 bp.
    ...Starting iteration # 1
    ...Starting iteration # 2
    ...Starting iteration # 3
    ...Starting iteration # 4
    ...Starting iteration # 5
    ...Starting iteration # 6
    ...Starting iteration # 7
    ...Starting iteration # 8
    ...Starting iteration # 9
    ...Starting iteration # 10
    ...Starting iteration # 11
    ...Starting iteration # 12
    ...Starting iteration # 13
    ...Starting iteration # 14
    :) DONE!
    ...visualizing round 2 results:
    :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
    :) -q flag was triggered, starting calculations for 1 threshold mapping quality
    :) -i flag was triggered, building mapq without
    :) -c flag was triggered, will remove temporary files after completion
    ...Remapping contact data from the original contig set to assembly
    ...Building track files
    ...Building the hic file
    Not including fragment map
    Start preprocess
    Writing header
    Writing body
    ..
    Writing footer

    Finished preprocess
    HiC file version: 8

    Calculating norms for zoom BP_2500000
    Calculating norms for zoom BP_1000000
    Calculating norms for zoom BP_500000
    Calculating norms for zoom BP_250000
    Calculating norms for zoom BP_100000
    Calculating norms for zoom BP_50000
    Calculating norms for zoom BP_25000
    Calculating norms for zoom BP_10000
    Calculating norms for zoom BP_5000
    Calculating norms for zoom BP_1000
    Writing expected
    Writing norms
    Finished writing norms
    ###############
    Starting polish:
    :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
    :) -q flag was triggered, performing polishing taking into account signal with minimum 1 mapping quality
    ...-j flag was triggered, will use Juicebox map purged2.resolved.hic
    ...-a flag was triggered, will use scaffold annotation file purged2.resolved_asm.scaffold_track.txt
    ...-b flag was triggered, will use superscaffold annotation file purged2.resolved_asm.superscaf_track.txt
    ...-s flag was triggered, will ignore all scaffolds shorter than 1000000 for polishing
    :) -c flag was triggered, starting calculations with 5% saturation level
    :) -w flag was triggered, performing cursory search for mismatches at 100000 resolution
    :) -d flag was triggered, depletion score will be averaged across a region bounded by 3000000 superdiagonal
    :) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold
    :) -n flag was triggered, performing mismatch region thinning at 1000 resolution
    :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
    :) -c flag was triggered, starting calculations with 5% saturation level
    :) -w flag was triggered, performing cursory search for mismatches at 100000 resolution
    :) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold
    :) -d flag was triggered, depletion score will be averaged across a region bounded by 3000000 superdiagonal
    :) -n flag was triggered, performing mismatch region thinning at 1000 resolution
    ...Dumping 100000 resolution matrix
    HiC file version: 8
    ...Estimating necessary saturation level for requested misassembly sensitivity
    ...Coarse resolution saturation level = 30.5271
    ...Analyzing near-diagonal mismatches
    ...Filtering mismatch calls without better alternative: Not functional yet, skipping
    ...Thinning mismatch region boundaries.
    ...Dumping 1000 resolution matrix
    HiC file version: 8
    ...Estimating necessary saturation level for requested misassembly sensitivity
    ...Fine resolution saturation level = 2
    ...Analyzing near-diagonal mismatches
    ...-p flag was triggered. Running with GNU Parallel support parameter set to true.
    ...-r flag was triggered, output will be labeled as *.polish.*
    ...applying edits to cprops file
    ...applying edits to mnd file
    :) -p flag was triggered. Running LIGer with GNU Parallel support parameter set to true.
    :) -q flag was triggered, starting calculations with 1 threshold mapping quality
    :) -s flag was triggered, starting calculations with 1000000 threshold starting contig/scaffold size
    ...Using cprops file: purged2.resolved.polish.cprops
    ...Using merged_nodups file: purged2.resolved.polish.mnd.txt
    ...Explicit scaffold set has been listed as input. Using set as a first iteration.
    ...Starting iteration # 1
    ...Starting iteration # 2
    ...Starting iteration # 3
    ...Starting iteration # 4
    ...Starting iteration # 5
    ...Starting iteration # 6
    :) DONE!
    :) -p flag was triggered. Running LIGer with GNU Parallel support parameter set to true.
    :) -q flag was triggered, starting calculations with 1 threshold mapping quality
    :) -s flag was triggered, starting calculations with 1000000 threshold starting contig/scaffold size
    ...Using cprops file: purged2.resolved.polish.cprops
    ...Using merged_nodups file: purged2.resolved.polish.mnd.txt
    ...Explicit scaffold set has been listed as input. Using set as a first iteration.
    ...Starting iteration # 1
    :) DONE!
    :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
    :) -q flag was triggered, starting calculations for 1 threshold mapping quality
    :) -i flag was triggered, building mapq without
    :) -c flag was triggered, will remove temporary files after completion
    ...Remapping contact data from the original contig set to assembly
    ...Building track files
    ...Building the hic file
    Not including fragment map
    Start preprocess
    Writing header
    Writing body
    ..
    Writing footer

    Finished preprocess
    HiC file version: 8

    Calculating norms for zoom BP_2500000
    Calculating norms for zoom BP_1000000
    Calculating norms for zoom BP_500000
    Calculating norms for zoom BP_250000
    Calculating norms for zoom BP_100000
    Calculating norms for zoom BP_50000
    Calculating norms for zoom BP_25000
    Calculating norms for zoom BP_10000
    Calculating norms for zoom BP_5000
    Calculating norms for zoom BP_1000
    Writing expected
    Writing norms
    Finished writing norms
    ###############
    Starting split:
    :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
    ...-q flag was triggered, will ignore all reads with mapping quality less then 1 for polishing
    ...-j flag was triggered, will use Juicebox map purged2.polished.hic
    ...-a flag was triggered, will use scaffold annotation file purged2.polished_asm.scaffold_track.txt
    ...-b flag was triggered, will use superscaffold annotation file purged2.polished_asm.superscaf_track.txt
    ...-s flag was triggered, will ignore all scaffolds shorter than 100000 for polishing
    :) -c flag was triggered, starting calculations with 5% saturation level
    :) -w flag was triggered, performing cursory search for mismatches at 100000 resolution
    :) -d flag was triggered, depletion score will be averaged across a region bounded by 3000000 superdiagonal
    :) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold
    :) -n flag was triggered, performing mismatch region thinning at 1000 resolution
    :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
    :) -c flag was triggered, starting calculations with 5% saturation level
    :) -w flag was triggered, performing cursory search for mismatches at 100000 resolution
    :) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold
    :) -d flag was triggered, depletion score will be averaged across a region bounded by 3000000 superdiagonal
    :) -n flag was triggered, performing mismatch region thinning at 1000 resolution
    ...Dumping 100000 resolution matrix
    HiC file version: 8
    ...Estimating necessary saturation level for requested misassembly sensitivity
    ...Coarse resolution saturation level = 30.5435
    ...Analyzing near-diagonal mismatches
    ...Filtering mismatch calls without better alternative: Not functional yet, skipping
    ...Thinning mismatch region boundaries.
    ...Dumping 1000 resolution matrix
    HiC file version: 8
    ...Estimating necessary saturation level for requested misassembly sensitivity
    ...Fine resolution saturation level = 2
    ...Analyzing near-diagonal mismatches
    ...-p flag was triggered. Running with GNU Parallel support parameter set to true.
    ...-r flag was triggered, output will be labeled as *.split.*
    ...applying edits to cprops file
    ...applying edits to mnd file
    :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
    :) -q flag was triggered, starting calculations for 1 threshold mapping quality
    :) -i flag was triggered, building mapq without
    :) -c flag was triggered, will remove temporary files after completion
    ...Remapping contact data from the original contig set to assembly
    ...Building track files
    ...Building the hic file
    Not including fragment map
    Start preprocess
    Writing header
    Writing body
    ..
    Writing footer

    Finished preprocess
    HiC file version: 8

    Calculating norms for zoom BP_2500000
    Calculating norms for zoom BP_1000000
    Calculating norms for zoom BP_500000
    Calculating norms for zoom BP_250000
    Calculating norms for zoom BP_100000
    Calculating norms for zoom BP_50000
    Calculating norms for zoom BP_25000
    Calculating norms for zoom BP_10000
    Calculating norms for zoom BP_5000
    Calculating norms for zoom BP_1000
    Writing expected
    Writing norms
    Finished writing norms
    ###############
    Starting sealing:
    ... -i flag was triggered, will attempt to place back only debris contigs/scaffolds and those less than 15000
    :) -p flag was triggered. Running with GNU Parallel support parameter set to true.
    :) -q flag was triggered, starting calculations for 1 threshold mapping quality
    :) -i flag was triggered, building mapq without
    :) -c flag was triggered, will remove temporary files after completion
    ...Remapping contact data from the original contig set to assembly
    ...Building track files
    ...Building the hic file
    Not including fragment map
    Start preprocess
    Writing header
    Writing body
    ..
    Writing footer

    Finished preprocess
    HiC file version: 8

    Calculating norms for zoom BP_2500000
    Calculating norms for zoom BP_1000000
    Calculating norms for zoom BP_500000
    Calculating norms for zoom BP_250000
    Calculating norms for zoom BP_100000
    Calculating norms for zoom BP_50000
    Calculating norms for zoom BP_25000
    Calculating norms for zoom BP_10000
    Calculating norms for zoom BP_5000
    Calculating norms for zoom BP_1000
    Writing expected
    Writing norms
    Finished writing norms
    ###############
    Finilizing output:
    ... -s flag was triggered, treating all contigs/scaffolds shorter than 15000 as unattempted.
    ... -l flag was triggered. Output will appear with headers of the form purged2_hic_scaffold_#.
    ... -g flag was triggered, making gap size between scaffolded draft sequences to be equal to 500.
    Analyzing the merged assembly
    ...trimming N overhangs
    ...adding gaps`

Another is the error reports: :| Warning: no explicit bundle size was listed. Will use the same one as listed for false positive size threshold: this is the most typical scenario. :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" :| Warning: No input for label1 was provided. Default for label1 is ":::fragment_" :| Warning: No input for label2 was provided. Default for label2 is ":::debris" tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe tail: error writing 'standard output': Broken pipe

I used to successfully finish 3d-dna pipeline on another cluster server (however I no longer have the opportunity to use it), the 'Warning: No input for label1 was provided. Default for label1 is ":::fragment_":| Warning: No input for label2 was provided. Default for label2 is ":::debris"' error was only reported for once, but this time, this error was reported over and over again, seems some problem with it.

My best, Tianyu

Huangtianyu-caas commented 3 years ago

Hi Olga,

I have changed to another cluster, and the same error still going on. Could you please check this a little bit?

Thank you in advance.

My best, Tianyu

dudcha commented 3 years ago

Hello Tianyu,

The warning is not of concern, this is just a note in an off chance that your contig names contain sequences like this (which would be a problem). The broken pipe tail error is discussed here: https://github.com/aidenlab/3d-dna/issues/46.

Best, Olga

Huangtianyu-caas commented 3 years ago

Hello Olga,

Thanks for providing the #46. I try to add dd insertion into the line 62 and 64 of construct-fasta-from-asm.sh, but I am not sure if I was doing right, because I am not a computor expert. I add the dd insertion as follow: tail -c +${index[${contig}]} ${fasta} | awk '$0~/>/{exit}1' | awk -f ${pipeline}/utils/reverse-fasta.awk - | dd obs=1M tail -c +${index[${contig}]} ${fasta} | awk '$0~/>/{exit}1' | dd obs=1M

I re-ran the pipeline, still can't work properly. I think I did not add the dd insertion in a right way.

My best, Tianyu

licheng0921 commented 3 years ago

Hi Tianyu,

I also meet this error.

Have you figured it out?

Best, Licheng

pramodpantha commented 2 years ago

Hello Olga,

Thanks for providing the #46. I try to add dd insertion into the line 62 and 64 of construct-fasta-from-asm.sh, but I am not sure if I was doing right, because I am not a computor expert. I add the dd insertion as follow: tail -c +${index[${contig}]} ${fasta} | awk '$0~/>/{exit}1' | awk -f ${pipeline}/utils/reverse-fasta.awk - | dd obs=1M tail -c +${index[${contig}]} ${fasta} | awk '$0~/>/{exit}1' | dd obs=1M

I re-ran the pipeline, still can't work properly. I think I did not add the dd insertion in a right way.

My best, Tianyu

Hi, I also have a similar issue. Could you please confirm if this is the correct way to add dd insertion into the lines 62 and 64? Thank you in advance.