search

google / deepvariant

DeepVariant is an analysis pipeline that uses a deep neural network to call genetic variants from next-generation DNA sequencing data.
BSD 3-Clause "New" or "Revised" License
3.24k stars 729 forks source link

No output vcf after long run #249

Closed se2cheeese closed 4 years ago

se2cheeese commented 4 years ago

Dears,

I just followed "DeepVariant quick start" guide, which was successful. Then, I've replaced the reference and reads with my own files and ran following commands but failed to get vcf files.

OUTPUT_DIR="${PWD}/quickstart-output" INPUT_DIR="${PWD}/quickstart-testdata" mkdir -p "${OUTPUT_DIR}" BIN_VERSION="0.9.0"

sudo docker run \ -v "${INPUT_DIR}":"/input" \ -v "${OUTPUT_DIR}:/output" \ gcr.io/deepvariant-docker/deepvariant:"${BIN_VERSION}" \ /opt/deepvariant/bin/run_deepvariant \ --model_type=WES \ --ref=/input/genome.fa \ --reads=/input/HC3-BC_RG_bwa.bam \ --regions "20:10,000,000-10,100,000" \ --output_vcf=/output/output.vcf.gz \ --output_gvcf=/output/output.g.vcf.gz \ --num_shards=12

Here is a log: 

***** Running the command:*****
time seq 0 11 | parallel -k --line-buffer /opt/deepvariant/bin/make_examples --mode calling --ref "/input/genome.fa" --reads "/input/HC3-BC_RG_bwa.bam" --examples "/tmp/deepvariant_tmp_output/make_examples.tfrecord@12.gz" --gvcf "/tmp/deepvariant_tmp_output/gvcf.tfrecord@12.gz" --regions "20:10,000,000-10,100,000" --task {}

I1208 07:03:00.340749 140610455504640 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
min_base_quality: 10
min_base_quality_mode: ENFORCED_BY_CLIENT

I1208 07:03:00.598805 140610455504640 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1208 07:03:00.619033 140610455504640 make_examples.py:1324] Preparing inputs
I1208 07:03:00.814711 140610455504640 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1208 07:03:00.841161 140610455504640 make_examples.py:1248] Common contigs are [u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'10', u'11', u'12', u'13', u'14', u'15', u'16', u'17', u'18', u'19', u'20', u'21', u'22', u'X', u'Y']
I1208 07:03:00.851603 140610455504640 make_examples.py:1330] Writing examples to /tmp/deepvariant_tmp_output/make_examples.tfrecord-00000-of-00012.gz
I1208 07:03:00.852047 140610455504640 make_examples.py:1334] Writing gvcf records to /tmp/deepvariant_tmp_output/gvcf.tfrecord-00000-of-00012.gz
I1208 07:03:00.870843 140610455504640 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2019-12-08 07:03:00.878937: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
I1208 07:03:00.416043 139802617698048 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
min_base_quality: 10
min_base_quality_mode: ENFORCED_BY_CLIENT

(skip...)

I1208 07:03:00.650166 140516170581760 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1208 07:03:00.666655 140516170581760 make_examples.py:1324] Preparing inputs
I1208 07:03:00.855743 140516170581760 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1208 07:03:00.888984 140516170581760 make_examples.py:1248] Common contigs are [u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'10', u'11', u'12', u'13', u'14', u'15', u'16', u'17', u'18', u'19', u'20', u'21', u'22', u'X', u'Y']
I1208 07:03:00.901866 140516170581760 make_examples.py:1330] Writing examples to /tmp/deepvariant_tmp_output/make_examples.tfrecord-00011-of-00012.gz
I1208 07:03:00.902448 140516170581760 make_examples.py:1334] Writing gvcf records to /tmp/deepvariant_tmp_output/gvcf.tfrecord-00011-of-00012.gz
I1208 07:03:00.955504 140516170581760 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2019-12-08 07:03:00.994095: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
I1208 07:03:22.274698 140516170581760 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1208 07:03:22.779856 140516170581760 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1208 07:03:32.588432 140516170581760 make_examples.py:1363] Task 11: 0 candidates (0 examples) [31.68s elapsed]
I1208 07:03:33.147475 140516170581760 make_examples.py:1380] Found 4 candidate variants
I1208 07:03:33.259099 140516170581760 make_examples.py:1381] Created 4 examples

real    0m50.885s
user    0m34.940s
sys 2m45.810s
Traceback (most recent call last):
  File "/opt/deepvariant/bin/run_deepvariant.py", line 317, in <module>
    app.run(main)
  File "/usr/local/lib/python2.7/dist-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/usr/local/lib/python2.7/dist-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/opt/deepvariant/bin/run_deepvariant.py", line 307, in main
    subprocess.check_call(command, shell=True, executable='/bin/bash')
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'time seq 0 11 | parallel -k --line-buffer /opt/deepvariant/bin/make_examples --mode calling --ref "/input/genome.fa" --reads "/input/HC3-BC_RG_bwa.bam" --examples "/tmp/deepvariant_tmp_output/make_examples.tfrecord@12.gz" --gvcf "/tmp/deepvariant_tmp_output/gvcf.tfrecord@12.gz" --regions "20:10,000,000-10,100,000" --task {}' returned non-zero exit status 3

I've actually tried several times with a small changes (like no regions) which also failed.

Could you please help me to solve this problem. Thank you in advance.

akolesnikov commented 4 years ago

Hi se2cheeese,

It looks like one of the shards failed during make_examples stage. In order to diagnose the problem we need to see logs with the failure. You may run it without shards (remove --num_shards flag). It should be pretty fast for 100K bases. Then attach the entire output so that we could look at it.

se2cheeese commented 4 years ago

Thank you akolesikov,

Yes, it works without num_shards flag.

Here is the output

$ sudo docker run \
> -v "${INPUT_DIR}":"/input" \
> -v "${OUTPUT_DIR}:/output" \
> gcr.io/deepvariant-docker/deepvariant:"${BIN_VERSION}" \
> /opt/deepvariant/bin/run_deepvariant \
> --model_type=WES \
> --ref=/input/genome.fa \
> --reads=/input/HC3-BC_RG_bwa.bam \
> --regions "20:10,000,000-10,100,000" \
> --output_vcf=/output/output.vcf.gz \
> --output_gvcf=/output/output.g.vcf.gz \

***** Running the command:*****
time seq 0 0 | parallel -k --line-buffer /opt/deepvariant/bin/make_examples --mode calling --ref "/input/genome.fa" --reads "/input/HC3-BC_RG_bwa.bam" --examples "/tmp/deepvariant_tmp_output/make_examples.tfrecord@1.gz" --gvcf "/tmp/deepvariant_tmp_output/gvcf.tfrecord@1.gz" --regions "20:10,000,000-10,100,000" --task {}

I1213 13:07:00.156721 139885760198400 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
min_base_quality: 10
min_base_quality_mode: ENFORCED_BY_CLIENT

I1213 13:07:00.265611 139885760198400 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:07:00.277568 139885760198400 make_examples.py:1324] Preparing inputs
I1213 13:07:00.354827 139885760198400 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:07:00.377454 139885760198400 make_examples.py:1248] Common contigs are [u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'10', u'11', u'12', u'13', u'14', u'15', u'16', u'17', u'18', u'19', u'20', u'21', u'22', u'X', u'Y']
I1213 13:07:00.383280 139885760198400 make_examples.py:1330] Writing examples to /tmp/deepvariant_tmp_output/make_examples.tfrecord-00000-of-00001.gz
I1213 13:07:00.383569 139885760198400 make_examples.py:1334] Writing gvcf records to /tmp/deepvariant_tmp_output/gvcf.tfrecord-00000-of-00001.gz
I1213 13:07:00.394412 139885760198400 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2019-12-13 13:07:00.397901: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
I1213 13:07:01.327224 139885760198400 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:07:01.414663 139885760198400 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:07:02.310832 139885760198400 make_examples.py:1363] Task 0: 0 candidates (0 examples) [1.93s elapsed]
I1213 13:07:05.401592 139885760198400 make_examples.py:1380] Found 28 candidate variants
I1213 13:07:05.402002 139885760198400 make_examples.py:1381] Created 28 examples

real    0m10.204s
user    0m5.490s
sys 0m3.310s

***** Running the command:*****
time /opt/deepvariant/bin/call_variants --outfile "/tmp/deepvariant_tmp_output/call_variants_output.tfrecord.gz" --examples "/tmp/deepvariant_tmp_output/make_examples.tfrecord@1.gz" --checkpoint "/opt/models/wes/model.ckpt"

I1213 13:07:08.439639 140638419556096 call_variants.py:313] Set KMP_BLOCKTIME to 0
2019-12-13 13:07:08.488881: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2697915000 Hz
2019-12-13 13:07:08.491470: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x58612b0 executing computations on platform Host. Devices:
2019-12-13 13:07:08.491562: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
2019-12-13 13:07:08.495160: I tensorflow/core/common_runtime/process_util.cc:71] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
I1213 13:07:08.525229 140638419556096 modeling.py:560] Initializing model with random parameters
W1213 13:07:08.527353 140638419556096 estimator.py:1760] Using temporary folder as model directory: /tmp/tmpr4M5u5
I1213 13:07:08.528274 140638419556096 estimator.py:201] Using config: {'_save_checkpoints_secs': 600, '_session_config': , '_keep_checkpoint_max': 100000, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fe88a5b6190>, '_model_dir': '/tmp/tmpr4M5u5', '_protocol': None, '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 0, '_tf_random_seed': None, '_save_summary_steps': 100, '_device_fn': None, '_experimental_distribute': None, '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_evaluation_master': '', '_eval_distribute': None, '_global_id_in_cluster': 0, '_master': ''}
I1213 13:07:08.528713 140638419556096 call_variants.py:381] Writing calls to /tmp/deepvariant_tmp_output/call_variants_output.tfrecord.gz
W1213 13:07:08.533111 140638419556096 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
I1213 13:07:08.550576 140638419556096 data_providers.py:367] self.input_read_threads=8
W1213 13:07:08.550924 140638419556096 deprecation.py:323] From /tmp/Bazel.runfiles_nYYnpl/runfiles/com_google_deepvariant/deepvariant/data_providers.py:372: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.experimental.parallel_interleave(...)`.
I1213 13:07:08.567331 140638419556096 data_providers.py:373] self.input_map_threads=48
I1213 13:07:08.591927 140638419556096 estimator.py:1111] Calling model_fn.
W1213 13:07:08.592129 140638419556096 deprecation.py:323] From /tmp/Bazel.runfiles_nYYnpl/runfiles/com_google_deepvariant/deepvariant/modeling.py:880: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1213 13:07:08.594911 140638419556096 deprecation.py:323] From /tmp/Bazel.runfiles_nYYnpl/runfiles/com_google_deepvariant/deepvariant/modeling.py:882: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
I1213 13:07:11.932379 140638419556096 estimator.py:1113] Done calling model_fn.
I1213 13:07:13.617564 140638419556096 monitored_session.py:222] Graph was finalized.
W1213 13:07:13.618191 140638419556096 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
I1213 13:07:13.621948 140638419556096 saver.py:1270] Restoring parameters from /opt/models/wes/model.ckpt
I1213 13:07:15.663943 140638419556096 session_manager.py:491] Running local_init_op.
I1213 13:07:15.711944 140638419556096 session_manager.py:493] Done running local_init_op.
I1213 13:07:16.176234 140638419556096 modeling.py:410] Reloading EMA...
I1213 13:07:16.177736 140638419556096 saver.py:1270] Restoring parameters from /opt/models/wes/model.ckpt
I1213 13:07:20.278557 140638419556096 call_variants.py:399] Processed 1 examples in 1 batches [1174.939 sec per 100]
I1213 13:07:20.328917 140638419556096 call_variants.py:401] Done evaluating variants

WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.

real    0m15.024s
user    0m13.890s
sys 0m3.410s

***** Running the command:*****
time /opt/deepvariant/bin/postprocess_variants --ref "/input/genome.fa" --infile "/tmp/deepvariant_tmp_output/call_variants_output.tfrecord.gz" --outfile "/output/output.vcf.gz" --nonvariant_site_tfrecord_path "/tmp/deepvariant_tmp_output/gvcf.tfrecord@1.gz" --gvcf_outfile "/output/output.g.vcf.gz"

2019-12-13 13:07:22.565874: I deepvariant/postprocess_variants.cc:88] Read from: /tmp/deepvariant_tmp_output/call_variants_output.tfrecord.gz
2019-12-13 13:07:22.566377: I deepvariant/postprocess_variants.cc:97] Done reading: /tmp/deepvariant_tmp_output/call_variants_output.tfrecord.gz. #entries in single_site_calls = 28
2019-12-13 13:07:22.566443: I deepvariant/postprocess_variants.cc:101] Total #entries in single_site_calls = 28
2019-12-13 13:07:22.566459: I deepvariant/postprocess_variants.cc:103] Start SortSingleSiteCalls
2019-12-13 13:07:22.566492: I deepvariant/postprocess_variants.cc:105] Done SortSingleSiteCalls
I1213 13:07:22.566857 139950873712384 postprocess_variants.py:972] CVO sorting took 1.82509422302e-05 minutes
I1213 13:07:22.567840 139950873712384 postprocess_variants.py:974] Transforming call_variants_output to variants.
I1213 13:07:22.568222 139950873712384 postprocess_variants.py:1000] Merging and writing variants to VCF and gVCF.
I1213 13:07:22.570799 139950873712384 genomics_writer.py:172] Writing /output/output.vcf.gz with NativeVcfWriter
I1213 13:07:22.573292 139950873712384 genomics_writer.py:172] Writing /output/output.g.vcf.gz with NativeVcfWriter
I1213 13:07:22.661263 139950873712384 postprocess_variants.py:1020] Finished writing VCF and gVCF in 0.00155058304469 minutes.
I1213 13:07:22.678944 139950873712384 genomics_reader.py:223] Reading /output/output.vcf.gz with NativeVcfReader

real    0m2.697s
user    0m2.250s
sys 0m1.410s

When it didn't work with num_shards flag, the entire output was like this

$ sudo docker run \
> -v "${INPUT_DIR}":"/input" \
> -v "${OUTPUT_DIR}:/output" \
> gcr.io/deepvariant-docker/deepvariant:"${BIN_VERSION}" \
> /opt/deepvariant/bin/run_deepvariant \
> --model_type=WES \
> --ref=/input/genome.fa \
> --reads=/input/HC3-BC_RG_bwa.bam \
> --regions "20:10,000,000-10,100,000" \
> --output_vcf=/output/output.vcf.gz \
> --output_gvcf=/output/output.g.vcf.gz \
> --num_shards=10

***** Running the command:*****
time seq 0 9 | parallel -k --line-buffer /opt/deepvariant/bin/make_examples --mode calling --ref "/input/genome.fa" --reads "/input/HC3-BC_RG_bwa.bam" --examples "/tmp/deepvariant_tmp_output/make_examples.tfrecord@10.gz" --gvcf "/tmp/deepvariant_tmp_output/gvcf.tfrecord@10.gz" --regions "20:10,000,000-10,100,000" --task {}

I1213 13:11:54.227790 140490637805312 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
min_base_quality: 10
min_base_quality_mode: ENFORCED_BY_CLIENT

I1213 13:11:54.379641 140490637805312 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.392796 140490637805312 make_examples.py:1324] Preparing inputs
I1213 13:11:54.896145 140490637805312 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.944020 140490637805312 make_examples.py:1248] Common contigs are [u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'10', u'11', u'12', u'13', u'14', u'15', u'16', u'17', u'18', u'19', u'20', u'21', u'22', u'X', u'Y']
I1213 13:11:54.960778 140490637805312 make_examples.py:1330] Writing examples to /tmp/deepvariant_tmp_output/make_examples.tfrecord-00000-of-00010.gz
I1213 13:11:54.961548 140490637805312 make_examples.py:1334] Writing gvcf records to /tmp/deepvariant_tmp_output/gvcf.tfrecord-00000-of-00010.gz
I1213 13:11:55.068834 140490637805312 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2019-12-13 13:11:55.109412: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
I1213 13:12:03.427617 140490637805312 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:12:04.700349 140490637805312 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:12:12.598862 140490637805312 make_examples.py:1363] Task 0: 0 candidates (0 examples) [17.64s elapsed]
I1213 13:12:13.068409 140490637805312 make_examples.py:1380] Found 1 candidate variants
I1213 13:12:13.223464 140490637805312 make_examples.py:1381] Created 1 examples
I1213 13:11:54.226648 140402484717312 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
min_base_quality: 10
min_base_quality_mode: ENFORCED_BY_CLIENT

I1213 13:11:54.767545 140402484717312 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.788374 140402484717312 make_examples.py:1324] Preparing inputs
I1213 13:11:54.979619 140402484717312 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:55.010253 140402484717312 make_examples.py:1248] Common contigs are [u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'10', u'11', u'12', u'13', u'14', u'15', u'16', u'17', u'18', u'19', u'20', u'21', u'22', u'X', u'Y']
I1213 13:11:55.017723 140402484717312 make_examples.py:1330] Writing examples to /tmp/deepvariant_tmp_output/make_examples.tfrecord-00001-of-00010.gz
I1213 13:11:55.018065 140402484717312 make_examples.py:1334] Writing gvcf records to /tmp/deepvariant_tmp_output/gvcf.tfrecord-00001-of-00010.gz
I1213 13:11:55.107007 140402484717312 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2019-12-13 13:11:55.127048: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
I1213 13:12:03.412683 140402484717312 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:12:04.702708 140402484717312 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:12:12.676522 140402484717312 make_examples.py:1363] Task 1: 4 candidates (4 examples) [17.66s elapsed]
I1213 13:12:13.028630 140402484717312 make_examples.py:1380] Found 7 candidate variants
I1213 13:12:13.211899 140402484717312 make_examples.py:1381] Created 7 examples
I1213 13:11:54.225912 139798917543680 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
min_base_quality: 10
min_base_quality_mode: ENFORCED_BY_CLIENT

I1213 13:11:54.380671 139798917543680 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.394714 139798917543680 make_examples.py:1324] Preparing inputs
I1213 13:11:54.895178 139798917543680 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.923032 139798917543680 make_examples.py:1248] Common contigs are [u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'10', u'11', u'12', u'13', u'14', u'15', u'16', u'17', u'18', u'19', u'20', u'21', u'22', u'X', u'Y']
I1213 13:11:54.930289 139798917543680 make_examples.py:1330] Writing examples to /tmp/deepvariant_tmp_output/make_examples.tfrecord-00002-of-00010.gz
I1213 13:11:54.930751 139798917543680 make_examples.py:1334] Writing gvcf records to /tmp/deepvariant_tmp_output/gvcf.tfrecord-00002-of-00010.gz
I1213 13:11:54.940401 139798917543680 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2019-12-13 13:11:54.943675: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
I1213 13:12:03.334026 139798917543680 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:12:04.708316 139798917543680 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:12:12.659555 139798917543680 make_examples.py:1363] Task 2: 3 candidates (3 examples) [17.73s elapsed]
I1213 13:12:13.045805 139798917543680 make_examples.py:1380] Found 7 candidate variants
I1213 13:12:13.209331 139798917543680 make_examples.py:1381] Created 7 examples
I1213 13:11:54.227793 140236020647680 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
min_base_quality: 10
min_base_quality_mode: ENFORCED_BY_CLIENT

I1213 13:11:54.403919 140236020647680 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.414350 140236020647680 make_examples.py:1324] Preparing inputs
I1213 13:11:54.893651 140236020647680 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.920942 140236020647680 make_examples.py:1248] Common contigs are [u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'10', u'11', u'12', u'13', u'14', u'15', u'16', u'17', u'18', u'19', u'20', u'21', u'22', u'X', u'Y']
I1213 13:11:54.928174 140236020647680 make_examples.py:1330] Writing examples to /tmp/deepvariant_tmp_output/make_examples.tfrecord-00003-of-00010.gz
I1213 13:11:54.928524 140236020647680 make_examples.py:1334] Writing gvcf records to /tmp/deepvariant_tmp_output/gvcf.tfrecord-00003-of-00010.gz
I1213 13:11:54.940502 140236020647680 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2019-12-13 13:11:54.943561: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
I1213 13:12:03.417463 140236020647680 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:12:04.708851 140236020647680 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:12:12.610981 140236020647680 make_examples.py:1363] Task 3: 0 candidates (0 examples) [17.68s elapsed]
I1213 13:12:13.094532 140236020647680 make_examples.py:1380] Found 1 candidate variants
I1213 13:12:13.241935 140236020647680 make_examples.py:1381] Created 1 examples
I1213 13:11:54.226402 140097611171584 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
min_base_quality: 10
min_base_quality_mode: ENFORCED_BY_CLIENT

I1213 13:11:54.378241 140097611171584 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.391170 140097611171584 make_examples.py:1324] Preparing inputs
I1213 13:11:54.893971 140097611171584 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.921802 140097611171584 make_examples.py:1248] Common contigs are [u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'10', u'11', u'12', u'13', u'14', u'15', u'16', u'17', u'18', u'19', u'20', u'21', u'22', u'X', u'Y']
I1213 13:11:54.928759 140097611171584 make_examples.py:1330] Writing examples to /tmp/deepvariant_tmp_output/make_examples.tfrecord-00004-of-00010.gz
I1213 13:11:54.929138 140097611171584 make_examples.py:1334] Writing gvcf records to /tmp/deepvariant_tmp_output/gvcf.tfrecord-00004-of-00010.gz
I1213 13:11:54.940500 140097611171584 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2019-12-13 13:11:54.943644: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
I1213 13:12:03.388587 140097611171584 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:12:04.700193 140097611171584 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:12:12.595916 140097611171584 make_examples.py:1363] Task 4: 0 candidates (0 examples) [17.66s elapsed]
I1213 13:12:13.016803 140097611171584 make_examples.py:1380] Found 3 candidate variants
I1213 13:12:13.202656 140097611171584 make_examples.py:1381] Created 3 examples
I1213 13:11:54.224806 139902364522240 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
min_base_quality: 10
min_base_quality_mode: ENFORCED_BY_CLIENT

I1213 13:11:54.382869 139902364522240 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.396718 139902364522240 make_examples.py:1324] Preparing inputs
I1213 13:11:54.893652 139902364522240 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.920939 139902364522240 make_examples.py:1248] Common contigs are [u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'10', u'11', u'12', u'13', u'14', u'15', u'16', u'17', u'18', u'19', u'20', u'21', u'22', u'X', u'Y']
I1213 13:11:54.928069 139902364522240 make_examples.py:1330] Writing examples to /tmp/deepvariant_tmp_output/make_examples.tfrecord-00005-of-00010.gz
I1213 13:11:54.928409 139902364522240 make_examples.py:1334] Writing gvcf records to /tmp/deepvariant_tmp_output/gvcf.tfrecord-00005-of-00010.gz
I1213 13:11:54.940448 139902364522240 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2019-12-13 13:11:54.943632: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
I1213 13:12:03.312068 139902364522240 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:12:04.712960 139902364522240 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:12:12.605319 139902364522240 make_examples.py:1363] Task 5: 0 candidates (0 examples) [17.68s elapsed]
I1213 13:12:12.983227 139902364522240 make_examples.py:1380] Found 0 candidate variants
I1213 13:12:13.147787 139902364522240 make_examples.py:1381] Created 0 examples
I1213 13:11:54.226497 140617939887872 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
min_base_quality: 10
min_base_quality_mode: ENFORCED_BY_CLIENT

I1213 13:11:54.378638 140617939887872 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.391983 140617939887872 make_examples.py:1324] Preparing inputs
I1213 13:11:54.913211 140617939887872 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.939115 140617939887872 make_examples.py:1248] Common contigs are [u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'10', u'11', u'12', u'13', u'14', u'15', u'16', u'17', u'18', u'19', u'20', u'21', u'22', u'X', u'Y']
I1213 13:11:54.948522 140617939887872 make_examples.py:1330] Writing examples to /tmp/deepvariant_tmp_output/make_examples.tfrecord-00006-of-00010.gz
I1213 13:11:54.949028 140617939887872 make_examples.py:1334] Writing gvcf records to /tmp/deepvariant_tmp_output/gvcf.tfrecord-00006-of-00010.gz
I1213 13:11:55.068927 140617939887872 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2019-12-13 13:11:55.127010: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
I1213 13:12:03.422019 140617939887872 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:12:04.700212 140617939887872 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:12:12.628093 140617939887872 make_examples.py:1363] Task 6: 0 candidates (0 examples) [17.68s elapsed]
I1213 13:12:13.083571 140617939887872 make_examples.py:1380] Found 3 candidate variants
I1213 13:12:13.237519 140617939887872 make_examples.py:1381] Created 3 examples
I1213 13:11:54.227164 140331447654144 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
min_base_quality: 10
min_base_quality_mode: ENFORCED_BY_CLIENT

I1213 13:11:54.379436 140331447654144 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.392060 140331447654144 make_examples.py:1324] Preparing inputs
I1213 13:11:54.893769 140331447654144 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.921667 140331447654144 make_examples.py:1248] Common contigs are [u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'10', u'11', u'12', u'13', u'14', u'15', u'16', u'17', u'18', u'19', u'20', u'21', u'22', u'X', u'Y']
I1213 13:11:54.928567 140331447654144 make_examples.py:1330] Writing examples to /tmp/deepvariant_tmp_output/make_examples.tfrecord-00007-of-00010.gz
I1213 13:11:54.928935 140331447654144 make_examples.py:1334] Writing gvcf records to /tmp/deepvariant_tmp_output/gvcf.tfrecord-00007-of-00010.gz
I1213 13:11:54.940444 140331447654144 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2019-12-13 13:11:54.943589: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
I1213 13:12:03.413748 140331447654144 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:12:04.700217 140331447654144 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:12:12.624783 140331447654144 make_examples.py:1363] Task 7: 0 candidates (0 examples) [17.69s elapsed]
I1213 13:12:13.050214 140331447654144 make_examples.py:1380] Found 3 candidate variants
I1213 13:12:13.219461 140331447654144 make_examples.py:1381] Created 3 examples
I1213 13:11:54.225383 140460245006080 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
min_base_quality: 10
min_base_quality_mode: ENFORCED_BY_CLIENT

I1213 13:11:54.378618 140460245006080 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.392364 140460245006080 make_examples.py:1324] Preparing inputs
I1213 13:11:54.905719 140460245006080 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.953198 140460245006080 make_examples.py:1248] Common contigs are [u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'10', u'11', u'12', u'13', u'14', u'15', u'16', u'17', u'18', u'19', u'20', u'21', u'22', u'X', u'Y']
I1213 13:11:54.963092 140460245006080 make_examples.py:1330] Writing examples to /tmp/deepvariant_tmp_output/make_examples.tfrecord-00008-of-00010.gz
I1213 13:11:54.963464 140460245006080 make_examples.py:1334] Writing gvcf records to /tmp/deepvariant_tmp_output/gvcf.tfrecord-00008-of-00010.gz
I1213 13:11:55.068834 140460245006080 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2019-12-13 13:11:55.090168: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
I1213 13:12:03.314467 140460245006080 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:12:04.700131 140460245006080 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:12:12.616221 140460245006080 make_examples.py:1363] Task 8: 0 candidates (0 examples) [17.65s elapsed]
I1213 13:12:13.152090 140460245006080 make_examples.py:1380] Found 2 candidate variants
I1213 13:12:13.250552 140460245006080 make_examples.py:1381] Created 2 examples
I1213 13:11:54.227202 140405993887488 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
min_base_quality: 10
min_base_quality_mode: ENFORCED_BY_CLIENT

I1213 13:11:54.386776 140405993887488 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.398395 140405993887488 make_examples.py:1324] Preparing inputs
I1213 13:11:54.894856 140405993887488 genomics_reader.py:223] Reading /input/HC3-BC_RG_bwa.bam with NativeSamReader
I1213 13:11:54.922434 140405993887488 make_examples.py:1248] Common contigs are [u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'10', u'11', u'12', u'13', u'14', u'15', u'16', u'17', u'18', u'19', u'20', u'21', u'22', u'X', u'Y']
I1213 13:11:54.929332 140405993887488 make_examples.py:1330] Writing examples to /tmp/deepvariant_tmp_output/make_examples.tfrecord-00009-of-00010.gz
I1213 13:11:54.929711 140405993887488 make_examples.py:1334] Writing gvcf records to /tmp/deepvariant_tmp_output/gvcf.tfrecord-00009-of-00010.gz
I1213 13:11:54.940470 140405993887488 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2019-12-13 13:11:54.943603: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728

real    0m33.273s
user    0m30.820s
sys 1m32.400s
Traceback (most recent call last):
  File "/opt/deepvariant/bin/run_deepvariant.py", line 317, in <module>
    app.run(main)
  File "/usr/local/lib/python2.7/dist-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/usr/local/lib/python2.7/dist-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/opt/deepvariant/bin/run_deepvariant.py", line 307, in main
    subprocess.check_call(command, shell=True, executable='/bin/bash')
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'time seq 0 9 | parallel -k --line-buffer /opt/deepvariant/bin/make_examples --mode calling --ref "/input/genome.fa" --reads "/input/HC3-BC_RG_bwa.bam" --examples "/tmp/deepvariant_tmp_output/make_examples.tfrecord@10.gz" --gvcf "/tmp/deepvariant_tmp_output/gvcf.tfrecord@10.gz" --regions "20:10,000,000-10,100,000" --task {}' returned non-zero exit status 1

The same error develops with your WES-case-study sample file/script.

It would be great to solve this problem. Thank you in advance.

se2cheeese commented 4 years ago

Dear akolesikov,

--regions "20:10,000,000-10,100,000" was ok without shards, however, for WES.bed I got a similar error as below. 

***** Running the command:*****
time /opt/deepvariant/bin/call_variants --outfile "/tmp/deepvariant_tmp_output/call_variants_output.tfrecord.gz" --examples "/tmp/deepvariant_tmp_output/make_examples.tfrecord@1.gz" --checkpoint "/opt/models/wes/model.ckpt"

I1213 19:19:36.445342 140624564107008 call_variants.py:313] Set KMP_BLOCKTIME to 0
2019-12-13 19:19:36.497919: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2697915000 Hz
2019-12-13 19:19:36.499703: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x52dd110 executing computations on platform Host. Devices:
2019-12-13 19:19:36.499773: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
2019-12-13 19:19:36.503204: I tensorflow/core/common_runtime/process_util.cc:71] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
I1213 19:19:36.524235 140624564107008 modeling.py:560] Initializing model with random parameters
W1213 19:19:36.526154 140624564107008 estimator.py:1760] Using temporary folder as model directory: /tmp/tmp6q1g_L
I1213 19:19:36.526648 140624564107008 estimator.py:201] Using config: {'_save_checkpoints_secs': 600, '_session_config': , '_keep_checkpoint_max': 100000, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fe550820190>, '_model_dir': '/tmp/tmp6q1g_L', '_protocol': None, '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 0, '_tf_random_seed': None, '_save_summary_steps': 100, '_device_fn': None, '_experimental_distribute': None, '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_evaluation_master': '', '_eval_distribute': None, '_global_id_in_cluster': 0, '_master': ''}
I1213 19:19:36.526952 140624564107008 call_variants.py:381] Writing calls to /tmp/deepvariant_tmp_output/call_variants_output.tfrecord.gz
W1213 19:19:36.531224 140624564107008 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
I1213 19:19:36.546206 140624564107008 data_providers.py:367] self.input_read_threads=8
W1213 19:19:36.546560 140624564107008 deprecation.py:323] From /tmp/Bazel.runfiles_DeisPh/runfiles/com_google_deepvariant/deepvariant/data_providers.py:372: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.experimental.parallel_interleave(...)`.
I1213 19:19:36.562076 140624564107008 data_providers.py:373] self.input_map_threads=48
I1213 19:19:36.584693 140624564107008 estimator.py:1111] Calling model_fn.
W1213 19:19:36.584897 140624564107008 deprecation.py:323] From /tmp/Bazel.runfiles_DeisPh/runfiles/com_google_deepvariant/deepvariant/modeling.py:880: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1213 19:19:36.587405 140624564107008 deprecation.py:323] From /tmp/Bazel.runfiles_DeisPh/runfiles/com_google_deepvariant/deepvariant/modeling.py:882: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
I1213 19:19:39.687763 140624564107008 estimator.py:1113] Done calling model_fn.
I1213 19:19:41.234924 140624564107008 monitored_session.py:222] Graph was finalized.
W1213 19:19:41.236058 140624564107008 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
I1213 19:19:41.240454 140624564107008 saver.py:1270] Restoring parameters from /opt/models/wes/model.ckpt
I1213 19:19:42.471465 140624564107008 session_manager.py:491] Running local_init_op.
I1213 19:19:42.514822 140624564107008 session_manager.py:493] Done running local_init_op.
I1213 19:19:42.911899 140624564107008 modeling.py:410] Reloading EMA...
I1213 19:19:42.913168 140624564107008 saver.py:1270] Restoring parameters from /opt/models/wes/model.ckpt

real    0m35.093s
user    0m23.000s
sys 2m0.390s
Traceback (most recent call last):
  File "/opt/deepvariant/bin/run_deepvariant.py", line 317, in <module>
    app.run(main)
  File "/usr/local/lib/python2.7/dist-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/usr/local/lib/python2.7/dist-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/opt/deepvariant/bin/run_deepvariant.py", line 307, in main
    subprocess.check_call(command, shell=True, executable='/bin/bash')
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'time /opt/deepvariant/bin/call_variants --outfile "/tmp/deepvariant_tmp_output/call_variants_output.tfrecord.gz" --examples "/tmp/deepvariant_tmp_output/make_examples.tfrecord@1.gz" --checkpoint "/opt/models/wes/model.ckpt"' returned non-zero exit status 247

I'm attaching the code and the entire output.

Thanks a lot. WES_20191214_samplenoshards.txt

error_output.txt

akolesnikov commented 4 years ago

I tried and was unsuccessful to reproduce the problem with the case study input data. I could try to reproduce the problem with your data if you can attach your BAM (just the 100,000 bases of it) and the reference. As a work around you could run without --num_shards flag, or you could also try to reduce the number of shards.

pichuan commented 4 years ago

Hi @se2cheeese I will close this issue for now because there's no reason update from you. But please feel free to follow up with more questions, either reopening this issue, or another one. Thank you!