Error running atacworks train

[albertdyu]

Hi there,

I'm running atacworks on some Drosophila ATAC-seq data we had generated, and it's throwing an error at me!

atacworks train \ --noisybw ClkZT14_2.sort.bam.cutsites.smoothed_100.coverage.bw \ --cleanbw ClkZT14_1.sort.bam.cutsites.smoothed_100.coverage.bw \ --cleanpeakfile ClkZT14_1.sort.bam.peaks.bed \ --genome dm6.chrom.sizes \ --val_chrom chr2R \ --holdout_chrom chr3L \ --out_home "./" \ --exp_name "856_train" \ --distributed

INFO:2022-05-22 17:50:19,291:AtacWorks-peak2bw] Reading input file INFO:2022-05-22 17:50:19,295:AtacWorks-peak2bw] Read 15265 peaks. INFO:2022-05-22 17:50:19,297:AtacWorks-peak2bw] Adding score INFO:2022-05-22 17:50:19,297:AtacWorks-peak2bw] Writing peaks to bedGraph file Discarding 0 entries outside sizes file. INFO:2022-05-22 17:50:19,335:AtacWorks-peak2bw] Writing peaks to bigWig file ./856_train_2022.05.22_17.50/bigwig_peakfiles/ClkZT14_1.sort.bam.peaks.bed.bw INFO:2022-05-22 17:50:19,364:AtacWorks-peak2bw] Done! INFO:2022-05-22 17:50:19,367:AtacWorks-intervals] Generating training intervals INFO:2022-05-22 17:50:20,831:AtacWorks-intervals] Generating val intervals INFO:2022-05-22 17:50:20,840:AtacWorks-bw2h5] Reading intervals INFO:2022-05-22 17:50:20,841:AtacWorks-bw2h5] Read 1691 intervals INFO:2022-05-22 17:50:20,841:AtacWorks-bw2h5] Selecting intervals with nonzero coverage Traceback (most recent call last): File "/home/albz/miniconda3/envs/atacworks/bin/atacworks", line 8, in sys.exit(main()) File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/site-packages/scripts/main.py", line 409, in main prefix, args.pad) File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/site-packages/atacworks/io/bw2h5.py", line 73, in bw2h5 intervals, noisybw) File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/site-packages/atacworks/io/bigwigio.py", line 106, in check_bigwig_intervals_nonzero check_bigwig_nonzero, axis=1, args=(bw,)) File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/site-packages/pandas/core/frame.py", line 6906, in apply return op.get_result() File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/site-packages/pandas/core/apply.py", line 186, in get_result return self.apply_standard() File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/site-packages/pandas/core/apply.py", line 292, in apply_standard self.apply_series_generator() File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/site-packages/pandas/core/apply.py", line 321, in apply_series_generator results[i] = self.f(v) File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/site-packages/pandas/core/apply.py", line 112, in f return func(x, *args, **kwds) File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/site-packages/atacworks/io/bigwigio.py", line 90, in check_bigwig_nonzero result = bw.values(interval[0], interval[1], interval[2]) RuntimeError: ('Invalid interval bounds!', 'occurred at index 1680')

Any thoughts on what could be going wrong?

[avantikalal]

@albertdyu, it looks like there's a problem reading the values for the 1680th training interval. If you look at the output files that are generated by this command you should be able to find an intervals/ folder which will contain a file with train in the name that should gave 1691 rows. You can look at the 1680th row in this file. It could be an issue with this chromosome name not being included in your bigwig file, or your bigwig file being produced using a genome assembly different from dm6.chrom.sizes.

[albertdyu]

That did the trick! I accidentally used a sizes file with slightly different chromosome sizes relative to the assembly I mapped to - good catch! Thank you!

I was able to successfully train a model, but now I'm having some trouble with denoising, haha...

atacworks denoise \ --noisybw ClkZT14_2.sort.bam.cutsites.smoothed_100.coverage.bw \ --genome gfachrome.sizes \ --weights_path ./856_train_latest/model_best.pth.tar \ --out_home "./" \ --exp_name "856_ZT14_2_denoise" \ --distributed \ --num_workers 0

INFO:2022-05-22 19:19:49,827:AtacWorks-intervals] Generating intervals tiling across all chromosomes in sizes file: gfachrome.sizes INFO:2022-05-22 19:19:49,841:AtacWorks-intervals] Done! INFO:2022-05-22 19:19:49,841:AtacWorks-bw2h5] Reading intervals INFO:2022-05-22 19:19:49,842:AtacWorks-bw2h5] Read 2747 intervals INFO:2022-05-22 19:19:49,842:AtacWorks-bw2h5] Writing data in 3 batches. INFO:2022-05-22 19:19:49,842:AtacWorks-bw2h5] Extracting data for each batch and writing to h5 file INFO:2022-05-22 19:19:49,842:AtacWorks-bw2h5] batch 0 of 3 INFO:2022-05-22 19:19:58,719:AtacWorks-bw2h5] Done! Saved to ./856_ZT14_2_denoise_2022.05.22_19.19/bw2h5/ClkZT14_2.sort.bam.cutsites.smoothed_100.coverage.bw.denoise.h5 INFO:2022-05-22 19:19:58,719:AtacWorks-main] Checking input files for compatibility Building model: resnet ... Loading model weights from ./856_train_latest/model_best.pth.tar... Finished loading. Finished building. Inference -------------------- [ 0/2747] Inference -------------------- [ 50/2747] Inference #------------------- [ 100/2747] Inference #------------------- [ 150/2747] Inference #------------------- [ 200/2747] Inference ##------------------ [ 250/2747] Inference ##------------------ [ 300/2747] Inference ###----------------- [ 350/2747] Inference ###----------------- [ 400/2747] Inference ###----------------- [ 450/2747] Inference ####---------------- [ 500/2747] Inference ####---------------- [ 550/2747] Inference ####---------------- [ 600/2747] Inference #####--------------- [ 650/2747] Inference #####--------------- [ 700/2747] Inference #####--------------- [ 750/2747] Inference ######-------------- [ 800/2747] Inference ######-------------- [ 850/2747] Inference #######------------- [ 900/2747] Inference #######------------- [ 950/2747] Inference #######------------- [1000/2747] Inference ########------------ [1050/2747] Inference ########------------ [1100/2747] Traceback (most recent call last): File "/home/albz/miniconda3/envs/atacworks/bin/atacworks", line 8, in sys.exit(main()) File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/site-packages/scripts/main.py", line 569, in main worker(args.gpu_idx, ngpus_per_node, args, res_queue) File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/site-packages/scripts/worker.py", line 290, in infer_worker pad=args.pad) File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/site-packages/atacworks/dl4atac/infer.py", line 80, in infer res_queue.put((idxes, batch_res)) File "", line 2, in put File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod raise convert_to_error(kind, result) multiprocessing.managers.RemoteError:

Traceback (most recent call last): File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/multiprocessing/managers.py", line 228, in serve_client request = recv() File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/multiprocessing/connection.py", line 251, in recv return _ForkingPickler.loads(buf.getbuffer()) File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 282, in rebuild_storage_fd fd = df.detach() File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach return reduction.recv_handle(conn) File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle return recvfds(s, 1)[0] File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds len(ancdata)) RuntimeError: received 0 items of ancdata

Process Process-2: Traceback (most recent call last): File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/site-packages/scripts/main.py", line 217, in writer if not res_queue.empty(): File "", line 2, in empty File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/multiprocessing/managers.py", line 756, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/multiprocessing/connection.py", line 404, in _send_bytes self._send(header + buf) File "/home/albz/miniconda3/envs/atacworks/lib/python3.6/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

It outputs a half-finished bedgraph file before going down.

I'm running:

CUDA 11.6 PyTorch 1.7.1 Python 3.6.7

Any thoughts? I appreciate your help!