search

vanheeringen-lab / ANANSE

Prediction of key transcription factors in cell fate determination using enhancer networks. See full ANANSE documentation for detailed installation instructions and usage examples.
http://anansepy.readthedocs.io
MIT License
77 stars 16 forks source link

No regions in the binding file overlap with given regions! #189

Closed liyangyang12 closed 1 year ago

liyangyang12 commented 1 year ago

Hi, I got an error when using ananse network for non-model organisms.

$ ananse network -e ananse_data/stringtie/MS.txt -g M_albus_1.0 -n 1 -o results/ananse_network/MS_network.tsv results/ananse_binding/MS/binding.h5 RY)8Q2O@A1L`J2D%S0X2%F9

$ ananse view --list-regions /data/liyangyang/ATAC-seq/core_scripts/results/ananse_binding/MS/binding.h5| head P1M `G I3X8@ HKI~JWE1I

Is the "." in the chromosome related to this error?

what should i do to solve this problem?

Thanks a lot for your time and effort!

siebrenf commented 1 year ago

Hey liyangyang12,

in ANANSE network, we connect the (putative) enhancer regions from the binding.h5 file to nearby genes from the gene annotation files. This error occurred because none of the chromosome names between the binding.h5 file and the gene annotation file overlapped!

There are several ways fix this. The easiest way would be for you to change the chromosome names in the gene annotation files to match the ones in your binding.h5 (these should be the same as the chromosome names in your genome FASTA file).

Tip: you can obtain an assembly_report with all chromosome names from NCBI. Easiest way is with genomepy install M_albus_1.0 --provider NCBI --annotation

liyangyang12 commented 1 year ago

Thank you!

The program looks normal now except for a warning.

$ananse network -e ananse_data/stringtie/MS.txt -g ../genome/GCF_001952655.1_M_albus_1.0_genomic.fa -a ../genome/Malbus_translate.bed12 -n 20 -o results/ananse_network/MS_network.tsv results/ananse_binding/MS/binding.h5

EKQ OH` VHJWI0{86M`BTI

Do I need to provide a GTF file?

Thanks a lot for your time and effort!

siebrenf commented 1 year ago

Those warnings are from a dependency. The gtf is needed to convert to gene names, but i see your TFs match the BED and expression file 100%, so that should be ok!

liyangyang12 commented 1 year ago

OK, thank you!

siebrenf commented 1 year ago

Closing this issue now. Feel free to start a new issue if you encounter another problem!

ScientistJake commented 10 months ago

@siebrenf , I'm hoping you can help.
I am running into the same issue when running ananse network using a custom genome. The genome was installed successfully using genomepy.

Here is the output:

ananse network   24hpf.binding/binding.h5   -e apoc_normalized_cpm.txt   -g apoculata_v2.0   -o 24hpf.network   --ncore 1   --columns X3 X4

2023-11-11 09:05:15 | INFO | Loading expression data
2023-11-11 09:05:15 | INFO | 0% of TFs found in both BED and expression file(s)
2023-11-11 09:05:15 | WARNING | Converting genes in expression table and BED to HGNC symbols
2023-11-11 09:05:18 | DEBUG | 87% of TFs found in the expression file(s)
2023-11-11 09:05:19 | DEBUG | 100% of TFs found in the BED file
2023-11-11 09:05:19 | INFO | 87% of TFs found in both BED and expression file(s)
2023-11-11 09:05:19 | DEBUG | Loading tf binding activity data
2023-11-11 09:05:19 | INFO | Loading binding data
2023-11-11 09:05:19 | INFO | Using all 555 TFs.
2023-11-11 09:05:19 | INFO | Using all 10599 regions.
2023-11-11 09:05:19 | ERROR | An error has been caught in function '<module>', process 'MainProcess' (35243), thread 'MainThread' (140327450965824):
Traceback (most recent call last):

> File "/usr/local/anaconda3/envs/ananse/bin/ananse", line 599, in <module>
    args.func(args)
    │    │    └ Namespace(annotation=None, binding='24hpf.binding/binding.h5', column=['X3', 'X4'], fin_expression=['apoc_normalized_cpm.txt'...
    │    └ <function network at 0x7fa018511430>
    └ Namespace(annotation=None, binding='24hpf.binding/binding.h5', column=['X3', 'X4'], fin_expression=['apoc_normalized_cpm.txt'...
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/ananse/commands/network.py", line 37, in network
    b.run_network(
    │ └ <function Network.run_network at 0x7fa00d576700>
    └ <ananse.network.Network object at 0x7fa00d58d910>
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/ananse/network.py", line 616, in run_network
    df_binding = self.aggregate_binding(
                 │    └ <function Network.aggregate_binding at 0x7fa00d576550>
                 └ <ananse.network.Network object at 0x7fa00d58d910>
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/ananse/network.py", line 366, in aggregate_binding
    raise ValueError(

ValueError: No regions in the binding file overlap with given regions! Use `ananse view --list-regions /home/warnerj/Desktop/projects/coral_embryos/atac/Warner_8168_221216A5/bwa_out/gimme/ananse/24hpf.binding/binding.h5` to inspect the regions in the file.
Closing remaining open files:/home/warnerj/Desktop/projects/coral_embryos/atac/Warner_8168_221216A5/bwa_out/gimme/ananse/24hpf.binding/binding.h5...done

The binding file looks ok:

ananse view --list-regions /home/warnerj/Desktop/projects/coral_embryos/atac/Warner_8168_221216A5/bwa_out/gimme/ananse/24hpf.binding/binding.h5 | head
INFO:matplotlib.font_manager:Failed to extract font properties from /usr/share/fonts/truetype/noto/NotoColorEmoji.ttf: In FT2Font: Can not load face (unknown file format; error code 0x2)
chromosome_10:10124338-10124826
chromosome_10:101637-102359
chromosome_10:10314850-10315598
chromosome_10:10320295-10321390
chromosome_10:10322561-10323471
chromosome_10:10328220-10329183
chromosome_10:10331442-10332375
chromosome_10:10494471-10494823
chromosome_10:10548666-10549393
chromosome_10:10549698-10550282

And these chrom names match the isntalled genome:

grep '>' ~/.local/share/genomes/apoculata_v2.0/apoculata_v2.0.fa
>chromosome_1
>chromosome_2
>chromosome_3
>chromosome_4
>chromosome_5
>chromosome_6
>chromosome_7
>chromosome_8
>chromosome_9
>chromosome_10
>chromosome_11
>chromosome_12
>chromosome_13
>chromosome_14

The annotation also looks ok:

head ~/.local/share/genomes/apoculata_v2.0/apoculata_v2.0.annotation.gtf 
chromosome_1    .       start_codon     16728400        16728402        0       -       0       gene_id "Apoc2_1.1511"; transcript_id "Apoc2_1.1511-RA"; Name "Apoc2_1.1511";
chromosome_1    .       CDS     16727641        16728402        0       -       0       gene_id "Apoc2_1.1511"; transcript_id "Apoc2_1.1511-RA"; Name "Apoc2_1.1511";
chromosome_1    .       exon    16727638        16728402        0       -       .       gene_id "Apoc2_1.1511"; transcript_id "Apoc2_1.1511-RA"; Name "Apoc2_1.1511";
chromosome_1    .       stop_codon      16727638        16727640        0       -       0       gene_id "Apoc2_1.1511"; transcript_id "Apoc2_1.1511-RA"; Name "Apoc2_1.1511";
chromosome_1    .       transcript      16727638        16728402        0       -       .       gene_id "Apoc2_1.1511"; transcript_id "Apoc2_1.1511-RA"; Name "Apoc2_1.1511";
chromosome_1    .       gene    16727638        16728402        0       -       .       gene_id "Apoc2_1.1511"; Name "Apoc2_1.1511";
chromosome_1    EVM     start_codon     938211  938213  0       -       0       gene_id "Apoc2_1.70"; transcript_id "Apoc2_1.70-RA"; Name "Apoc2_1.70";
chromosome_1    EVM     CDS     938146  938213  0       -       0       gene_id "Apoc2_1.70"; transcript_id "Apoc2_1.70-RA"; Name "Apoc2_1.70";
chromosome_1    EVM     exon    938146  938213  0       -       .       gene_id "Apoc2_1.70"; transcript_id "Apoc2_1.70-RA"; Name "Apoc2_1.70";
chromosome_1    EVM     CDS     927988  928027  0       -       1       gene_id "Apoc2_1.70"; transcript_id "Apoc2_1.70-RA"; Name "Apoc2_1.70";

The gene models match the expression file:

head apoc_normalized_cpm.txt
        X1      X2      X3      X4      X5      X6      X7      X8      X9      X10
Apoc2_10.2743-RA        0.345462964286786       0.613183548539739       0.672737278969806       0.632722846017469       0.435101777368553      0.485081733297985       0.841242318503322       0.827186351189691       0.81178130212478        0.270838853746221
Apoc2_14.6750-RA        0.0431828705358483      0.0322728183441968      0       0       0.0669387349797773      0.0606352166622482    0.0580167116209188       0       0.516588101352133       0.270838853746221
Apoc2_9.1020-RA 0.25909722321509        0.516365093507148       1.00910591845471        0.659086297934864       1.30530533210566      0.757940208278102        0.754217251071944       0.48252537152732        0.996277052607685       1.3241010627593
Apoc2_2.921-RA  1.07957176339621        0.968184550325903       0.155247064377648       0.105453807669578       0.301224307408998     0.303176083311241        0.203058490673216       0.379127077628608       0.627285551641876       0.240745647774419
Apoc2_2.1737-RA 0       0       0       0       15.5297865153083        21.6164547400915        112.436387121341        128.696409805929       95.1260089489856        124.435406693403
Apoc2_8.1836-RA 0.431828705358483       0       0.232870596566471       0.369088326843524       0.0669387349797773      0.212223258317869      0       0.0344660979662371      0       0.0601864119436046
Apoc2_2.818-RA  0       0       0       0       0       0.0303176083311241      0.290083558104594       0.172330489831186       0.368991500965809      0.120372823887209
Apoc2_2.51-RA   0       0.129091273376787       0       0.0263634519173945      0       0       0.203058490673216       0       0.295193200772647      0.090279617915407
Apoc2_9.2513-RA 38.8214006117276        36.9846498224495        41.0110995064286        41.2324387988051        13.0530533210566      13.2184772323701 3.80009461117018        3.03301662102887        1.84495750482905        2.67829533149041

I'm stumped!

siebrenf commented 10 months ago

Hey Jake, this might be an incompatibility with the newer pandas versions. I'll try to release a new version of ANANSE this week. If you wanna get started sooner, try to install pandas 1.5.

ScientistJake commented 10 months ago

@siebrenf , Thanks! That seems to have worked to get past the aggregate binding function. Unfortunately ananse network then quickly fails at the distributed step. This error still occurs when setting to 1 cpu and with using a very limited regions file. Full log below: 

INFO:matplotlib.font_manager:Failed to extract font properties from /usr/share/fonts/truetype/noto/NotoColorEmoji.ttf: In FT2Font: Can not load face (unknown file format; error code 0x2)
INFO:distributed.http.proxy:To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
INFO:distributed.scheduler:State start
INFO:distributed.scheduler:  Scheduler at:     tcp://127.0.0.1:35495
INFO:distributed.scheduler:  dashboard at:  http://127.0.0.1:8787/status
INFO:distributed.nanny:        Start Nanny at: 'tcp://127.0.0.1:33573'
INFO:matplotlib.font_manager:Failed to extract font properties from /usr/share/fonts/truetype/noto/NotoColorEmoji.ttf: In FT2Font: Can not load face (unknown file format; error code 0x2)
INFO:distributed.worker:      Start worker at:      tcp://127.0.0.1:45419
INFO:distributed.worker:         Listening to:      tcp://127.0.0.1:45419
INFO:distributed.worker:          Worker name:                          0
INFO:distributed.worker:         dashboard at:            127.0.0.1:46087
INFO:distributed.worker:Waiting to connect to:      tcp://127.0.0.1:35495
INFO:distributed.worker:-------------------------------------------------
INFO:distributed.worker:              Threads:                          2
INFO:distributed.worker:               Memory:                 125.82 GiB
INFO:distributed.worker:      Local Directory: /tmp/dask-scratch-space/worker-7zfoqiw4
INFO:distributed.worker:-------------------------------------------------
INFO:distributed.scheduler:Register worker <WorkerState 'tcp://127.0.0.1:45419', name: 0, status: init, memory: 0, processing: 0>
INFO:distributed.scheduler:Starting worker compute stream, tcp://127.0.0.1:45419
INFO:distributed.worker:        Registered to:      tcp://127.0.0.1:35495
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:40988
INFO:distributed.worker:-------------------------------------------------
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:35495
INFO:distributed.scheduler:Receive client connection: Client-bbad34d1-83bc-11ee-95b0-989096a2dafd
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:40996
2023-11-15 08:41:52 | INFO | Loading expression data
2023-11-15 08:41:52 | INFO | 0% of TFs found in both BED and expression file(s)
2023-11-15 08:41:52 | WARNING | Converting genes in expression table and BED to HGNC symbols
2023-11-15 08:41:52 | DEBUG | 86% of TFs found in the expression file(s)
2023-11-15 08:41:52 | DEBUG | 100% of TFs found in the BED file
2023-11-15 08:41:52 | INFO | 86% of TFs found in both BED and expression file(s)
2023-11-15 08:41:52 | DEBUG | Loading tf binding activity data
2023-11-15 08:41:52 | INFO | Loading binding data
2023-11-15 08:41:52 | INFO | Using all 268 TFs.
2023-11-15 08:41:52 | INFO | Using all 449 regions.
^MAggregating:   0%|          | 0/14 [00:00<?, ?contig/s]^MAggregating on chr5. Overall progress:   0%|          | 0/14 [00:00<?, ?contig/s]^MAggregating on chr5. Overall progress:   7%|▋         | 1/14 [00:02<00:31,  2.40s/contig]^MAggregating on chr10. Overall progress:   7%|▋         | 1/14 [00:02<00:31,  2.40s/contig]^MAggregating on chr10. Overall progress:  14%|█▍        | 2/14 [00:04<00:29,  2.43s/contig]^MAggregating on chr7. Overall progress:  14%|█▍        | 2/14 [00:04<00:29,  2.43s/contig] ^MAggregating on chr7. Overall progress:  21%|██▏       | 3/14 [00:07<00:26,  2.42s/contig]^MAggregating on chr13. Overall progress:  21%|██▏       | 3/14 [00:07<00:26,  2.42s/contig]^MAggregating on chr13. Overall progress:  29%|██▊       | 4/14 [00:09<00:24,  2.42s/contig]^MAggregating on chr11. Overall progress:  29%|██▊       | 4/14 [00:09<00:24,  2.42s/contig]^MAggregating on chr11. Overall progress:  36%|███▌      | 5/14 [00:12<00:21,  2.42s/contig]^MAggregating on chr3. Overall progress:  36%|███▌      | 5/14 [00:12<00:21,  2.42s/contig] ^MAggregating on chr3. Overall progress:  43%|████▎     | 6/14 [00:14<00:19,  2.41s/contig]^MAggregating on chr2. Overall progress:  43%|████▎     | 6/14 [00:14<00:19,  2.41s/contig]^MAggregating on chr2. Overall progress:  50%|█████     | 7/14 [00:16<00:16,  2.41s/contig]^MAggregating on chr12. Overall progress:  50%|█████     | 7/14 [00:16<00:16,  2.41s/contig]^MAggregating on chr12. Overall progress:  57%|█████▋    | 8/14 [00:19<00:14,  2.41s/contig]^MAggregating on chr14. Overall progress:  57%|█████▋    | 8/14 [00:19<00:14,  2.41s/contig]^MAggregating on chr14. Overall progress:  64%|██████▍   | 9/14 [00:21<00:12,  2.41s/contig]^MAggregating on chr6. Overall progress:  64%|██████▍   | 9/14 [00:21<00:12,  2.41s/contig] ^MAggregating on chr6. Overall progress:  71%|███████▏  | 10/14 [00:24<00:09,  2.41s/contig]^MAggregating on chr1. Overall progress:  71%|███████▏  | 10/14 [00:24<00:09,  2.41s/contig]^MAggregating on chr1. Overall progress:  79%|███████▊  | 11/14 [00:26<00:07,  2.44s/contig]^MAggregating on chr9. Overall progress:  79%|███████▊  | 11/14 [00:26<00:07,  2.44s/contig]^MAggregating on chr9. Overall progress:  86%|████████▌ | 12/14 [00:29<00:04,  2.44s/contig]^MAggregating on chr8. Overall progress:  86%|████████▌ | 12/14 [00:29<00:04,  2.44s/contig]^MAggregating on chr8. Overall progress:  93%|█████████▎| 13/14 [00:31<00:02,  2.43s/contig]^MAggregating on chr4. Overall progress:  93%|█████████▎| 13/14 [00:31<00:02,  2.43s/contig]^MAggregating on chr4. Overall progress: 100%|██████████| 14/14 [00:33<00:00,  2.41s/contig]^MAggregating on chr4. Overall progress: 100%|██████████| 14/14 [00:33<00:00,  2.42s/contig]ERROR:distributed.core:Exception while handling op shuffle_receive
Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 604, in shuffle_receive
    await shuffle.receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 481, in receive
    await self._receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 496, in _receive
    groups = await self.offload(self._repartition_buffers, filtered)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 134, in offload
    return await asyncio.get_running_loop().run_in_executor(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 505, in _repartition_buffers
    groups = split_by_partition(table, self.column)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 996, in split_by_partition
    assert len(partitions) == len(shards)
AssertionError
ERROR:distributed.core:Exception while handling op shuffle_receive
Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 604, in shuffle_receive
    await shuffle.receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 481, in receive
    await self._receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 496, in _receive
    groups = await self.offload(self._repartition_buffers, filtered)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 134, in offload
    return await asyncio.get_running_loop().run_in_executor(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 505, in _repartition_buffers
    groups = split_by_partition(table, self.column)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 996, in split_by_partition
    assert len(partitions) == len(shards)
AssertionError
ERROR:distributed.core:Exception while handling op shuffle_receive
Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 604, in shuffle_receive
    await shuffle.receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 481, in receive
    await self._receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 496, in _receive
    groups = await self.offload(self._repartition_buffers, filtered)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 134, in offload
    return await asyncio.get_running_loop().run_in_executor(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 505, in _repartition_buffers
    groups = split_by_partition(table, self.column)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 996, in split_by_partition
    assert len(partitions) == len(shards)
AssertionError
ERROR:distributed.shuffle._comms:
Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_comms.py", line 71, in _process
    await self.send(address, shards)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 125, in send
    return await self.rpc(address).shuffle_receive(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 1359, in send_recv_from_rpc
    return await send_recv(comm=comm, op=key, **kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 1143, in send_recv
    raise exc.with_traceback(tb)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 604, in shuffle_receive
    await shuffle.receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 481, in receive
    await self._receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 496, in _receive
    groups = await self.offload(self._repartition_buffers, filtered)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 134, in offload
    return await asyncio.get_running_loop().run_in_executor(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 505, in _repartition_buffers
    groups = split_by_partition(table, self.column)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 996, in split_by_partition
    assert len(partitions) == len(shards)
AssertionError
ERROR:distributed.shuffle._comms:
Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_comms.py", line 71, in _process
    await self.send(address, shards)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 125, in send
    return await self.rpc(address).shuffle_receive(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 1359, in send_recv_from_rpc
    return await send_recv(comm=comm, op=key, **kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 1143, in send_recv
    raise exc.with_traceback(tb)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 604, in shuffle_receive
    await shuffle.receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 481, in receive
    await self._receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 496, in _receive
    groups = await self.offload(self._repartition_buffers, filtered)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 134, in offload
    return await asyncio.get_running_loop().run_in_executor(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 505, in _repartition_buffers
    groups = split_by_partition(table, self.column)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 996, in split_by_partition
    assert len(partitions) == len(shards)
AssertionError
ERROR:distributed.core:Exception while handling op shuffle_receive
Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 603, in shuffle_receive
    shuffle = await self._get_shuffle_run(shuffle_id, run_id)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 704, in _get_shuffle_run
    raise shuffle._exception
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 604, in shuffle_receive
    await shuffle.receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 481, in receive
    await self._receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 496, in _receive
    groups = await self.offload(self._repartition_buffers, filtered)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 134, in offload
    return await asyncio.get_running_loop().run_in_executor(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 505, in _repartition_buffers
    groups = split_by_partition(table, self.column)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 996, in split_by_partition
    assert len(partitions) == len(shards)
    ERROR:distributed.shuffle._comms:
Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_comms.py", line 71, in _process
    await self.send(address, shards)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 125, in send
    return await self.rpc(address).shuffle_receive(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 1359, in send_recv_from_rpc
    return await send_recv(comm=comm, op=key, **kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 1143, in send_recv
    raise exc.with_traceback(tb)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 603, in shuffle_receive
    shuffle = await self._get_shuffle_run(shuffle_id, run_id)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 704, in _get_shuffle_run
    raise shuffle._exception
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 604, in shuffle_receive
    await shuffle.receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 481, in receive
    await self._receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 496, in _receive
    groups = await self.offload(self._repartition_buffers, filtered)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 134, in offload
    return await asyncio.get_running_loop().run_in_executor(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 505, in _repartition_buffers
    groups = split_by_partition(table, self.column)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 996, in split_by_partition
    assert len(partitions) == len(shards)
AssertionError
WARNING:distributed.worker:Compute Failed
Key:       ('shuffle-transfer-c6ba947c7a1bdc00ffa17247dc6db342', 11)
Function:  shuffle_transfer
args:      (                       tf_target  weighted_binding  _partitions
0     Apoc2_12.2308—Apoc2_7.1046          1.962226            5
1     Apoc2_12.2308—Apoc2_7.1047          1.894814            5
2     Apoc2_12.2308—Apoc2_7.1050          0.207537            5
3     Apoc2_12.2308—Apoc2_7.1255          1.299553            5
4     Apoc2_12.2308—Apoc2_7.1256          1.318022            5
...                          ...               ...          ...
6427    Apoc2_12.1393—Apoc2_7.65          2.000000            3
6428   Apoc2_12.1393—Apoc2_7.681          3.958551            3
6429   Apoc2_12.1393—Apoc2_7.711          1.000000            3
6430   Apoc2_12.1393—Apoc2_7.724          0.119455            3
6431   Apoc2_12.1393—Apoc2_7.729          2.000000            3

[6432 rows x 3 columns], 'c6ba947c7a1bdc00ffa17247dc6db342', 11, 14, '_partitions', {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13})
kwargs:    {}
Exception: "RuntimeError('shuffle_transfer failed during shuffle c6ba947c7a1bdc00ffa17247dc6db342')"

WARNING:distributed.worker:Compute Failed
Key:       ('shuffle-transfer-c6ba947c7a1bdc00ffa17247dc6db342', 12)
Function:  shuffle_transfer
args:      (                       tf_target  weighted_binding  _partitions
0     Apoc2_12.2308—Apoc2_8.1117          2.006809            5
1     Apoc2_12.2308—Apoc2_8.1124          1.011392            5
2     Apoc2_12.2308—Apoc2_8.1613          1.000000            5
3     Apoc2_12.2308—Apoc2_8.2474          1.000000            5
4     Apoc2_12.2308—Apoc2_8.2598          3.995726            5
...                          ...               ...          ...
2943  Apoc2_12.1393—Apoc2_8.2655          1.000000            3
2944  Apoc2_12.1393—Apoc2_8.2658          0.002624            3
2945   Apoc2_12.1393—Apoc2_8.502          1.000000            3
2946   Apoc2_12.1393—Apoc2_8.504          0.245149            3
2947   Apoc2_12.1393—Apoc2_8.791          1.000000            3

[2948 rows x 3 columns], 'c6ba947c7a1bdc00ffa17247dc6db342', 12, 14, '_partitions', {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13})
kwargs:    {}
Exception: "RuntimeError('shuffle_transfer failed during shuffle c6ba947c7a1bdc00ffa17247dc6db342')"

2023-11-15 08:42:27 | ERROR | An error has been caught in function '<module>', process 'MainProcess' (5552), thread 'MainThread' (139883518396224):
Traceback (most recent call last):

  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_shuffle.py", line 62, in shuffle_transfer
    return _get_worker_extension().add_partition(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 647, in add_partition
    shuffle = self.get_or_create_shuffle(shuffle_id, type=type, **kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 903, in get_or_create_shuffle
    return sync(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/utils.py", line 418, in sync
    raise exc.with_traceback(tb)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/utils.py", line 391, in f
    result = yield future
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/tornado/gen.py", line 767, in run
    value = future.result()
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 739, in _get_or_create_shuffle
    raise shuffle._exception
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 603, in shuffle_receive
    shuffle = await self._get_shuffle_run(shuffle_id, run_id)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 704, in _get_shuffle_run
    raise shuffle._exception
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 604, in shuffle_receive
    await shuffle.receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 481, in receive
    await self._receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 496, in _receive
    groups = await self.offload(self._repartition_buffers, filtered)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 134, in offload
    return await asyncio.get_running_loop().run_in_executor(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 505, in _repartition_buffers
    groups = split_by_partition(table, self.column)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 996, in split_by_partition
    assert len(partitions) == len(shards)

AssertionError: assert len(partitions) == len(shards)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

> File "/usr/local/anaconda3/envs/ananse/bin/ananse", line 599, in <module>
    args.func(args)
    │    │    └ Namespace(annotation=None, binding='24hpf.binding/binding.h5', column=['X3', 'X4'], fin_expression=['../apoc_normalized_cpm.t...
    │    └ <function network at 0x7f38b15aaaf0>
    └ Namespace(annotation=None, binding='24hpf.binding/binding.h5', column=['X3', 'X4'], fin_expression=['../apoc_normalized_cpm.t...
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/ananse/commands/network.py", line 37, in network
    b.run_network(
    │ └ <function Network.run_network at 0x7f38b0eda160>
    └ <ananse.network.Network object at 0x7f38b0db4eb0>
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/ananse/network.py", line 616, in run_network
    df_binding = self.aggregate_binding(
                 │    └ <function Network.aggregate_binding at 0x7f38b0edff70>
                 └ <ananse.network.Network object at 0x7f38b0db4eb0>
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/ananse/network.py", line 462, in aggregate_binding
    df = ddf.compute()
         │   └ <function DaskMethodsMixin.compute at 0x7f38bc6ca0d0>
         └ Dask DataFrame Structure:
                                      weighted_binding
           npartitions=14                             
           Apoc2_1.110...
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/dask/base.py", line 314, in compute
    (result,) = compute(self, traverse=False, **kwargs)
                │       │                       └ {}
                │       └ Dask DataFrame Structure:
                │                                    weighted_binding
                │         npartitions=14                             
                │         Apoc2_1.110...
                └ <function compute at 0x7f38bc6ca550>
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/dask/base.py", line 599, in compute
    results = schedule(dsk, keys, **kwargs)
              │        │    │       └ {}
              │        │    └ [[('sort_index-bfadc84d21345053f479d29502b5412f', 0), ('sort_index-bfadc84d21345053f479d29502b5412f', 1), ('sort_index-bfadc8...
              │        └ HighLevelGraph with 3 layers.
              │          <dask.highlevelgraph.HighLevelGraph object at 0x7f388e75feb0>
              │           0. assign-f2468bb5a8c70a7ec9c1e7...
              └ <bound method Client.get of <Client: 'tcp://127.0.0.1:35495' processes=1 threads=2, memory=125.82 GiB>>
              File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/dask/base.py", line 314, in compute
    (result,) = compute(self, traverse=False, **kwargs)
                │       │                       └ {}
                │       └ Dask DataFrame Structure:
                │                                    weighted_binding
                │         npartitions=14                             
                │         Apoc2_1.110...
                └ <function compute at 0x7f38bc6ca550>
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/dask/base.py", line 599, in compute
    results = schedule(dsk, keys, **kwargs)
              │        │    │       └ {}
              │        │    └ [[('sort_index-bfadc84d21345053f479d29502b5412f', 0), ('sort_index-bfadc84d21345053f479d29502b5412f', 1), ('sort_index-bfadc8...
              │        └ HighLevelGraph with 3 layers.
              │          <dask.highlevelgraph.HighLevelGraph object at 0x7f388e75feb0>
              │           0. assign-f2468bb5a8c70a7ec9c1e7...
              └ <bound method Client.get of <Client: 'tcp://127.0.0.1:35495' processes=1 threads=2, memory=125.82 GiB>>
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/client.py", line 3226, in get
    results = self.gather(packed, asynchronous=asynchronous, direct=direct)
              │    │      │                    │                    └ None
              │    │      │                    └ None
              │    │      └ [[<Future: cancelled, key: ('sort_index-bfadc84d21345053f479d29502b5412f', 0)>, <Future: cancelled, key: ('sort_index-bfadc84...
              │    └ <function Client.gather at 0x7f38b16e5700>
              └ <Client: 'tcp://127.0.0.1:35495' processes=1 threads=2, memory=125.82 GiB>
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/client.py", line 2361, in gather
    return self.sync(
           │    └ <function SyncMethodMixin.sync at 0x7f38bc41f8b0>
           └ <Client: 'tcp://127.0.0.1:35495' processes=1 threads=2, memory=125.82 GiB>
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/utils.py", line 351, in sync
    return sync(
           └ <function sync at 0x7f38bc41f940>
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/utils.py", line 418, in sync
    raise exc.with_traceback(tb)
          │   │              └ <traceback object at 0x7f388e2ba0c0>
          │   └ <method 'with_traceback' of 'BaseException' objects>
          └ RuntimeError('shuffle_transfer failed during shuffle c6ba947c7a1bdc00ffa17247dc6db342')
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/utils.py", line 391, in f
    result = yield future
                   └ <Task finished name='Task-2902' coro=<Client._gather() done, defined at /usr/local/anaconda3/envs/ananse/lib/python3.8/site-p...
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/tornado/gen.py", line 767, in run
    value = future.result()
            └ None
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/client.py", line 2224, in _gather
    raise exception.with_traceback(traceback)
          │         │              └ <traceback object at 0x7f388e2ac140>
          │         └ <method 'with_traceback' of 'BaseException' objects>
          └ RuntimeError('shuffle_transfer failed during shuffle c6ba947c7a1bdc00ffa17247dc6db342')
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_shuffle.py", line 72, in shuffle_transfer
    raise RuntimeError(f"shuffle_transfer failed during shuffle {id}") from e

RuntimeError: shuffle_transfer failed during shuffle c6ba947c7a1bdc00ffa17247dc6db342
INFO:distributed.nanny:Closing Nanny at 'tcp://127.0.0.1:33573'. Reason: nanny-close
INFO:distributed.nanny:Nanny asking worker to close. Reason: nanny-close
siebrenf commented 10 months ago

I've also pinned dask to version 2023.9.1 in the next version of ANANSE (which should become available about an hour or so after this PR is merged!)

ScientistJake commented 10 months ago

@siebrenf , I reinstalled using conda and everything works as the environment builds with pandas 1.5.3. The installation I was using above was with pip install git+https://github.com/vanheeringen-lab/ANANSE. Thanks again for looking into this!

StevenBai97 commented 2 months ago

@siebrenf , Thanks! That seems to have worked to get past the aggregate binding function. Unfortunately ananse network then quickly fails at the distributed step. This error still occurs when setting to 1 cpu and with using a very limited regions file. Full log below:

INFO:matplotlib.font_manager:Failed to extract font properties from /usr/share/fonts/truetype/noto/NotoColorEmoji.ttf: In FT2Font: Can not load face (unknown file format; error code 0x2)
INFO:distributed.http.proxy:To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
INFO:distributed.scheduler:State start
INFO:distributed.scheduler:  Scheduler at:     tcp://127.0.0.1:35495
INFO:distributed.scheduler:  dashboard at:  http://127.0.0.1:8787/status
INFO:distributed.nanny:        Start Nanny at: 'tcp://127.0.0.1:33573'
INFO:matplotlib.font_manager:Failed to extract font properties from /usr/share/fonts/truetype/noto/NotoColorEmoji.ttf: In FT2Font: Can not load face (unknown file format; error code 0x2)
INFO:distributed.worker:      Start worker at:      tcp://127.0.0.1:45419
INFO:distributed.worker:         Listening to:      tcp://127.0.0.1:45419
INFO:distributed.worker:          Worker name:                          0
INFO:distributed.worker:         dashboard at:            127.0.0.1:46087
INFO:distributed.worker:Waiting to connect to:      tcp://127.0.0.1:35495
INFO:distributed.worker:-------------------------------------------------
INFO:distributed.worker:              Threads:                          2
INFO:distributed.worker:               Memory:                 125.82 GiB
INFO:distributed.worker:      Local Directory: /tmp/dask-scratch-space/worker-7zfoqiw4
INFO:distributed.worker:-------------------------------------------------
INFO:distributed.scheduler:Register worker <WorkerState 'tcp://127.0.0.1:45419', name: 0, status: init, memory: 0, processing: 0>
INFO:distributed.scheduler:Starting worker compute stream, tcp://127.0.0.1:45419
INFO:distributed.worker:        Registered to:      tcp://127.0.0.1:35495
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:40988
INFO:distributed.worker:-------------------------------------------------
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:35495
INFO:distributed.scheduler:Receive client connection: Client-bbad34d1-83bc-11ee-95b0-989096a2dafd
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:40996
2023-11-15 08:41:52 | INFO | Loading expression data
2023-11-15 08:41:52 | INFO | 0% of TFs found in both BED and expression file(s)
2023-11-15 08:41:52 | WARNING | Converting genes in expression table and BED to HGNC symbols
2023-11-15 08:41:52 | DEBUG | 86% of TFs found in the expression file(s)
2023-11-15 08:41:52 | DEBUG | 100% of TFs found in the BED file
2023-11-15 08:41:52 | INFO | 86% of TFs found in both BED and expression file(s)
2023-11-15 08:41:52 | DEBUG | Loading tf binding activity data
2023-11-15 08:41:52 | INFO | Loading binding data
2023-11-15 08:41:52 | INFO | Using all 268 TFs.
2023-11-15 08:41:52 | INFO | Using all 449 regions.
^MAggregating:   0%|          | 0/14 [00:00<?, ?contig/s]^MAggregating on chr5. Overall progress:   0%|          | 0/14 [00:00<?, ?contig/s]^MAggregating on chr5. Overall progress:   7%|▋         | 1/14 [00:02<00:31,  2.40s/contig]^MAggregating on chr10. Overall progress:   7%|▋         | 1/14 [00:02<00:31,  2.40s/contig]^MAggregating on chr10. Overall progress:  14%|█▍        | 2/14 [00:04<00:29,  2.43s/contig]^MAggregating on chr7. Overall progress:  14%|█▍        | 2/14 [00:04<00:29,  2.43s/contig] ^MAggregating on chr7. Overall progress:  21%|██▏       | 3/14 [00:07<00:26,  2.42s/contig]^MAggregating on chr13. Overall progress:  21%|██▏       | 3/14 [00:07<00:26,  2.42s/contig]^MAggregating on chr13. Overall progress:  29%|██▊       | 4/14 [00:09<00:24,  2.42s/contig]^MAggregating on chr11. Overall progress:  29%|██▊       | 4/14 [00:09<00:24,  2.42s/contig]^MAggregating on chr11. Overall progress:  36%|███▌      | 5/14 [00:12<00:21,  2.42s/contig]^MAggregating on chr3. Overall progress:  36%|███▌      | 5/14 [00:12<00:21,  2.42s/contig] ^MAggregating on chr3. Overall progress:  43%|████▎     | 6/14 [00:14<00:19,  2.41s/contig]^MAggregating on chr2. Overall progress:  43%|████▎     | 6/14 [00:14<00:19,  2.41s/contig]^MAggregating on chr2. Overall progress:  50%|█████     | 7/14 [00:16<00:16,  2.41s/contig]^MAggregating on chr12. Overall progress:  50%|█████     | 7/14 [00:16<00:16,  2.41s/contig]^MAggregating on chr12. Overall progress:  57%|█████▋    | 8/14 [00:19<00:14,  2.41s/contig]^MAggregating on chr14. Overall progress:  57%|█████▋    | 8/14 [00:19<00:14,  2.41s/contig]^MAggregating on chr14. Overall progress:  64%|██████▍   | 9/14 [00:21<00:12,  2.41s/contig]^MAggregating on chr6. Overall progress:  64%|██████▍   | 9/14 [00:21<00:12,  2.41s/contig] ^MAggregating on chr6. Overall progress:  71%|███████▏  | 10/14 [00:24<00:09,  2.41s/contig]^MAggregating on chr1. Overall progress:  71%|███████▏  | 10/14 [00:24<00:09,  2.41s/contig]^MAggregating on chr1. Overall progress:  79%|███████▊  | 11/14 [00:26<00:07,  2.44s/contig]^MAggregating on chr9. Overall progress:  79%|███████▊  | 11/14 [00:26<00:07,  2.44s/contig]^MAggregating on chr9. Overall progress:  86%|████████▌ | 12/14 [00:29<00:04,  2.44s/contig]^MAggregating on chr8. Overall progress:  86%|████████▌ | 12/14 [00:29<00:04,  2.44s/contig]^MAggregating on chr8. Overall progress:  93%|█████████▎| 13/14 [00:31<00:02,  2.43s/contig]^MAggregating on chr4. Overall progress:  93%|█████████▎| 13/14 [00:31<00:02,  2.43s/contig]^MAggregating on chr4. Overall progress: 100%|██████████| 14/14 [00:33<00:00,  2.41s/contig]^MAggregating on chr4. Overall progress: 100%|██████████| 14/14 [00:33<00:00,  2.42s/contig]ERROR:distributed.core:Exception while handling op shuffle_receive
Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 604, in shuffle_receive
    await shuffle.receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 481, in receive
    await self._receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 496, in _receive
    groups = await self.offload(self._repartition_buffers, filtered)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 134, in offload
    return await asyncio.get_running_loop().run_in_executor(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 505, in _repartition_buffers
    groups = split_by_partition(table, self.column)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 996, in split_by_partition
    assert len(partitions) == len(shards)
AssertionError
ERROR:distributed.core:Exception while handling op shuffle_receive
Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 604, in shuffle_receive
    await shuffle.receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 481, in receive
    await self._receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 496, in _receive
    groups = await self.offload(self._repartition_buffers, filtered)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 134, in offload
    return await asyncio.get_running_loop().run_in_executor(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 505, in _repartition_buffers
    groups = split_by_partition(table, self.column)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 996, in split_by_partition
    assert len(partitions) == len(shards)
AssertionError
ERROR:distributed.core:Exception while handling op shuffle_receive
Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 604, in shuffle_receive
    await shuffle.receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 481, in receive
    await self._receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 496, in _receive
    groups = await self.offload(self._repartition_buffers, filtered)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 134, in offload
    return await asyncio.get_running_loop().run_in_executor(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 505, in _repartition_buffers
    groups = split_by_partition(table, self.column)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 996, in split_by_partition
    assert len(partitions) == len(shards)
AssertionError
ERROR:distributed.shuffle._comms:
Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_comms.py", line 71, in _process
    await self.send(address, shards)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 125, in send
    return await self.rpc(address).shuffle_receive(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 1359, in send_recv_from_rpc
    return await send_recv(comm=comm, op=key, **kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 1143, in send_recv
    raise exc.with_traceback(tb)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 604, in shuffle_receive
    await shuffle.receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 481, in receive
    await self._receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 496, in _receive
    groups = await self.offload(self._repartition_buffers, filtered)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 134, in offload
    return await asyncio.get_running_loop().run_in_executor(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 505, in _repartition_buffers
    groups = split_by_partition(table, self.column)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 996, in split_by_partition
    assert len(partitions) == len(shards)
AssertionError
ERROR:distributed.shuffle._comms:
Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_comms.py", line 71, in _process
    await self.send(address, shards)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 125, in send
    return await self.rpc(address).shuffle_receive(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 1359, in send_recv_from_rpc
    return await send_recv(comm=comm, op=key, **kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 1143, in send_recv
    raise exc.with_traceback(tb)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 604, in shuffle_receive
    await shuffle.receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 481, in receive
    await self._receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 496, in _receive
    groups = await self.offload(self._repartition_buffers, filtered)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 134, in offload
    return await asyncio.get_running_loop().run_in_executor(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 505, in _repartition_buffers
    groups = split_by_partition(table, self.column)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 996, in split_by_partition
    assert len(partitions) == len(shards)
AssertionError
ERROR:distributed.core:Exception while handling op shuffle_receive
Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 603, in shuffle_receive
    shuffle = await self._get_shuffle_run(shuffle_id, run_id)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 704, in _get_shuffle_run
    raise shuffle._exception
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 604, in shuffle_receive
    await shuffle.receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 481, in receive
    await self._receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 496, in _receive
    groups = await self.offload(self._repartition_buffers, filtered)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 134, in offload
    return await asyncio.get_running_loop().run_in_executor(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 505, in _repartition_buffers
    groups = split_by_partition(table, self.column)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 996, in split_by_partition
    assert len(partitions) == len(shards)
    ERROR:distributed.shuffle._comms:
Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_comms.py", line 71, in _process
    await self.send(address, shards)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 125, in send
    return await self.rpc(address).shuffle_receive(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 1359, in send_recv_from_rpc
    return await send_recv(comm=comm, op=key, **kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 1143, in send_recv
    raise exc.with_traceback(tb)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 603, in shuffle_receive
    shuffle = await self._get_shuffle_run(shuffle_id, run_id)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 704, in _get_shuffle_run
    raise shuffle._exception
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 604, in shuffle_receive
    await shuffle.receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 481, in receive
    await self._receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 496, in _receive
    groups = await self.offload(self._repartition_buffers, filtered)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 134, in offload
    return await asyncio.get_running_loop().run_in_executor(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 505, in _repartition_buffers
    groups = split_by_partition(table, self.column)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 996, in split_by_partition
    assert len(partitions) == len(shards)
AssertionError
WARNING:distributed.worker:Compute Failed
Key:       ('shuffle-transfer-c6ba947c7a1bdc00ffa17247dc6db342', 11)
Function:  shuffle_transfer
args:      (                       tf_target  weighted_binding  _partitions
0     Apoc2_12.2308—Apoc2_7.1046          1.962226            5
1     Apoc2_12.2308—Apoc2_7.1047          1.894814            5
2     Apoc2_12.2308—Apoc2_7.1050          0.207537            5
3     Apoc2_12.2308—Apoc2_7.1255          1.299553            5
4     Apoc2_12.2308—Apoc2_7.1256          1.318022            5
...                          ...               ...          ...
6427    Apoc2_12.1393—Apoc2_7.65          2.000000            3
6428   Apoc2_12.1393—Apoc2_7.681          3.958551            3
6429   Apoc2_12.1393—Apoc2_7.711          1.000000            3
6430   Apoc2_12.1393—Apoc2_7.724          0.119455            3
6431   Apoc2_12.1393—Apoc2_7.729          2.000000            3

[6432 rows x 3 columns], 'c6ba947c7a1bdc00ffa17247dc6db342', 11, 14, '_partitions', {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13})
kwargs:    {}
Exception: "RuntimeError('shuffle_transfer failed during shuffle c6ba947c7a1bdc00ffa17247dc6db342')"

WARNING:distributed.worker:Compute Failed
Key:       ('shuffle-transfer-c6ba947c7a1bdc00ffa17247dc6db342', 12)
Function:  shuffle_transfer
args:      (                       tf_target  weighted_binding  _partitions
0     Apoc2_12.2308—Apoc2_8.1117          2.006809            5
1     Apoc2_12.2308—Apoc2_8.1124          1.011392            5
2     Apoc2_12.2308—Apoc2_8.1613          1.000000            5
3     Apoc2_12.2308—Apoc2_8.2474          1.000000            5
4     Apoc2_12.2308—Apoc2_8.2598          3.995726            5
...                          ...               ...          ...
2943  Apoc2_12.1393—Apoc2_8.2655          1.000000            3
2944  Apoc2_12.1393—Apoc2_8.2658          0.002624            3
2945   Apoc2_12.1393—Apoc2_8.502          1.000000            3
2946   Apoc2_12.1393—Apoc2_8.504          0.245149            3
2947   Apoc2_12.1393—Apoc2_8.791          1.000000            3

[2948 rows x 3 columns], 'c6ba947c7a1bdc00ffa17247dc6db342', 12, 14, '_partitions', {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13})
kwargs:    {}
Exception: "RuntimeError('shuffle_transfer failed during shuffle c6ba947c7a1bdc00ffa17247dc6db342')"

2023-11-15 08:42:27 | ERROR | An error has been caught in function '<module>', process 'MainProcess' (5552), thread 'MainThread' (139883518396224):
Traceback (most recent call last):

  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_shuffle.py", line 62, in shuffle_transfer
    return _get_worker_extension().add_partition(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 647, in add_partition
    shuffle = self.get_or_create_shuffle(shuffle_id, type=type, **kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 903, in get_or_create_shuffle
    return sync(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/utils.py", line 418, in sync
    raise exc.with_traceback(tb)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/utils.py", line 391, in f
    result = yield future
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/tornado/gen.py", line 767, in run
    value = future.result()
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 739, in _get_or_create_shuffle
    raise shuffle._exception
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 603, in shuffle_receive
    shuffle = await self._get_shuffle_run(shuffle_id, run_id)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 704, in _get_shuffle_run
    raise shuffle._exception
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/core.py", line 922, in _handle_comm
    result = await result
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 604, in shuffle_receive
    await shuffle.receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 481, in receive
    await self._receive(data)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 496, in _receive
    groups = await self.offload(self._repartition_buffers, filtered)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 134, in offload
    return await asyncio.get_running_loop().run_in_executor(
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 505, in _repartition_buffers
    groups = split_by_partition(table, self.column)
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_worker_extension.py", line 996, in split_by_partition
    assert len(partitions) == len(shards)

AssertionError: assert len(partitions) == len(shards)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

> File "/usr/local/anaconda3/envs/ananse/bin/ananse", line 599, in <module>
    args.func(args)
    │    │    └ Namespace(annotation=None, binding='24hpf.binding/binding.h5', column=['X3', 'X4'], fin_expression=['../apoc_normalized_cpm.t...
    │    └ <function network at 0x7f38b15aaaf0>
    └ Namespace(annotation=None, binding='24hpf.binding/binding.h5', column=['X3', 'X4'], fin_expression=['../apoc_normalized_cpm.t...
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/ananse/commands/network.py", line 37, in network
    b.run_network(
    │ └ <function Network.run_network at 0x7f38b0eda160>
    └ <ananse.network.Network object at 0x7f38b0db4eb0>
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/ananse/network.py", line 616, in run_network
    df_binding = self.aggregate_binding(
                 │    └ <function Network.aggregate_binding at 0x7f38b0edff70>
                 └ <ananse.network.Network object at 0x7f38b0db4eb0>
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/ananse/network.py", line 462, in aggregate_binding
    df = ddf.compute()
         │   └ <function DaskMethodsMixin.compute at 0x7f38bc6ca0d0>
         └ Dask DataFrame Structure:
                                      weighted_binding
           npartitions=14                             
           Apoc2_1.110...
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/dask/base.py", line 314, in compute
    (result,) = compute(self, traverse=False, **kwargs)
                │       │                       └ {}
                │       └ Dask DataFrame Structure:
                │                                    weighted_binding
                │         npartitions=14                             
                │         Apoc2_1.110...
                └ <function compute at 0x7f38bc6ca550>
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/dask/base.py", line 599, in compute
    results = schedule(dsk, keys, **kwargs)
              │        │    │       └ {}
              │        │    └ [[('sort_index-bfadc84d21345053f479d29502b5412f', 0), ('sort_index-bfadc84d21345053f479d29502b5412f', 1), ('sort_index-bfadc8...
              │        └ HighLevelGraph with 3 layers.
              │          <dask.highlevelgraph.HighLevelGraph object at 0x7f388e75feb0>
              │           0. assign-f2468bb5a8c70a7ec9c1e7...
              └ <bound method Client.get of <Client: 'tcp://127.0.0.1:35495' processes=1 threads=2, memory=125.82 GiB>>
              File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/dask/base.py", line 314, in compute
    (result,) = compute(self, traverse=False, **kwargs)
                │       │                       └ {}
                │       └ Dask DataFrame Structure:
                │                                    weighted_binding
                │         npartitions=14                             
                │         Apoc2_1.110...
                └ <function compute at 0x7f38bc6ca550>
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/dask/base.py", line 599, in compute
    results = schedule(dsk, keys, **kwargs)
              │        │    │       └ {}
              │        │    └ [[('sort_index-bfadc84d21345053f479d29502b5412f', 0), ('sort_index-bfadc84d21345053f479d29502b5412f', 1), ('sort_index-bfadc8...
              │        └ HighLevelGraph with 3 layers.
              │          <dask.highlevelgraph.HighLevelGraph object at 0x7f388e75feb0>
              │           0. assign-f2468bb5a8c70a7ec9c1e7...
              └ <bound method Client.get of <Client: 'tcp://127.0.0.1:35495' processes=1 threads=2, memory=125.82 GiB>>
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/client.py", line 3226, in get
    results = self.gather(packed, asynchronous=asynchronous, direct=direct)
              │    │      │                    │                    └ None
              │    │      │                    └ None
              │    │      └ [[<Future: cancelled, key: ('sort_index-bfadc84d21345053f479d29502b5412f', 0)>, <Future: cancelled, key: ('sort_index-bfadc84...
              │    └ <function Client.gather at 0x7f38b16e5700>
              └ <Client: 'tcp://127.0.0.1:35495' processes=1 threads=2, memory=125.82 GiB>
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/client.py", line 2361, in gather
    return self.sync(
           │    └ <function SyncMethodMixin.sync at 0x7f38bc41f8b0>
           └ <Client: 'tcp://127.0.0.1:35495' processes=1 threads=2, memory=125.82 GiB>
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/utils.py", line 351, in sync
    return sync(
           └ <function sync at 0x7f38bc41f940>
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/utils.py", line 418, in sync
    raise exc.with_traceback(tb)
          │   │              └ <traceback object at 0x7f388e2ba0c0>
          │   └ <method 'with_traceback' of 'BaseException' objects>
          └ RuntimeError('shuffle_transfer failed during shuffle c6ba947c7a1bdc00ffa17247dc6db342')
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/utils.py", line 391, in f
    result = yield future
                   └ <Task finished name='Task-2902' coro=<Client._gather() done, defined at /usr/local/anaconda3/envs/ananse/lib/python3.8/site-p...
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/tornado/gen.py", line 767, in run
    value = future.result()
            └ None
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/client.py", line 2224, in _gather
    raise exception.with_traceback(traceback)
          │         │              └ <traceback object at 0x7f388e2ac140>
          │         └ <method 'with_traceback' of 'BaseException' objects>
          └ RuntimeError('shuffle_transfer failed during shuffle c6ba947c7a1bdc00ffa17247dc6db342')
  File "/usr/local/anaconda3/envs/ananse/lib/python3.8/site-packages/distributed/shuffle/_shuffle.py", line 72, in shuffle_transfer
    raise RuntimeError(f"shuffle_transfer failed during shuffle {id}") from e

RuntimeError: shuffle_transfer failed during shuffle c6ba947c7a1bdc00ffa17247dc6db342
INFO:distributed.nanny:Closing Nanny at 'tcp://127.0.0.1:33573'. Reason: nanny-close
INFO:distributed.nanny:Nanny asking worker to close. Reason: nanny-close

Have you fixed this issue? I'm having the same problem as you. The error is "distributed.core - ERROR - Exception while handling op shuffle_receive".