search

nanoporetech / dorado

Oxford Nanopore's Basecaller
https://nanoporetech.com/
Other
495 stars 59 forks source link

Dorado 0.5.3 - memory issues - BAM truncated? #768

Open cfrias-cnag opened 5 months ago

cfrias-cnag commented 5 months ago

Issue Report

We have some data that I am not able to perform the basecalling. I tried with two approach, but I got different errors in both test.

First command (passing all the pod5 reads)

/dorado-0.5.3-linux-x64/bin/dorado duplex \
/dorado-0.5.3-linux-x64/models/dna_r10.4.1_e8.2_400bps_sup@v4.3.0 \
$INPUT \
-v -r -x "cuda:all" --min-qscore 10 --modified-bases 5mC_5hmC \
> XXX_dorado0.5.3-duplex-5mC_5hmC@v1-sup.bam 2> XXX_dorado0.5.3-duplex-5mC_5hmC@v1-sup.log

First issue

//
...
[2024-03-10 12:30:23.590] [debug] Sorted channel 1212
Killed
//

Second command (two jobs jobs, one for pod5 fail reads and one for pod5 pass reads)

/dorado-0.5.3-linux-x64/bin/dorado duplex \
/dorado-0.5.3-linux-x64/models/dna_r10.4.1_e8.2_400bps_sup@v4.3.0 \
./XXX_rebasecalling/methylation_basecalling_240123_5mC_5hmC-ChannelTable/pod5_pass/split_by_channel_pod5_pass/ \
--min-qscore 10 --modified-bases 5mC_5hmC \
> XXXX_dorado0.5.3-duplex-5mC_5hmC@v1-sup.channel_pod5pass.bam 2> XXX_dorado0.5.3-duplex-5mC_5hmC@v1-sup.channel_pod5pass.log

Second issue

fail reads

//
...
[2024-04-19 09:41:19.214] [info] > Starting Stereo Duplex pipeline
[2024-04-19 09:41:19.233] [info] > Reading read channel info
[2024-04-19 09:41:34.020] [info] > Processed read channel info
**[2024-04-19 23:58:25.388] [error] basic_string::substr: __pos (which is 7411) > this->size() (which is 5817)**
[2024-04-21 10:22:34.399] [info] > Simplex reads basecalled: 470223
[2024-04-21 10:22:34.399] [info] > Simplex reads filtered: 1908745
[2024-04-21 10:22:34.399] [info] > Duplex reads basecalled: 28201
[2024-04-21 10:22:34.399] [info] > Duplex reads filtered: 10628
[2024-04-21 10:22:34.400] [info] > Duplex rate: 3.4319797%
[2024-04-21 10:22:34.400] [info] > Basecalled @ Bases/s: 1.218011e+05
//

pass reads

//
..
[2024-04-19 09:33:52.687] [info] > No duplex pairs file provided, pairing will be performed automatically
[2024-04-19 09:37:10.935] [info]  - set batch size for cuda:0 to 640
[2024-04-19 09:37:19.733] [info]  - set batch size for cuda:0 to 1792
[2024-04-19 09:37:19.733] [info] > Starting Stereo Duplex pipeline
[2024-04-19 09:37:19.763] [info] > Reading read channel info
[2024-04-19 09:40:27.087] [info] > Processed read channel info
**[2024-04-23 02:56:05.933] [error] basic_string::substr: __pos (which is 11734) > this->size() (which is 11684)**
//

BAM file

For fail pod5 job is fine, but for the pass pod5 job the BAM file seems truncated.

samtools view XXXX_dorado0.5.3-duplex-5mC_5hmC@v1-sup.channel_pod5pass.bam | most
[W::bam_hdr_read] EOF marker is absent. The input is probably truncated

Please, could you teld me how can we solve?

Many thanks in advance! Cristina

HalfPhoton commented 5 months ago

Hi @cfrias-cnag, is this running on a compute cluster?

cfrias-cnag commented 5 months ago

Hi @HalfPhoton, no, we use a local computer. Did you have a similar issue?

HalfPhoton commented 5 months ago

No, I'm trying to understand why this issue is happening - You used $INPUT in your example and I was thinking that this could be related to other issues we've seen with users running on compute clusters.

Can you complete the issue template to give us more information to help you?

cfrias-cnag commented 4 months ago

Hi @HalPhoton,

No, in the bash script I don't use the var $INPUT, I putted the full PATH.

cfrias-cnag commented 4 months ago

Can you complete the issue template to give us more information to help you?

dorado version used

dorado: 0.5.3 dorado: v0.6.1

lsb_release -a

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.2 LTS Release: 22.04 Codename: jammy

nvidia-smi

Wed May  8 13:43:53 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA RTX A6000                On | 00000000:21:00.0 Off |                  Off |
| 65%   84C    P2              300W / 300W|  44603MiB / 49140MiB |    100%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

More info

Please, let me know if you need more information.

Many thanks in advance! Cristina

vellamike commented 4 months ago

Hello @cfrias-cnag does basecalling finish on the pass read set or does dorado crash? It's not clear from your log

cfrias-cnag commented 4 months ago

Hi Mike,

Sorry, I mean from pass reads.

Log file: “[2024-04-24 10:38:16.693] [info] Running: "duplex" @.***" "/scratch_isilon/groups/pbt/cfrias/ont02/output/240123_BCLLATLAS_240_5208AI_PAU40160_rebasecalling/methylation_basecalling_240123_5mC_5hmC-ChannelTable/pod5_pass/split_by_channel_pod5_pass/" "--min-qscore" "10" "--modified-bases" "5mC_5hmC" [2024-04-24 10:38:16.719] [info] > No duplex pairs file provided, pairing will be performed automatically [2024-04-24 10:41:29.710] [info] cuda:0 using chunk size 9996, batch size 576 [2024-04-24 10:41:34.615] [info] cuda:0 using chunk size 10000, batch size 896 [2024-04-24 10:41:35.562] [info] > Starting Stereo Duplex pipeline [2024-04-24 10:41:35.580] [info] > Reading read channel info [2024-04-24 10:44:38.647] [info] > Processed read channel info [2024-04-26 23:01:43.012] [error] basic_string::substr: __pos (which is 10484) > this->size() (which is 10227)”

I thought that maybe I can try to extract the pod5 basecalled from the uBAM, using pod5_subset and launch again the basecalling with those reads. (in case the error is related to some pod5 read )

What do you think about is?

Many thanks! Cris

De: Mike Vella @.> Enviado el: viernes, 17 de mayo de 2024 12:59 Para: nanoporetech/dorado @.> CC: Maria Cristina Frias Lopez @.>; Mention @.> Asunto: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)

You don't often get email from @.**@.>. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

Hello @cfrias-cnaghttps://github.com/cfrias-cnag does basecalling finish on the pass read set or does dorado crash? It's not clear from your log

— Reply to this email directly, view it on GitHubhttps://github.com/nanoporetech/dorado/issues/768#issuecomment-2117320263, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGMPNOLJMPURVSX24Q57OXDZCXPHFAVCNFSM6AAAAABGWOYSPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJXGMZDAMRWGM. You are receiving this because you were mentioned.Message ID: @.**@.>>

HalfPhoton commented 4 months ago

Could you add -vv to your command to generate a detailed log?

HalfPhoton commented 4 months ago

Also - can I confirm with you: This job was running for 2.5 days?

[2024-04-24 10:44:38.647] [info] > Processed read channel info
[2024-04-26 23:01:43.012] [error] basic_string::substr: __pos (which is 10484) > this->size() (which is 10227)”
cfrias-cnag commented 4 months ago

No, the job was running 2 days more, 5 in total.

The uBAM file finished to write on 29/04/2024 (and started on 24/04/2024):

@.***

I found a similar message using the FAIL reads basecalling, but then the log file show that the basecalling continues and said that it finished successfully, and furthermore the uBAM is not truncated.

Thanks! Cris

De: Richard Harris @.> Enviado el: viernes, 17 de mayo de 2024 14:11 Para: nanoporetech/dorado @.> CC: Maria Cristina Frias Lopez @.>; Mention @.> Asunto: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)

You don't often get email from @.**@.>. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

Also - can I confirm with you: This job was running for 2.5 days?

[2024-04-24 10:44:38.647] [info] > Processed read channel info

[2024-04-26 23:01:43.012] [error] basic_string::substr: __pos (which is 10484) > this->size() (which is 10227)”

— Reply to this email directly, view it on GitHubhttps://github.com/nanoporetech/dorado/issues/768#issuecomment-2117459128, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGMPNOLO3SZVODPZ5CBDSKLZCXXWZAVCNFSM6AAAAABGWOYSPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJXGQ2TSMJSHA. You are receiving this because you were mentioned.Message ID: @.**@.>>

cfrias-cnag commented 4 months ago

Hi Richard,

Thanks!

I've just launched the same command adding -vv.

Cris

De: Richard Harris @.> Enviado el: viernes, 17 de mayo de 2024 14:10 Para: nanoporetech/dorado @.> CC: Maria Cristina Frias Lopez @.>; Mention @.> Asunto: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)

You don't often get email from @.**@.>. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

Could you add -vv to your command to generate a detailed log?

- Reply to this email directly, view it on GitHubhttps://github.com/nanoporetech/dorado/issues/768#issuecomment-2117457184, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGMPNOLABSOZ5KPXBEVCF5LZCXXRVAVCNFSM6AAAAABGWOYSPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJXGQ2TOMJYGQ. You are receiving this because you were mentioned.Message ID: @.**@.>>

cfrias-cnag commented 4 months ago

Hi Richard,

The job finished.

The uBAM is truncated.

Please, could you tell me how I can send you the log file ( 19GB )?

I've just launched the same basecalling with last version of dorado v.0.7.0 (dorado-0.7.0-linux-x64). (The previous log is produced by dorado v.0.6.1)

Many thanks in advance! Cristina

From: Maria Cristina Frias Lopez Sent: Friday, May 17, 2024 4:13 PM To: nanoporetech/dorado @.>; nanoporetech/dorado @.> Cc: Mention @.>; Matthew John Ingham @.> Subject: RE: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)

Hi Richard,

Thanks!

I’ve just launched the same command adding -vv.

Cris

De: Richard Harris @.> Enviado el: viernes, 17 de mayo de 2024 14:10 Para: nanoporetech/dorado @.> CC: Maria Cristina Frias Lopez @.>; Mention @.> Asunto: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)

You don't often get email from @.**@.>. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

Could you add -vv to your command to generate a detailed log?

— Reply to this email directly, view it on GitHubhttps://github.com/nanoporetech/dorado/issues/768#issuecomment-2117457184, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGMPNOLABSOZ5KPXBEVCF5LZCXXRVAVCNFSM6AAAAABGWOYSPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJXGQ2TOMJYGQ. You are receiving this because you were mentioned.Message ID: @.**@.>>

HalfPhoton commented 4 months ago

Hi @cfrias-cnag, sorry for the delayed response. Can you update dorado to 0.7.0 and try on a smaller dataset using --max-reads 10000 to see if the issue persists?

cfrias-cnag commented 4 months ago

Dear Richard,

I updated the version of dorado to the last one,

But I launched before you wrote so when It finished (if the error persists), I will re-launch with the parameter you suggested me(--max-reads 10000) because I did before you wrote me.

I will let you know how it is going.

Many thanks in advance! Cristina

PD: I’m also looking at the community for any update, following this one: https://github.com/nanoporetech/dorado/issues/855

I have to comment with you, that I got similar errors during the basecalling of the “fail reads” but like the other user it were skipped “alone” (by dorado, I didn’t filer any read) and it finished successfully. But, in my case, the issue is related to the subset of “pass read”, after showed this error, dorado was hung and died without any further messages.

De: Richard Harris @.> Enviado el: jueves, 30 de mayo de 2024 11:42 Para: nanoporetech/dorado @.> CC: Maria Cristina Frias Lopez @.>; Mention @.> Asunto: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)

Hi @cfrias-cnaghttps://github.com/cfrias-cnag, sorry for the delayed response. Can you update dorado to 0.7.0 and try on a smaller dataset using --max-reads 10000 to see if the issue persists?

— Reply to this email directly, view it on GitHubhttps://github.com/nanoporetech/dorado/issues/768#issuecomment-2139170677, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGMPNONQEOWA3ZQMA2XPZ7DZE3X6PAVCNFSM6AAAAABGWOYSPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZZGE3TANRXG4. You are receiving this because you were mentioned.Message ID: @.**@.>>

cfrias-cnag commented 4 months ago

Dear Richard,

The process with dorado v.0.7.0 died after 7 hours running (11GB of uBAM).

With the previous versions of dorado, the process run about 5 days and was produced a bigger uBAM "truncated" (150GB ).

Let me check adding the parameter you suggested me (--max-reads 10000) .

Regards, Cristina

From: Maria Cristina Frias Lopez Sent: Monday, June 3, 2024 12:40 PM To: nanoporetech/dorado @.>; nanoporetech/dorado @.> Cc: Mention @.>; Matthew John Ingham @.>; Simon Heath @.***> Subject: RE: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)

Dear Richard,

I updated the version of dorado to the last one,

But I launched before you wrote so when It finished (if the error persists), I will re-launch with the parameter you suggested me(--max-reads 10000) because I did before you wrote me.

I will let you know how it is going.

Many thanks in advance!

Cristina

PD: I’m also looking at the community for any update, following this one:

https://github.com/nanoporetech/dorado/issues/855

I have to comment with you, that I got similar errors during the basecalling of the “fail reads” but like the other user it were skipped “alone” (by dorado, I didn’t filer any read) and it finished successfully. But, in my case, the issue is related to the subset of “pass read”, after showed this error, dorado was hung and died without any further messages.

De: Richard Harris @.> Enviado el: jueves, 30 de mayo de 2024 11:42 Para: nanoporetech/dorado @.> CC: Maria Cristina Frias Lopez @.>; Mention @.> Asunto: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)

Hi @cfrias-cnaghttps://github.com/cfrias-cnag, sorry for the delayed response. Can you update dorado to 0.7.0 and try on a smaller dataset using --max-reads 10000 to see if the issue persists?

— Reply to this email directly, view it on GitHubhttps://github.com/nanoporetech/dorado/issues/768#issuecomment-2139170677, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGMPNONQEOWA3ZQMA2XPZ7DZE3X6PAVCNFSM6AAAAABGWOYSPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZZGE3TANRXG4. You are receiving this because you were mentioned.Message ID: @.**@.>>

cfrias-cnag commented 4 months ago

Dear Richard,

Seems that this parameter doesn't exist: --max-reads 10000

[cid:f8779588-78c5-4151-92c8-f840dafffa56]

[cid:8519a5a5-46a2-4294-9561-dbb3bf689bb8]

From: Maria Cristina Frias Lopez @.> Sent: Tuesday, June 4, 2024 11:14 AM To: nanoporetech/dorado @.>; nanoporetech/dorado @.> Cc: Mention @.>; Matthew John Ingham @.>; Simon Heath @.> Subject: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)

Dear Richard,

The process with dorado v.0.7.0 died after 7 hours running (11GB of uBAM).

With the previous versions of dorado, the process run about 5 days and was produced a bigger uBAM "truncated" (150GB ).

Let me check adding the parameter you suggested me (--max-reads 10000) .

Regards, Cristina

From: Maria Cristina Frias Lopez Sent: Monday, June 3, 2024 12:40 PM To: nanoporetech/dorado @.>; nanoporetech/dorado @.> Cc: Mention @.>; Matthew John Ingham @.>; Simon Heath @.***> Subject: RE: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)

Dear Richard,

I updated the version of dorado to the last one,

But I launched before you wrote so when It finished (if the error persists), I will re-launch with the parameter you suggested me(--max-reads 10000) because I did before you wrote me.

I will let you know how it is going.

Many thanks in advance!

Cristina

PD: I’m also looking at the community for any update, following this one:

https://github.com/nanoporetech/dorado/issues/855

I have to comment with you, that I got similar errors during the basecalling of the “fail reads” but like the other user it were skipped “alone” (by dorado, I didn’t filer any read) and it finished successfully. But, in my case, the issue is related to the subset of “pass read”, after showed this error, dorado was hung and died without any further messages.

De: Richard Harris @.> Enviado el: jueves, 30 de mayo de 2024 11:42 Para: nanoporetech/dorado @.> CC: Maria Cristina Frias Lopez @.>; Mention @.> Asunto: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)

Hi @cfrias-cnaghttps://github.com/cfrias-cnag, sorry for the delayed response. Can you update dorado to 0.7.0 and try on a smaller dataset using --max-reads 10000 to see if the issue persists?

— Reply to this email directly, view it on GitHubhttps://github.com/nanoporetech/dorado/issues/768#issuecomment-2139170677, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGMPNONQEOWA3ZQMA2XPZ7DZE3X6PAVCNFSM6AAAAABGWOYSPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZZGE3TANRXG4. You are receiving this because you were mentioned.Message ID: @.**@.>>

HalfPhoton commented 4 months ago

https://github.com/nanoporetech/dorado/blob/release-v0.7/dorado/cli/basecaller.cpp#L473

https://github.com/nanoporetech/dorado/blob/release-v0.7/dorado/cli/demux.cpp#L106

--max-reads should be available in both simplex and duplex basecalling. Can you share your full command again?

cfrias-cnag commented 4 months ago

Hi Richard,

I am using "dorado duplex", not "dorado basecaller" neither "dorado demux" (the links you send me).

Here you have the full command:

/home/cfrias/software/dorado-0.7.0-linux-x64/bin/dorado duplex \ @. \ /scratch_isilon/groups/pbt/cfrias/ont02/ \ --min-qscore 10 --modified-bases 5mC_5hmC \ --max-reads 10000 \ -vv > @._pod5pass.dorado-0.7.0-maxreads_10000.bam 2> @.***_pod5pass.dorado-0.7.0-maxreads_10000.log

I also installed the last version, but I can not see this parameter in the "duplex", just in the "simplex":

duplex

[cid:c514db7b-7515-4a82-9d86-a25dd680d7d1]

simplex

[cid:194c69a2-1bfa-462d-8aec-fc76017a6116]

Many thanks in advance,

Cris

From: Richard Harris @.> Sent: Tuesday, June 4, 2024 2:39 PM To: nanoporetech/dorado @.> Cc: Maria Cristina Frias Lopez @.>; Mention @.> Subject: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)

https://github.com/nanoporetech/dorado/blob/release-v0.7/dorado/cli/basecaller.cpp#L473

https://github.com/nanoporetech/dorado/blob/release-v0.7/dorado/cli/demux.cpp#L106

--max-reads should be available in both simplex and duplex basecalling. Can you share your full command again?

— Reply to this email directly, view it on GitHubhttps://github.com/nanoporetech/dorado/issues/768#issuecomment-2147434816, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGMPNOOAM3Y22H4V6V6HH73ZFWYRJAVCNFSM6AAAAABGWOYSPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBXGQZTIOBRGY. You are receiving this because you were mentioned.Message ID: @.***>

cfrias-cnag commented 3 months ago

Hi Richard,

I hope you are doing well.

Please, could you let us know if there are any update?

I am using "dorado-0.7.1-linux-x64/bin/dorado duplex" but I can't see the "--max-reads" option:

https://github.com/nanoporetech/dorado/blob/release-v0.7/dorado/cli/duplex.cpp

Thanks! Cris

From: Maria Cristina Frias Lopez @.> Sent: Tuesday, June 4, 2024 3:22 PM To: nanoporetech/dorado @.>; nanoporetech/dorado @.> Cc: Mention @.>; Simon Heath @.>; Matthew John Ingham @.> Subject: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)

Hi Richard,

I am using "dorado duplex", not "dorado basecaller" neither "dorado demux" (the links you send me).

Here you have the full command:

/home/cfrias/software/dorado-0.7.0-linux-x64/bin/dorado duplex \ @. \ /scratch_isilon/groups/pbt/cfrias/ont02/ \ --min-qscore 10 --modified-bases 5mC_5hmC \ --max-reads 10000 \ -vv > @._pod5pass.dorado-0.7.0-maxreads_10000.bam 2> @.***_pod5pass.dorado-0.7.0-maxreads_10000.log

I also installed the last version, but I can not see this parameter in the "duplex", just in the "simplex":

duplex

[cid:c514db7b-7515-4a82-9d86-a25dd680d7d1]

simplex

[cid:194c69a2-1bfa-462d-8aec-fc76017a6116]

Many thanks in advance,

Cris

From: Richard Harris @.> Sent: Tuesday, June 4, 2024 2:39 PM To: nanoporetech/dorado @.> Cc: Maria Cristina Frias Lopez @.>; Mention @.> Subject: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)

https://github.com/nanoporetech/dorado/blob/release-v0.7/dorado/cli/basecaller.cpp#L473

https://github.com/nanoporetech/dorado/blob/release-v0.7/dorado/cli/demux.cpp#L106

--max-reads should be available in both simplex and duplex basecalling. Can you share your full command again?

— Reply to this email directly, view it on GitHubhttps://github.com/nanoporetech/dorado/issues/768#issuecomment-2147434816, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGMPNOOAM3Y22H4V6V6HH73ZFWYRJAVCNFSM6AAAAABGWOYSPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBXGQZTIOBRGY. You are receiving this because you were mentioned.Message ID: @.***>

malton-ont commented 3 months ago

Probably related to issue 855.