Closed riker1 closed 1 year ago
I'm having issues now with multiple users who cannot send messages to the list: bridge seems to have trouble parsing
[Sun Nov 6 22:35:06 2022] mail.cgi: bridge - irrecoverable error processing message. Skipping message: parse_data: unable to open in-memory file handle at plugins/bridge line 2545. [Sun Nov 6 22:35:06 2022] mail.cgi: at plugins/bridge line 1217. [Sun Nov 6 22:35:06 2022] mail.cgi: bridge::catch {...} ("parse_data: unable to open in-memory file handle at plugins/b"...) called at DADA/perllib/Try/Tiny.pm line 123 [Sun Nov 6 22:35:06 2022] mail.cgi: Try::Tiny::try(CODE(0x55c70a37ba00), Try::Tiny::Catch=REF(0x55c70a37bbc8)) called at plugins/bridge line 1225 [Sun Nov 6 22:35:06 2022] mail.cgi: bridge::start(undef) called at plugins/bridge line 1024 [Sun Nov 6 22:35:06 2022] mail.cgi: bridge::scheduled_task("_all") called at /usr/lib/cgi-bin/dada/DADA/App.pm line 16815 [Sun Nov 6 22:35:06 2022] mail.cgi: DADA::App::try {...} () called at DADA/perllib/Try/Tiny.pm line 102 [Sun Nov 6 22:35:06 2022] mail.cgi: eval {...} called at DADA/perllib/Try/Tiny.pm line 93 [Sun Nov 6 22:35:06 2022] mail.cgi: Try::Tiny::try(CODE(0x55c708a9c110), Try::Tiny::Catch=REF(0x55c708c44378)) called at /usr/lib/cgi-bin/dada/DADA/App.pm line 16819 [Sun Nov 6 22:35:06 2022] mail.cgi: DADA::App::schedules(DADA::App=HASH(0x55c7072dcfc8)) called at /usr/local/share/perl/5.34.0/CGI/Application.pm line 144 [Sun Nov 6 22:35:06 2022] mail.cgi: eval {...} called at /usr/local/share/perl/5.34.0/CGI/Application.pm line 143 [Sun Nov 6 22:35:06 2022] mail.cgi: CGI::Application::__get_body(DADA::App=HASH(0x55c7072dcfc8), "_schedAPSGX2R9C37HQ9ZS") called at /usr/local/share/perl/5.34.0/CGI/Application.pm line 192
any ideas? I've tried reconfiguring the settings in dada to parse via tmp/file either in .dada_mail_settings or in /tmp makes no difference. server has plenty of RAM/Disk/ & CPU so I'm at a loss as to do try something else.
Internal Problem The attached message was sent to your mailing list to be sent out to as a mass mailing. but an internal error in the app itself stopped the message from being sent out. Additional tries will not be attempted.
At your discretion, you may look over the message, and manually send out the contents through the list control panel.
Errors are below:
parse_data: unable to open in-memory file handle at plugins/bridge line 2545.
Can you attach the source of the message (the simpler the message, the better) that's illustrating this problem?
Here’s the original message sender tried and it didn’t go through. I tried to also send it as I am an authorized sender and same failure.
Here’s output I pasted into a file showing bridge trying to work on the message mail-sample.txt https://github.com/justingit/dada-mail/files/9863383/mail-sample.txt
[snip /]
I'm looking for the SOURCE of the message, so headers and all, message encoded, etc. Without that, I don't have something to recreate the issue.
Here are examples of sources of messages,
https://github.com/justingit/dada-mail/tree/main/app/dada/t/corpus/email_messages
Justin,
Sorry about the confusion. Here is raw source, one sent by subscriber and the other my attempt to try to figure out what was going on:
== this was the first message that the subscriber sent to list ==
Received: from exch16n1.yorkcollege.local (172.16.139.53) by exch16n1.yorkcollege.local (172.16.139.53) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.13 via Mailbox Transport; Sat, 22 Oct 2022 17:25:38 -0400 Received: from exch16n4.yorkcollege.local (172.16.139.56) by exch16n1.yorkcollege.local (172.16.139.53) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.13; Sat, 22 Oct 2022 17:25:38 -0400 Received: from exch16n4.yorkcollege.local ([fe80::1198:8cf9:f067:e35e]) by exch16n4.yorkcollege.local ([fe80::1198:8cf9:f067:e35e%2]) with mapi id 15.01.2507.013; Sat, 22 Oct 2022 17:25:37 -0400 From: John Drobnicki @.> To: Yorktalk @.> Subject: Calls for contributors Thread-Topic: Calls for contributors Thread-Index: AdjlVdXoM+EptrfiTi2+msUTWdNg0AAtM3FvABSMDfw= Date: Sat, 22 Oct 2022 17:25:37 -0400 Message-ID: @.> References: @*.**@*.> In-Reply-To: @.> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-Exchange-Organization-SCL: -1 X-MS-TNEF-Correlator: @.> MIME-Version: 1.0 X-MS-Exchange-Organization-MessageDirectionality: Originating X-MS-Exchange-Organization-AuthSource: exch16n4.yorkcollege.local X-MS-Exchange-Organization-AuthAs: Internal X-MS-Exchange-Organization-AuthMechanism: 04 X-MS-Exchange-Organization-Network-Message-Id: 69540fb5-c56c-4c21-c31f-08dab473f679 Return-Path: @. X-MS-Exchange-Organization-Recipient-P2-Type: Bcc X-MS-Exchange-Transport-EndToEndLatency: 00:00:00.8753071 X-MS-Exchange-Processed-By-BccFoldering: 15.01.2507.013 Content-type: multipart/alternative; boundary="B_3750676610_1261466448"
This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible.
--B_3750676610_1261466448 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: quoted-printable
Call for Papers: Religion and James Bond
"...The volume will focus on a number of themes, including, but not limited= to: sin/redemption, sexuality, political theology, and religious plurality.= .. Though the volume will focus predominately on the films, I also hope to = see proposals on Ian Fleming=E2=80=99s novels, the history and development of Bond= , and perhaps even the video games and collectibles. We will have to see wha= t the call for papers brings in, but there is a lot of fertile ground in the= world of 007..."
https://networks.h-net.org/node/73374/announcements/11360067/call-papers-re= ligion-and-james-bond
=20
=20
Call for Papers: Global Futures
"The editors of New Global Studies invite proposals for essays on the subje= ct of =E2=80=98global futures=E2=80=99. Essays may cover any historical period..."
https://networks.h-net.org/node/73374/announcements/11347895/call-papers-gl= obal-futures
=20
=20
Advances and Applications in Catalysis with Earth-Abundant Metals
"...The ACS Publications journals Organometallics and Organic Process Resea= rch & Development are partnering to produce a Virtual Special Issue featurin= g the latest work on earth-abundant metal catalysis from research teams acro= ss industrial process chemistry and academia. Topics appropriate for this i= ssue will focus on the design and applications of green and sustainable liga= nds and catalysts involving earth-abundant metals from main group and transi= tion metals across the Periodic Table. Research that capitalizes on industry= -academic collaborations and work from start-up companies and national labo= ratories is welcome..."
https://axial.acs.org/2022/08/05/call-for-papers-advances-and-applications-= in-catalysis-with-earth-abundant-metals/
=20
=20
Resource recovery and recycling from water streams: Advanced membrane techn= ologies and case studies
"...This Special Issue in ACS ES&T Water is seeking rigorous research artic= les, reviews, and perspectives on the latest advances in membrane technologi= es for resource recovery and recycling from diverse water streams (e.g. salt= lake brines, brewery wastewater, landfill leachate, anaerobically treated = sewage, and more)..."
https://axial.acs.org/2022/04/25/call-for-papers-resource-recovery-and-recy= cling-from-water-streams-advanced-membrane-technologies-and-case-studies/
=20
=20
Applications of Artificial Intelligence, Machine Learning, and Data Analyti= cs in Water Environments
"...This Special Issue Call for Papers from ACS ES&T Water seeks rigorous r= esearch articles, reviews, and perspectives on the current progress, researc= h, opportunities and challenges in applying AI/ML and data analytics to solv= ing environmental problems related to water, and to identify research prior= ities our community should focus on in the near future..."
https://axial.acs.org/2022/06/16/applications-of-artificial-intelligence-ma= chine-learning-and-data-analytics-in-water-environments/
=20
=20
The Future of Agriculture and Food: Sustainable Approaches to Achieve Zero = Hunger
"...This virtual special issue [of Journal of Agricultural and Food Chemist= ry, ACS Agricultural Science & Technology, and ACS Food Science & Technology= ] is designed to raise awareness of the role of agriculture and food in achi= eving sustainable development goals. It will also support the Sustainable Fu= tures Initiative by highlighting the need for rapid innovation and the deve= lopment of sustainable production methods... Manuscripts should address adva= nces in food or agricultural research which contribute to more robust, plent= iful, nutritional, palatable and sustainable food systems..."
https://axial.acs.org/2022/08/19/call-for-papers-virtual-special-issue-on-t= he-future-of-agriculture-and-food-sustainable-approaches-to-achieve-zero-hun= ger/
=20
=20
Dissecting the role of immune responses in complex brain disorders: transla= tional and clinical insights
"Neurodevelopmental, psychiatric and neurodegenerative diseases together co= ntribute a high global burden of morbidity and mortality yet insights into u= nderlying mechanisms of etiology, therapeutic advances and changes to clini= cal practice remain limited. The development and application of cutting edge= technologies and approaches, including disease models, multi-omics, data in= tegration, and artificial intelligence, have advanced our understanding of = disease risk and mechanisms underlying these complex brain disorders... To c= apture advances in this growing, multi-disciplinary area, Genome Medicine is= pleased to announce a call for papers for our upcoming special issue..."
https://genomemedicine.biomedcentral.com/articles/collections/dissecting-br= ain-disorders
=20
=20
Emerging Evolutionary Computing Tools for Machine Learning Applications in = Health Care
"...This special issue [of Soft Computing: A Fusion of Foundations, Methodo= logies and Applications] aims to compile and unify all the novel approaches = involving recent ECT advancements applied to ML seeking health care solution= s globally. The Special Issue invites a large spectrum of recent advances i= n the field of nature-inspired techniques, machine learning algorithms for h= ealth applications in specific. Contributions pertaining to new techniques, = novel evaluation methods and practices, variants of existing algorithms and = challenges relevant to the theme..."
https://www.springer.com/journal/500/updates/23603306
=20
--B_3750676610_1261466448 Content-type: text/html; charset="UTF-8" Content-transfer-encoding: quoted-printable
--B_3750676610_1261466448—
== this was my attempt to resend the message using catchall mailbox I use for messages etc ==
MIME-Version: 1.0 Date: Tue, 25 Oct 2022 14:52:02 -0400 From: @. To: @. Subject: Calls for contributors | Retransmit User-Agent: Roundcube Webmail Message-ID: @.> X-Sender: @. Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit
Call for Papers: Religion and James Bond "...The volume will focus on a number of themes, including, but not limited to: sin/redemption, sexuality, political theology, and a religious plurality... Though the volume will focus predominately on the films, I also hope to see proposals on Ian Fleming’s novels, the history and development of Bond, and perhaps even the video games and collectibles. We will have to see what the call for papers brings in, but there is a lot of fertile ground in the world of 007..." https://networks.h-net.org/node/73374/announcements/11360067/call-papers-religion-and-james-bond
Call for Papers: Global Futures "The editors of New Global Studies invite proposals for essays on the subject of ‘global futures’. Essays may cover any historical period..." https://networks.h-net.org/node/73374/announcements/11347895/call-papers-global-futures
Advances and Applications in Catalysis with Earth-Abundant Metals "...The ACS Publications journals Organometallics and Organic Process Research & Development are partnering to produce a Virtual Special Issue featuring the latest work on earth-abundant metal catalysis from research teams across industrial process chemistry and academia. Topics appropriate for this issue will focus on the design and applications of green and sustainable ligands and catalysts involving earth-abundant metals from main group and transition metals across the Periodic Table. Research that capitalizes on industry-academic collaborations and work from start-up companies and national laboratories is welcome..." https://axial.acs.org/2022/08/05/call-for-papers-advances-and-applications-in-catalysis-with-earth-abundant-metals/
Resource recovery and recycling from water streams: Advanced membrane technologies and case studies "...This Special Issue in ACS ES&T Water is seeking rigorous research articles, reviews, and perspectives on the latest advances in membrane technologies for resource recovery and recycling from diverse water streams (e.g. salt lake brines, brewery wastewater, landfill leachate, anaerobically treated sewage, and more)..." https://axial.acs.org/2022/04/25/call-for-papers-resource-recovery-and-recycling-from-water-streams-advanced-membrane-technologies-and-case-studies/
Applications of Artificial Intelligence, Machine Learning, and Data Analytics in Water Environments "...This Special Issue Call for Papers from ACS ES&T Water seeks rigorous research articles, reviews, and perspectives on the current progress, research, opportunities and challenges in applying AI/ML and data analytics to solving environmental problems related to water, and to identify research priorities our community should focus on in the near future..." https://axial.acs.org/2022/06/16/applications-of-artificial-intelligence-machine-learning-and-data-analytics-in-water-environments/
The Future of Agriculture and Food: Sustainable Approaches to Achieve Zero Hunger "...This virtual special issue [of Journal of Agricultural and Food Chemistry, ACS Agricultural Science & Technology, and ACS Food Science & Technology] is designed to raise awareness of the role of agriculture and food in achieving sustainable development goals. It will also support the Sustainable Futures Initiative by highlighting the need for rapid innovation and the development of sustainable production methods... Manuscripts should address advances in food or agricultural research which contribute to more robust, plentiful, nutritional, palatable and sustainable food systems..." https://axial.acs.org/2022/08/19/call-for-papers-virtual-special-issue-on-the-future-of-agriculture-and-food-sustainable-approaches-to-achieve-zero-hunger/
Dissecting the role of immune responses in complex brain disorders: translational and clinical insights "Neurodevelopmental, psychiatric and neurodegenerative diseases together contribute a high global burden of morbidity and mortality yet insights into underlying mechanisms of etiology, therapeutic advances and changes to clinical practice remain limited. The development and application of cutting edge technologies and approaches, including disease models, multi-omics, data integration, and artificial intelligence, have advanced our understanding of disease risk and mechanisms underlying these complex brain disorders... To capture advances in this growing, multi-disciplinary area, Genome Medicine is pleased to announce a call for papers for our upcoming special issue..." https://genomemedicine.biomedcentral.com/articles/collections/dissecting-brain-disorders
Emerging Evolutionary Computing Tools for Machine Learning Applications in Health Care "...This special issue [of Soft Computing: A Fusion of Foundations, Methodologies and Applications] aims to compile and unify all the novel approaches involving recent ECT advancements applied to ML seeking health care solutions globally. The Special Issue invites a large spectrum of recent advances in the field of nature-inspired techniques, machine learning algorithms for health applications in specific. Contributions pertaining to new techniques, novel evaluation methods and practices, variants of existing algorithms and challenges relevant to the theme..." https://www.springer.com/journal/500/updates/23603306
Thanks @riker1
I'm having the same issue after upgrading to the newest version of dada_mail-v11_20_0_stable_2022-10-03. It's not every message that's sent to the list, but it seems to be happening more frequently. Here is the lastest message that I'm getting this error on
I've tried testing both @riker1 and @edcoates messages, and they both process correctly. Thus, I'm having trouble recreating the problem itself. line 2545 in bridge is just a left curly brace, so yeah: not super helpful in pointing at what the problem is.
https://github.com/justingit/dada-mail/blob/main/app/dada/plugins/bridge#L2545
I may have to set up a test list for one of you to send messages to, to try to have the problem happen on a Dada Mail I have full control of. Either of you willing to send messages to a test list?
I can later today.. let me know the particulars.
Regards
Eric
Eric S. Tyrer II, MUP about.me/erics.t https://about.me/erics.t
(347) 393-6507
"The key is to keep company only with people who uplift you, whose presence calls forth your best.” —Epictetus
On Nov 14, 2022, at 2:13 PM, Justin J @.***> wrote:
I've tried testing both @riker1 https://github.com/riker1 and @edcoates https://github.com/edcoates messages, and they both process correctly. Thus, I'm having trouble recreating the problem itself. line 2545 in bridge is just a left curly brace, so yeah: not super helpful in pointing at what the problem is.
https://github.com/justingit/dada-mail/blob/main/app/dada/plugins/bridge#L2545
I may have to set up a test list for one of you to send messages to, to try to have the problem happen on a Dada Mail I have full control of. Either of you willing to send messages to a test list?
— Reply to this email directly, view it on GitHub https://github.com/justingit/dada-mail/issues/1135#issuecomment-1314250159, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKAZHIAGUOIKPK5PRT4ANLWIKFMPANCNFSM6AAAAAAROJB674. You are receiving this because you were mentioned.
@riker1 if you can just privately email me the email address you'll be using to send to the test list, I'll add you as an authorized sender. My email is listed at, https://dadamailproject.com/contact/
Let me know the particulars. I will email you the address I will be using.
Thanks,
Ed Coates
On Mon, Nov 14, 2022 at 1:13 PM Justin J @.***> wrote:
I've tried testing both @riker1 https://github.com/riker1 and @edcoates https://github.com/edcoates messages, and they both process correctly. Thus, I'm having trouble recreating the problem itself. line 2545 in bridge is just a left curly brace, so yeah: not super helpful in pointing at what the problem is.
https://github.com/justingit/dada-mail/blob/main/app/dada/plugins/bridge#L2545
I may have to set up a test list for one of you to send messages to, to try to have the problem happen on a Dada Mail I have full control of. Either of you willing to send messages to a test list?
— Reply to this email directly, view it on GitHub https://github.com/justingit/dada-mail/issues/1135#issuecomment-1314250159, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIENUBNZDMPFNSAXX3FFX73WIKFMPANCNFSM6AAAAAAROJB674 . You are receiving this because you were mentioned.Message ID: @.***>
One difference I have noticed is that when upgraded to this version it told me that it had found a newer version of perl at /usr/local/cpanel/3rdparty/bin/perl (v5.30.0) and I chose it. The installer made the appropriate changes for the files except for the extensions and plugins. They seem to still point to /usr/bin/perl which is version 5.16.3 on my system.
Not sure if that might be the issue, but thought it was worth noting.
Ed
Plugins aren't called directly ie - the URLs SHOULDN'T be this,
https://example.com/dada/plugins/bridge
but should look like this,
https://example.com/dada/mail.cgi/plugins/bridge
"/plugins/bridge" is just information given to the mail.cgi script in a cleverer way.
So, there shouldn't be a difference in Perl versions used in Dada Mail.
Did anyone have any trouble with the messages they sent?
No, but continue having the same problem with a few subscribers on my mailing list
Ed Coates
On Thu, Dec 8, 2022, 11:42 AM Justin J @.***> wrote:
Did anyone have any trouble with the messages they sent?
— Reply to this email directly, view it on GitHub https://github.com/justingit/dada-mail/issues/1135#issuecomment-1343074208, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIENUBJULRIFBVP7XDPHYSLWMIMXRANCNFSM6AAAAAAROJB674 . You are receiving this because you were mentioned.Message ID: @.***>
Do you have some sort of anti-spam appliance? We have ProofPoint anti-spam and I was wondering if the “safe links” feature is overloading the parser (tidy) what do you think?
Just an idea… not necessarily a great one
Bests -est2
Eric S. Tyrer II, MUP about.me/erics.t https://about.me/erics.t
(347) 393-6507
"The key is to keep company only with people who uplift you, whose presence calls forth your best.” —Epictetus
On Dec 8, 2022, at 12:42 PM, Justin J @.***> wrote:
Did anyone have any trouble with the messages they sent?
— Reply to this email directly, view it on GitHub https://github.com/justingit/dada-mail/issues/1135#issuecomment-1343074208, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKAZHNZNRSO6ZYNYHNWPATWMIMXRANCNFSM6AAAAAAROJB674. You are receiving this because you were mentioned.
Here’s the original message sender tried and it didn’t go through. I tried to also send it as I am an authorized sender and same failure. Here’s output I pasted into a file showing bridge trying to work on the message mail-sample.txt https://github.com/justingit/dada-mail/files/9863383/mail-sample.txt
I'm going to use this as an example, as it's easy to see what's going on.
There's unicode characters in this message. For example:
I also hope to see proposals on Ian Fleming’s novels, the history
and development of Bond
This all look innocuous enough, but the, "‘" character is not an ASCII single quote, ' but Unicode Character “′” (U+2032)
https://www.compart.com/en/unicode/U+2032
The message itself is supposed to be encoded in 7bit, which only allows ASCII characters.
In this case, the bug is perhaps with the software which made this email message, "Roundcube Webmail" - it should have used a different encoding like 8bit - or better still: quoted-printable. I've tried to recreate this problem with my own email address, sending with RoundCube, but it does the right thing and uses 8bit encoding. My version is, 1.3.15.
But nevertheless, Dada Mail should be able to handle something like this without die'ing and this is very much a bug in Dada Mail/bridge. Here's a test script that closely mimics what Dada Mail does internally:
#!/usr/bin/perl
use strict;
use warnings;
use Encode qw(encode decode);
use MIME::Parser;
my $msg = q{
From: example@example.com
To: example@example.com
Subject: Calls for contributors | Retransmit
Content-type: text/plain; charset="UTF-8"; format="flowed"
Content-Transfer-Encoding: 7bit
Though the volume will focus predominately on the
films, I also hope to see proposals on Ian Fleming’s novels};
my $parser = MIME::Parser->new;
$parser->tmp_to_core(1); # this actually won't change things
$msg = decode('UTF-8', $msg );
my $entity = $parser->parse_data($msg);
print $entity->as_string;
parse_data: unable to open in-memory file handle at untitled 12 line 24.
The only problem is this line,
$msg = decode('UTF-8', $msg );
Remove that, and the above doesn't fail to run. There's only one place in all of Dada Mail where this happens,
If you change that line to,
$msg = safely_decode( $msg );
Things may actually work a lot better! I can't imagine why it's written that way, but it's been that way for 8 years. There may be other things that need to be tweaked, but I'm working on it.
What an altogether difficult bug to track down, given a lot of the problems people are having are from invisible, or seemingly innocuous characters.
Dear Justin.
I tried to submit this possible bug via the mailing list but the captcha kept preventing me from 2x opting I apologize in advance if this duplicates that.
I have a discussion list configured to use Bridge to pick messages up from a POP3 box. A subscriber/member sent two messages: a short 4-sentence paragraph with a link, following message composed of 7 or so paragraphs with 8 or so URLs therein regarding sources looking for contributors.
The first, short message went through fine and went to archive as usual. The longer second message disappeared. Upon tailing postfix, syslog, and dada error log I found an error message:
Mailing List: YorkTalk (yorktalk)
Capabilities: PIPELINING: CAPA: SASL: PLAIN LOGIN USER: RESP-CODES: UIDL: AUTH-RESP-CODE: TOP:
APOP may NOT be supported. SSL Supported Authentication via POP.
POP3 Login succeeded!
Message count: 1
Message Size: 7.62K Received message is below maximum size allowed. Received message is below maximum soft size allowed.
_bridge - irrecoverable error processing message. Skipping message: parsedata: unable to open in-memory file handle at plugins/bridge line 2545.
Sending a copy of the message to the List Owner
Processing Digests:
Mailing List: YorkTalk (yorktalk)
Line 2545 in the bridge app is not very helpful as it points to various variables etc. I've tried switching from parsing messages on disk in .dada_files and using RAM and /tmp folder. Neither adjustment of those settings resulted in any positive change, either remediating the parsing problem and or sending the message out to the list.
I've included some output that I think may be useful mail-sample.txt
Any thoughts or suggestions would be greatly appreciated.
In health,
-est2