search

clonemeagain / plugin-fwd-rewriter

An osTicket plugin to rewrite incoming emails
GNU General Public License v2.0
9 stars 0 forks source link

Does it work for you? #1

Open clonemeagain opened 7 years ago

clonemeagain commented 7 years ago

Accidentally left an old repo up and @blueyeguy tried to use it, have you had more luck with this one mate?

I use a version of this in prod, I should probably just use this one and move my mods to my mod to their own mod! :-) :kiwi_fruit:

blueyeguy commented 7 years ago

I have it installed, and I think configured, but when I forward to the ticket system email address, it still has me as the forwarder, rather than the person on the ticket I am forwarding.

I don't see any errors in the PHP log or anywhere else, so I am not sure how to trouble shoot it.

clonemeagain commented 7 years ago

Sorry mate, again, didn't really pay much attention to it, shouldn't have pushed code that hadn't been tested.. feel like a jackass. Updated.

What version of osTicket are you using? This only works for 1.10+ To see things in the admin log, you need to tick the box "Show rewriting in logs:".. and actually that doesn't even work.. wtf? I'm looking into it.

Also, it seems that version was still broken.. sheeze. Sorry man. Updated, tested properly this time, it works. Latest commit/update is working on mine for fetched email in several different formats.. but I can't test forwards from all clients. What client are you forwarding from? (outlook/thunderbird/gmail etc?) If you can share the From: line for a few that dont' work, or submit a pull request to update the regex to match them, that'd be great.

A few caveats (that are now in README):

The debugging has been cranked all the way up for now, so, watch your webserver logs.. maybe test it a few times, if happy, change DUMPWHOLETHING to false.

blueyeguy commented 7 years ago

Yes we are 1.10

osTicket Version | v1.10 (901e5ea) Web Server Software | Microsoft-IIS/7.5 MySQL Version | 5.7.17 PHP Version | 5.6.24

I am forwarding from Outlook, all of our email accounts are exchange fetching IMAP

With the plugin enabled, my fetching stops completely for all our emails. There is no entry in the Admin Panel>Dashboard>System Logs but i get the following PHP error:

[20-Jul-2017 16:15:00 UTC] osTicket RewritePlugin: Bootstrappin.. [20-Jul-2017 16:15:05 UTC] osTicket RewritePlugin: Received signal ticket.create.before [20-Jul-2017 16:15:05 UTC] PHP Fatal error: Call to a member function get() on null in C:\inetpub\wwwroot\support\include\plugins\plugin-fwd-rewriter-master\class.RewriterPlugin.php on line 161

clonemeagain commented 7 years ago

I can see the problem, I'll be at the office in an hour and will have it patched asap.

Any other rewrites you'd find useful? I was considering a key:value replacements type thing, so matched email keys would get replaced with the new value. Or adding admin regex patterns to find the original sender, good for custom software emails.

Do you need anything like that?

clonemeagain commented 7 years ago

Actually, it's called Rewriter, maybe a find/replace function. For any text. Hmm. Could be useful.

clonemeagain commented 7 years ago

Playing with the regex rewriter, damn, too powerful!

Have to filter out things that we shouldn't touch, like attachments. Yup, we could actually rewrite/remove attachments if we want.. which could be good.

As a simple find/replace pattern system, it works well, good for swearwords or protecting internal device addresses, like device@internal.company.tld => device@company.tld etc. Probably other uses I haven't thought of yet. https://github.com/clonemeagain/plugin-fwd-rewriter/commit/9bc00f07e0d2b12170383744011416ab101adb22

I could add a rewrite filter to strip attachments from email addresses that match a pattern, or that contain a string or something.. doable. Also good for pranking people! Messages from Jim@company.com get every mention of cloud changed to mystery zone for instance.. lmao.

I'm sure I'm missing something, maybe messages from the API or Web could be included, then the rewriter could run over them and remove swearwords or something.. could be useful. See if anyone wants it.

blueyeguy commented 7 years ago

Sorry for the delay in responding, this is our busy season and I have not had the time to spend on this. The latest code does not kick out any errors for me, in any of my logs, but it also does not seem to make any changes to the tickets. Tickets are still created with the forwarder as the user, and not the sender of the email that was forwarded. I wish I knew a lot more about the code, but honestly when you asked if I could "submit a pull request to update the regex to match them", I honestly have no idea how to do that....

clonemeagain commented 7 years ago

Is it actually turned on? You can configure it while it's not even enabled. The manage -> plugins page lists all plugins and let's you enable them.

blueyeguy commented 7 years ago

Yes lol, I noticed that remark on yours when you said you have forgotten to enable it.

clonemeagain commented 7 years ago

And you've set the forwarders' domain as allowed? Maybe I should add a "rewrite all forwards" option.

blueyeguy commented 7 years ago

I am forwarding one from our own domain, but we are office365, so do I need the outlook.com domain and not our organizational domain?

clonemeagain commented 7 years ago

Ok, tested with an Office365 forward:

<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Arpwatch Server &lt;arpwatch@proxy.domain.local&gt;<br>

Classy. Maybe I need a new approach.. it seemed so easy when I started.

Latest commit has been tested with forwarded messages from Office 365, it works!

Had to rewrite a fair chunk. It's fairly accurate.. fairly.

blueyeguy commented 7 years ago

I appreciate all the time and effort you are putting into this. After installing this latest code, my website kicks out a http error 500 and i have the following in my php logs:

[02-Aug-2017 01:30:01 UTC] PHP Fatal error: Call to a member function logDebug() on null in C:\inetpub\wwwroot\support\include\plugins\plugin-fwd-rewriter-master\class.RewriterPlugin.php on line 556

clonemeagain commented 7 years ago

That should help mate, it has been logging up a storm on mine, but I forget not everyone has the bootstrap mod! I have this: https://github.com/osTicket/osTicket/pull/2907 merged in my code.. because I like to be able to use $ost in plugins. But I had forgotten it wasn't in core yet. :-)

blueyeguy commented 7 years ago

That has the site back up, but its still creating the email as the forwarder, not the original sender. PHP log indicates that it thinks I am forwarding my own email rather than someone elses:

[02-Aug-2017 01:50:00 UTC] osTicket RewritePlugin: Bootstrappin.. [02-Aug-2017 01:50:39 UTC] osTicket RewritePlugin: Received signal ticket.create.before [02-Aug-2017 01:50:39 UTC] osTicket RewritePlugin: Matched forwarded subject: FW: Interviews [02-Aug-2017 01:50:39 UTC] The forwarded message is from the same person, they forwarded to themselves? bailing

clonemeagain commented 7 years ago

Could you remove any confidential details and post an example email?

On Wed, 2 Aug 2017, 23:37 blueyeguy notifications@github.com wrote:

That has the site back up, but its still creating the email as the forwarder, not the original sender. PHP log indicates that it thinks I am forwarding my own email rather than someone elses:

[02-Aug-2017 01:50:00 UTC] osTicket RewritePlugin: Bootstrappin.. [02-Aug-2017 01:50:39 UTC] osTicket RewritePlugin: Received signal ticket.create.before [02-Aug-2017 01:50:39 UTC] osTicket RewritePlugin: Matched forwarded subject: FW: Interviews [02-Aug-2017 01:50:39 UTC] The forwarded message is from the same person, they forwarded to themselves? bailing

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/clonemeagain/plugin-fwd-rewriter/issues/1#issuecomment-319674306, or mute the thread https://github.com/notifications/unsubscribe-auth/AE15jwzQqvZx6rAP5aDGiwBujbogA3Dnks5sUHubgaJpZM4OcCj9 .

blueyeguy commented 7 years ago

I have done some additional troubleshooting/testing that might be helpful.

The primary scenario for our use would be the following: "user@ourdomain" sends to "agent@ourdomain" forwards to "support@ourdomain" = error above, forwarding to themselves.

This happens all the time where a user will email an agent directly rather than the support address, and we want to get those back into the ticket system for tracking, workflow, etc.

I tried the following scenarios:

"user@gmail" sends to "agent@ourdomain" forwards to "support@ourdomain" = success "user@ourdomain" sends to "user@gmail" forwards to "support@ourdomain" = success "user@gmail" sends to "user@otherdomain" forwards to "support@ourdomain" = success

so the problem seems to be when its all internal?

blueyeguy commented 7 years ago

Can I email you at clonemeagain@gmail.com

clonemeagain commented 7 years ago

Sure thing mate.

clonemeagain commented 7 years ago

Ok, I checked the message you sent (thanks!).

It appears a forward from inside an Outlook domain doesn't include the email address. Just the name. (Whether destination is internal or external, it appears to be by design, and unfixable). There is apparently a workaround where you'd forward the original as an attachment, however this plugin doesn't read attachments (yet, if this is the only way forward, then I might have to!).

Because of the inverted stack design of the plugin, it assumes that a From without an associated address isn't really a From, it could just be part of a sentence, so it goes back up the stack to try and find the previous sender, eg:

From: Me <me@there.com>
...
From: Them <them@somewhere.com>
..
From: Original Guy <zaphod@beeblebrox.yeah>

The intention would be to assign the ticket creator as Original Guy, not Me, or Them. HOWEVER, if Outlook is stripping internal email addresses, then they won't be recognized, so it goes back up the sender stack until it finds anything, then compares it.. in this case, it fails.

Outlook version of that:

From: ME
...
From: THEM
...
From: ORIGINAL GUY

Yes, it capitalizes for some reason.

Looks to be an Outlook issue:

It only affects internal email forwarded by Outlook, because Outlook knows best, and it knows that it can resolve the name, so it doesn't include the address. How helpful.

On my end, the plugin could initiate a User search based on the name, and assume that is the correct one.. "Name with no Email == internal user"..

How frustrating. How ripe for abuse. No wonder so many people get Phished/Scammed.

We've not encountered it at my work, because we're not a M$ shop.. very few (two) outlook users in the whole company. Most of us either use the G-Suite web interface, Thunderbird, or mobile clients. Or, you know, tickets.

I'll have a think about the best way to fix it, maybe research it some more.. there must be a way.

blueyeguy commented 7 years ago

I feel bad for having you chase your tail on this so much. You have put in quite a bit of time trying to make it work. At least we know for sure now the problem is our environment. We switched over to M$ from Lotus notes about 5 years ago, and it's been an interesting ride ever since. Thanks again for all the time and effort you spent.

clonemeagain commented 7 years ago

It's all good man, I'm just glad someone is using my code. Hard to know on here of anyone is using it, you only really see when another developer makes a fork or if someone with an account stars the repo.

I'm happy we figured it out though, and that I hadn't made some stupid mistake that broke your system.

If the original sender is an agent in the ticket system I have code from another module that can dig their details out if you like. Check it out here: https://github.com/clonemeagain/osticket-plugin-mentioner I could add that logic to this with an admin toggle: "Use Outlook mode for staff forwards", despite the issue, we can still make it work. Kinda depends on all forwarders being agents already. Is that something that would be useful? That's what I meant by name search.

blueyeguy commented 7 years ago

We use the mentioner one right now :-) or at least we are testing it to see if everyone finds it useful. Its handy when needing to include another agent in a message or add them as a collaborator. The response to it from the agents so far has been positive.

perryk commented 7 years ago

I like this idea of this plugin, however, I think I'd only want to enable it for emails forwarded from agents or even just from myself (as an agent also).

Also, agents and staff have the same email domain here so I don't think I can use the "Forward Rewritable Domains" option.

Cheers.