search

cardi / proofpoint-url-decoder

tools to mess around with proofpoint URLs
Creative Commons Zero v1.0 Universal
19 stars 4 forks source link

Extra space in urldefense v3 #10

Open GiovanniSalmeri opened 2 months ago

GiovanniSalmeri commented 2 months ago

When decoding emails with URLs mangled with urldefense v3, there is a little glitch: urldefense seems to add always a space after the URL, but this space is kept by decode_email.py. For example, if someone sends an email with <https://example.com>, the decoded email will have <https://example.com >. This is not absolutely a big deal, but perhaps the solution is easy enough.

Thank you very much for this very useful piece of software!

cardi commented 1 month ago

Thanks for the issue!

I have noticed the extra space in demangled URLs and from a couple emails I've checked, I've verified that in some cases there is an extra space inserted in emails that use quoted-printable encoding (i.e., Content-Transfer-Encoding: quoted-printable) with a Content-Type of `text/plain.

The extra space does not seem to be an issue with emails that are Base64-encoded or of Content-Type text/html, but I don't have many Base64-encoded examples to work with.

Two cases I've seen so far:

  1. With angle brackets: <https[:]//urldefense.us/v2/url?u=[...]&e=3D >
  2. Without angle brackets: https[:]//urldefense.us/v2/url?u=[...]&e=3D=20

=3D is =, and =20 is the space character: the space is encoded (escaped) if it is the last character of the encoded line.

(There are probably additional variants to these.)

Given that context, because the decoder does not process URLs (or emails) in their quoted-printable-encoded form, it will miss case (2), but ultimately the decoded URL does not look visibly off, as the URL is on a line by itself. (Even if we do process quoted-printables, we would still miss case (1).)

I think handling case (1) (which is much more noticeable) in a general way will require some careful thought, because it's not clear to me how to delimit a URL surrounded by angle brackets (<, >) (or other delimiters), as the space character is not part of the URL (if it was, it should be encoded as %20), without having to handle many different edge cases.

Thus, for now, I don't have a good fix for this issue, but I'll happily consider your (or anyone else's) suggestions if you have one.

GiovanniSalmeri commented 1 month ago

Thank you very much for looking into this small issue! I understand that a cure could easily be worse than the disease, and anyway this is a bug of urldefense, not of your decoder. I could add an ugly hack to my pipeline (for example something like s/([^>]) >/\1>/g), which probably would have few side-effects, but since the issue is only cosmetical perhaps I could better leave all as it is. I will come back to you if I have a more sensible idea!