[HOLD for payment 2023-04-03] [$1000] HTML characters are converted in chat (they shouldn't be converted)

[kavimuru]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Action Performed:

1. Enter < or any other HTML character in the compose box
2. Click on send

Expected Result

<should remain the same (i.e. we don't want to convert HTML characters)

Actual Result

< becomes the < sign

Workaround:

Unknown

Platforms:

Which of our officially supported platforms is this issue occurring on?

• [x] Android / native
• [x] Android / Chrome
• [x] iOS / native
• [x] iOS / Safari
• [x] MacOS / Chrome / Safari
• [x] MacOS / Desktop

Version Number: 1.2.78-0 Reproducible in staging?: y Reproducible in production?: y If this was caught during regression testing, add the test name, ID and link from TestRail: Email or phone of affected tester (no customers): Logs: https://stackoverflow.com/c/expensify/questions/4856 Notes/Photos/Videos:

https://user-images.githubusercontent.com/43996225/221594543-a616c80e-1b39-4c57-b9b7-2dfdc2f0f3d8.mov

https://user-images.githubusercontent.com/43996225/221594593-0ce68ce7-dc75-49b4-b19a-118429c75a92.mp4

Expensify/Expensify Issue URL: Issue reported by: @esh-g Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1677316598380029

View all open jobs on GitHub

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~012cc7a7ec930277c6
• Upwork Job ID: 1631456392901574656
• Last Price Increase: 2023-03-14

[MelvinBot]

Triggered auto assignment to @jliexpensify (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[MelvinBot]

Bug0 Triage Checklist (Main S/O)

• [x] This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
• [x] This bug is not a duplicate report (check E/App issues and #expensify-bugs)
  • If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
• [x] This bug is reproducible using the reproduction steps in the OP. S/O
  • If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
  • If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
• [x] This issue is filled out as thoroughly and clearly as possible
  • Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
• [x] I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

[jliexpensify]

I don't think this is a bug as typing < should convert to <

Querying here.

[jliexpensify]

Asking our wider engineering team for a second opinion, thanks for your patience!

[aldo-expensify]

Somewhat related to: https://github.com/Expensify/App/issues/15131

[jliexpensify]

Updated the Slack thread, sending to Upworks.

[MelvinBot]

Job added to Upwork: https://www.upwork.com/jobs/~012cc7a7ec930277c6

[MelvinBot]

Current assignee @jliexpensify is eligible for the External assigner, not assigning anyone new.

[MelvinBot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @s77rt (External)

[MelvinBot]

Triggered auto assignment to @NikkiWines (External), see https://stackoverflow.com/c/expensify/questions/7972 for more details.

[tienifr]

Proposal

Please re-state the problem that we are trying to solve in this issue.

If the message has HTML entities like <, &nbsp, it's not showing as is in the chat.

What is the root cause of that problem?

This is because we're unescaping HTML entities when displaying the message in the chat in here https://github.com/Expensify/App/blob/ce229bc1812ded576ce29d007fa3c03be8d1cd75/src/pages/home/report/ReportActionItemFragment.js#L123. The message was not properly encoded before so this decoding means the HTML entities in our message will be unexpectedly changed.

What changes do you think we should make in order to solve the problem?

We can htmlEncode the message when we parse it to HTML right after this line https://github.com/Expensify/expensify-common/blob/2e6d46d1c6227974f27475aa00d8cd7963bdde8b/lib/ExpensiMark.js#L308 We can add replacedText = Str.htmlEncode(replacedText);, then the message will be html-encoded and subsequently decoded properly when we display it.

What alternative solutions did you explore? (Optional)

We should remove Str.htmlDecode in https://github.com/Expensify/App/blob/ce229bc1812ded576ce29d007fa3c03be8d1cd75/src/pages/home/report/ReportActionItemFragment.js#L123 and other places if needed (basically any places that display the message should not be using Str.htmlDecode)

[bernhardoj]

Proposal

Please re-state the problem that we are trying to solve in this issue.

Text with HTML entity is decoded on the chat item.

What is the root cause of that problem?

We are decoding the text here:

https://github.com/Expensify/App/blob/2241edada07a73a6771a6fe5d74e294082222bd2/src/pages/home/report/ReportActionItemFragment.js#L121-L127

Why are we decoding it?

It is because when we send a chat, the chat we entered will be safely escaped/encoded. For example, if we send & &, it will become & & and will be decoded later into & &. As you can see, the final text is different from what we enter before.

& &amp -> & & ❌

Here is the expected and actual escaped comment:

Expected: & & Actual: & &

What changes do you think we should make in order to solve the problem?

We have 2 options here:

1. Replace the safe escape with just an escape. https://github.com/Expensify/expensify-common/blob/fff304449214480065bf550e562ad39e681f6f95/lib/ExpensiMark.js#L326-L327

I'm not totally sure about this first option because expensimark may be used on another project as well and I have no context on why it is safely escaped.

2. Escape the text we want to add before passing it to the parser here. https://github.com/Expensify/App/blob/9953eb896f60b676efd27541564a66326cbbcd34/src/libs/ReportUtils.js#L842-L845

[da7a90-backup]

Proposal

Please re-state the problem that we are trying to solve in this issue.

Messages containing raw HTML entities/characters are formatted after sending, they should be displayed as is.

What is the root cause of that problem?

Here Str.htmlDecode(text) decodes HTML entities within props.fragment.text https://github.com/Expensify/App/blob/ce229bc1812ded576ce29d007fa3c03be8d1cd75/src/pages/home/report/ReportActionItemFragment.js#L123

This is why they are rendered.

What changes do you think we should make in order to solve the problem?

Either: -remove Str.htmlDecode(text) and replace with just text -if it's still needed either add a isText parameter to the definition of the function with default false and return text as is if provided true -change the level or scope of decode here https://github.com/Expensify/expensify-common/blob/main/lib/str.js#L92

What alternative solutions did you explore? (Optional)

I explored encoding the message again like another proposal suggests but that would be redundant and doesn’t address the root issue

[MelvinBot]

📣 @da7a90-backup! 📣

Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork. Please follow these steps:

1. Get the email address used to login to your Expensify account. If you don't already have an Expensify account, create one here. If you have multiple accounts (e.g. one for testing), please use your main account email.
2. Get the link to your Upwork profile. It's necessary because we only pay via Upwork. You can access it by logging in, and then clicking on your name. It'll look like this. If you don't already have an account, sign up for one here.
3. Copy the format below and paste it in a comment on this issue. Replace the placeholder text with your actual details.

Format:

Contributor details
Your Expensify account email: <REPLACE EMAIL HERE>
Upwork Profile Link: <REPLACE LINK HERE>

[da7a90-backup]

Contributor details Your Expensify account email: sidbarrack@gmail.com Upwork Profile Link: https://www.upwork.com/freelancers/~018a8adf30b1e9aea5

[MelvinBot]

✅ Contributor details stored successfully. Thank you for contributing to Expensify!

[s77rt]

@tienifr Thanks for the proposal! I think the RCA is not correct.

[s77rt]

@bernhardoj Thanks for the proposal! Same thing ^ the RCA is not correct.

[s77rt]

@da7a90-backup Thanks for the proposal. Same same ^^ the RCA is not correct.

[s77rt]

@jliexpensify I think this is intended to prevent double-escape. See safeEscape.

cc @yuwenmemon @tgolen since you added this here https://github.com/Expensify/expensify-common/pull/97. Do you think we should allow double-escape?

[tgolen]

It's been a while, and I think that code was primarily used in OldDot. Looking at this bug report though, I think it's a feature and not a bug. If someone is copy/pasting a bunch of encoded text, I think we would want to convert it.

[esh-g]

@tgolen, But what if someone is sharing some HTML page code with these encodings? We would not want to convert in that case... And moreover, every platform like whatsapp and slack also don't convert

[tgolen]

But what if someone is sharing some HTML page code with these encodings?

Is there something wrong with that?

[esh-g]

Well, It is better for consistency because the browser should always get < instead of < and we do skip escaping characters like   so I see no reason why < should be an exception

[yuwenmemon]

Fun fact: I've separated each word of this sentence with an  

While I understand the viewpoint that what you type into the input is what should be displayed back to you. It's certainly not a paradigm that's universal across the internet.

I do think that it shouldn't be escaped within a code snippet, which is currently happening:

[yuwenmemon]

Some context on safeEscape...

I believe this was due to inconsistencies between Expensify and certain 3rd party integrations that were sending us categories with HTML entities already escaped.

So a category like Yuwen & Tim might be Yuwen & Tim from one integration and Yuwen & Tim in another (i.e. already escaped). We did want to store all categories with HTML entities escaped. However if we just used escape the latter category would end up displaying as Yuwen & Tim (encoded as Yuwen & Tim)

[tienifr]

I believe this was due to inconsistencies between Expensify and certain 3rd party integrations that were sending us categories with HTML entities already escaped.

Thanks for the context @yuwenmemon , I think if this is the case we should:

• Handle escape/not escape at the integration level, for example if an integration sends already-escaped HTML entities then we'll not escape, if an integration sends not-escaped HTML entities then we'll escape. We shouldn't just use a general safeEscape to handle all those integrations because it can cause side effect like this issue.
• Then we should use only normal escape in https://github.com/Expensify/expensify-common/blob/2e6d46d1c6227974f27475aa00d8cd7963bdde8b/lib/ExpensiMark.js#L308 if the user types text in the composer

Considering that other major chat apps like Slack, WhatsApp show the typed text as is, I think we should do the same as well.

[s77rt]

Not overdue. Still discussing if we want to fix this...

[NikkiWines]

Re-assigning to @yuwenmemon (discussed 1:1) as he has more context here

[s77rt]

@yuwenmemon Any updates on how to proceed here?

[pecanoro]

Let's put a HOLD on this issue until this PR https://github.com/Expensify/App/pull/15340 gets merged.

[pecanoro]

Actually, this one seems unrelated as it's encoding all messages not only IOU.

[yuwenmemon]

Right @pecanoro but we should probably mirror the behavior of the input in that PR. What do you think?

[pecanoro]

Yes, I agree!

[s77rt]

@yuwenmemon https://github.com/Expensify/App/issues/15508#issuecomment-1453267179

[yuwenmemon]

@s77rt based on the fact that we want to mirror the other inputs, double-escaping makes sense here.

[s77rt]

@yuwenmemon in that case, should we close the issue?

[yuwenmemon]

@s77rt Sorry to clarify I mean that HTML characters should not be converted in chat. Since if we look at the other PR, we're not going to escape them in names, IOUs, etc.

[s77rt]

@yuwenmemon Alright, so what's holding us from modifying safeEscape to return _.escape(s);

[MelvinBot]

@yuwenmemon @s77rt @jliexpensify this issue was created 2 weeks ago. Are we close to approving a proposal? If not, what's blocking us from getting this issue assigned? Don't hesitate to create a thread in #expensify-open-source to align faster in real time. Thanks!

[yuwenmemon]

Sorry, @s77rt - nothing is holding us! Let's pick a proposal. From the looks of it I'd say @bernhardoj is describing what we'd want to do here - do you agree?

[s77rt]

@yuwenmemon Can you revert the price back to $1000? I think none of the proposals got the right RCA but @bernhardoj proposed the right solution. Is that a blocker? if not let's hire him!

[yuwenmemon]

yeah the RCA thing was a bug - and agreed to revert to $1000. Thanks!

[MelvinBot]

Upwork job price has been updated to $1000

[MelvinBot]

📣 @bernhardoj You have been assigned to this job by @yuwenmemon! Please apply to this job in Upwork and leave a comment on the Github issue letting us know when we can expect a PR to be ready for review 🧑‍💻 Keep in mind: Code of Conduct | Contributing 📖

[bernhardoj]

PR is ready for review.

cc: @s77rt

[s77rt]

@bernhardoj I think you implemented the solution in the wrong place, I'm expecting an upstream fix on expensify-common. With your current solution we have something like _.escape(_.unescape(_.escape(s))) which is redundant, it should be simply _.escape(s).

[bernhardoj]

@s77rt sorry if I misunderstood, but based on @yuwenmemon comment here https://github.com/Expensify/App/issues/15508#issuecomment-1466150996, it's okay to double escape it because of this case https://github.com/Expensify/App/issues/15508#issuecomment-1454067374

[s77rt]

@bernhardoj I see we are a bit lost in the terms here :sweat_smile:

1. Allow double escaping: by calling _.escape twice
2. Allow double escaping: by not preventing it in the first place i.e. do not call _.unescape

Both approaches will fix the issue, I just think that 2 is more straightforward cc @yuwenmemon thoughts here? Option 1 requires changes on E/App only while Option 2 requires an upstream fix (expensify-common).

[yuwenmemon]

Yes, @bernhardoj we're looking for you to do what you said here:

Replace the safe escape with just an escape. https://github.com/Expensify/expensify-common/blob/fff304449214480065bf550e562ad39e681f6f95/lib/ExpensiMark.js#L326-L327 I'm not totally sure about this first option because expensimark may be used on another project as well and I have no context on why it is safely escaped.