Front end text length validation is more strict than back end

[seancolsen]

Summary

In #1080 we made TextBoxCell.svelte and TextAreaCell.svelte set the maxlength attribute based on the field length from Postgres. This change introduces a problem (which I previously described in #1057 as a hypothetical) where the client and server use slightly different algorithms to perform the same validation.

Steps to reproduce

1. Execute the following SQL on the Mathesar db:

   CREATE TABLE "emojis" (
       "id" serial NOT NULL,
       "emoji" VARCHAR(6),
       CONSTRAINT "emojis_pk" PRIMARY KEY ("id")
   );
   INSERT INTO "emojis" ("emoji") VALUES
   (E'\uD83D\uDC69\u200D\u2764\uFE0F\u200D\uD83D\uDC69');

2. Load the emojis table in the Mathesar UI.

3. Observe a cell displays:

   👩‍❤️‍👩

4. On https://emojipedia.org/couple-with-heart-woman-woman/ click "Copy" (to be extra sure you have exactly this emoji copied into your clipboard).

5. Edit the cell. Delete all contents. Paste the copied value into the cell.

6. Expect the cell to display:

   👩‍❤️‍👩

7. Observe the cell displaying the following (because the value has been truncated).

   👩‍❤️‍

8. At this point the UI makes it impossible to re-enter the data that was originally stored in Postgres because the front-end validation is more strict than the back-end validation. JavaScript says that string has length 8. Postgres says it has length 6.

Additional considerations

• The Zero-width joiner is likely the cause of the discrepancy in length calculation. In addition to emoji's it's also used in some natural languages.
• There may be other unicode code points which lead to string length discrepancies between JavaScript and Postgres.

Desired behavior

This is a bit tricky. Here are a few options.

Option (01): Server-side validation when saving a cell

Allow the user to enter values into the UI that are too long. Wait until they save the cell. Receive an error from the API. Display the error to the user, indicating that their value is too long.

Option (02): Dual validation with improved front-end logic

Figure out a way to calculate string length in JavaScript that reliably matches string length in Postgres.

Option (03): Server-side validation while typing

This approach just feels ugly, but I put it in here for the sake of brainstorming on how to solve this problem.

1. Create a dedicated endpoint like /api/validation/length/.

2. Send requests to that endpoint like /api/validation/length/?value=foo

3. Get responses like

   {
     "length": 3
   }

4. Hit this API as the user types, with some debouncing similar to an autocomplete. Turn the cell red and display an error message of the length is too long.

[seancolsen]

My opinions:

In #1057, I initially argued for (01).

However, after considering the UX in more detail I'll admit that the hardline approach of (01) lacks pragmatism. Specifically, in a situation where the user has entered a value that's too long, the user needs to have a clear understanding of where the limit is within their text so that they can shorten it. This is like composing a tweet. The UX considerations make me lean more towards (02) or (03).

I'm inclined to spend some time trying to get (02) to work. I'd like to do a bit of research and testing. If it's not viable, then maybe pursue (03).

[seancolsen]

@kgodey I put this in [Beta] Better Editing Experience so that we don't have to worry about it until after Alpha. Even though I think this is important to address at some point, I think that users will be unlikely-enough to run into the bug that we can safely deprioritize it for now.

[kgodey]

@kgodey I put this in [Beta] Better Editing Experience so that we don't have to worry about it until after Alpha. Even though I think this is important to address at some point, I think that users will be unlikely-enough to run into the bug that we can safely deprioritize it for now.

Makes sense.

[github-actions[bot]]

This issue has not been updated in 90 days and is being marked as stale.