search

tc39 / tg5

TC39-TG5: Experiments in programming language standardization. https://ecma-international.org/task-groups/tc39-tg5/
19 stars 1 forks source link

User feedback survey for MessageFormat #3

Open echeran opened 6 months ago

echeran commented 6 months ago

Hi, I'm curious if your group would be interested in designing a user feedback survey/form/study that would help the MessageFormat specification.

We currently have a technical preview version of the spec in CLDR's LDML v45.

We have technical preview implementations in ICU v75 to match, both in C++ (ICU4C) and Java (ICU4J). ICU & CLDR have releases 2 times a year, and new APIs in ICU typically go through a process of stages: technical preview (for 1+ releases), beta (at least a few releases), final (if no complaints received in the last few years of beta).

There's also an ECMA-402 proposal for Intl.MessageFormat. The ECMA-402 proposal is of course tied to having a stabilized spec, which needs feedback from users to weigh in on design decisions.

It would really help to figure out from users of any technical preview implementations of the current version of the new MF spec what they think. Ex: what features are useful? not useful? easy? difficult? etc. Also, in order to rush and meet the deadline for the spec along with ICU implementations ready for the joint CLDR v45 / ICU v75 release, some decisions were postponed for the post CLDR v45 period, or we took a short term compromise resolution. So now is a good time to revisit, reflect & collect feedback. It would confer more clarity & confidence to take forward our future work.

cc @aphillips @eemeli @mihnita @stasm @catamorphism

codehag commented 6 months ago

This would be potentially interesting and a good way to evaluate the MessageFormat syntax as part of the MessageFormat proposal in TC39. As you mentioned, the 402 proposal is currently blocked on the user evaluation, but a well designed survey could help in soliciting and evaluating that feedback. Since it has an impact on a TC39 specification, I believe it is in scope.

codehag commented 6 months ago

@echeran Do you have a list of questions that you would want to answer? This would help when thinking about the study.

echeran commented 6 months ago

Good question. Here's a first attempt. I first have to remind myself of who we think the target audience of interested people are, and as we see it, that group is potentially diverse in their backgrounds:

Some questions that come to mind:

cc @mradbourne who also might be interested

codehag commented 5 months ago

We discussed this potential study as part of TG5's meeting last Wednesday. Michael Coblenz had some good suggestions and a few questions. We'd like to schedule a breakout meeting in about 3 months time. What does your availability look like and are there others who would be interested in joining?

Prior to this meeting, we have a leading question. Rather than start with questions you would ask potential survey memebers, we want to focus on the goal. For you as designers, what do you want to know? What do you want to verify?

echeran commented 5 months ago

I should be around and available to meet in 3 months time, unless it's during the 4th week of October. Did you mean 3 months, or 3 weeks?

Asking about the goal is a good question to start from. At a high level, we want to gather feedback for the API design & behavior described by the spec text. As designers, we've had disagreements when coming to decisions (& sometimes after decisions were made). Sometimes that boiled down to a disagreement of what the requirements are, or what the evaluative criteria are to decide among the options. Ex: different people filling in the sentence, "I think people using this feature would prefer ____" with different answers. Without having some type of data / feedback, it's hard to really evaluate those arguments, even when we sort out the alternatives.

So I'd like to know how people perceive the new API design, in order to have more tangible insights. If nothing else, I think it would allow us to view feedback all together, since I still find it mysterious whose feedback tends to make how much of an impression on the group's focus and why.

Of course, we can get into specifics of technical decisions and ask people how they like the syntax, what they think about the function registry, and the various things in the spec that are required & optional to implement. Maybe the linguistic features that they would like their messages to support that they have problems with right now. And we can get to even more granular details, like their expectations on syntax whitespace handling, expectations on composing functions in a message, error handling behavior, function registry schemas, etc.

I wasn't sure what level of specificity you were asking, so I tried to cover a wide range. I hope an answer to your question is somewhere in there... Other people might be interested, so I should ask in our next meeting to find out.

codehag commented 5 months ago

sorry, i had meant 3 weeks! What is your availability in the coming weeks?

echeran commented 5 months ago

Here's a calendar time planning doodle thing for next week. I'll be on vacation for the 3 weeks following that, though. Once I'm back, I should be around with no major planned absences for a while.

mcoblenz commented 5 months ago

I replied to the poll. Hopefully we can find a time that works.

sffc commented 5 months ago

I added my availability; it's a bit limited due to time zones but you don't need to consider me required!

mcoblenz commented 5 months ago

There aren’t any times that work for us all this week. Closest is Thursday 2:30 - 3:30, but Eemeli isn’t available then. Do we want to meet anyway or defer until after Elango is back from vacation?

Michael

On Jun 24, 2024, at 8:48 AM, Shane F. Carr @.***> wrote:

I added my availability; it's a bit limited due to time zones but you don't need to consider me required!

— Reply to this email directly, view it on GitHub https://github.com/tc39/tg5/issues/3#issuecomment-2186885859, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACD4YD6J6UI5RVSHANFMYSLZJA5TZAVCNFSM6AAAAABHL53TX6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBWHA4DKOBVHE. You are receiving this because you are subscribed to this thread.

codehag commented 5 months ago

Since im in CEST its a little hard for me to attend, but since I am not critical for this meeting beyond organization feel free to exclude me.

mcoblenz commented 5 months ago

Would it be better to defer until after Elango gets back from vacation? I’ll be available then too.

Michael

On Jun 26, 2024, at 8:17 AM, yulia @.***> wrote:

Since im in CEST its a little hard for me to attend, but since I am not critical for this meeting beyond organization feel free to exclude me.

— Reply to this email directly, view it on GitHub https://github.com/tc39/tg5/issues/3#issuecomment-2191972423, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACD4YD24S2VK73DMGNEEH2DZJLLPVAVCNFSM6AAAAABHL53TX6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJRHE3TENBSGM. You are receiving this because you are subscribed to this thread.

echeran commented 5 months ago

I don't mind touching base for 30 mins tomorrow before vacation, just so I can get a better sense of what I should do to be properly useful here.

I set up a Meet meeting at 10am Pacific Time tomorrow (6/28) if that works, happy to move the time around, or we can just defer until 3 weeks later.

On Monday in our weekly meeting, @eemeli mentioned that you would be interested in knowing specific items that we're wanting feedback on. @aphillips pointed out that our wiki page "Things that Need Doing" has a section pointing to the repo issues that are marked as being feedback on our Technical Preview implementation.

mcoblenz commented 5 months ago

I can do 30 minutes tomorrow but only 30 minutes, since I have another meeting at 11. Where can I find the meeting link?

Michael

On Jun 27, 2024, at 2:58 PM, Elango Cheran @.***> wrote:

I don't mind touching base for 30 mins tomorrow before vacation, just so I can get a better sense of what I should do to be properly useful here.

I set up a Meet meeting at 10am Pacific Time tomorrow (6/28) if that works, happy to move the time around, or we can just defer until 3 weeks later.

On Monday in our weekly meeting, @eemeli https://github.com/eemeli mentioned that you would be interested in knowing specific items that we're wanting feedback on. @aphillips https://github.com/aphillips pointed out that our wiki page "Things that Need Doing" https://github.com/unicode-org/message-format-wg/wiki/Things-That-Need-Doing has a section pointing to the repo issues https://github.com/unicode-org/message-format-wg/issues?q=is%3Aissue+is%3Aopen+label%3APreview-Feedback that are marked as being feedback on our Technical Preview implementation.

— Reply to this email directly, view it on GitHub https://github.com/tc39/tg5/issues/3#issuecomment-2195727143, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACD4YD3MYPLNXHRSQ4FPJJDZJSDIZAVCNFSM6AAAAABHL53TX6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJVG4ZDOMJUGM. You are receiving this because you are subscribed to this thread.

echeran commented 4 months ago

Here it is (forgot to paste it): meet.google.com/tuf-eupk-qsg

mradbourne commented 4 months ago

Hi all. Hope the call went well. What was the outcome? Any next steps?

mcoblenz commented 4 months ago

On one hand, the general problem space is very interesting! Here we have a programming language that targets localizers, who may not have much programming background. The programming language needs to be nontrivial (there are branches, variables, etc.) but also safe (localizers should never be surprised by a string that is generated by their programs). It needs to support a wide variety of natural languages and be embeddable in many different text-based programming languages. Definitely interesting.

But on the other hand, it seems that most of the major decisions (for which research could be impactful) have already been made. So I worry that a research program here would have only limited practical impact.

One way I could imagine starting would be to do a usability study of the current design and see what challenges people face when using it. This could give some evidence about what will happen if MessageFormat 2 is handed to people to use, and give the team some high-level points to consider. We could use tasks that are relevant to some of the current design questions. I see some discussion about variable binding (allowing re-binding or not), unquoted literals, errors that can result, and details about how matching works. Then, once that’s done, we can use the results to either refine the design or to generate quantitative experiments about specific design decisions.

Does this direction seem like it would be of interest to the team?

On Jul 22, 2024, at 3:10 AM, Matt Radbourne @.***> wrote:

Hi all. Hope the call went well. What was the outcome? Any next steps?

— Reply to this email directly, view it on GitHub https://github.com/tc39/tg5/issues/3#issuecomment-2242589365, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACD4YD77N6ZR2IYS7GTOKEDZNTLDFAVCNFSM6AAAAABHL53TX6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBSGU4DSMZWGU. You are receiving this because you are subscribed to this thread.

echeran commented 3 months ago

Yes, I think this sounds interesting. It would great to focus it around translators since we are designing with them in mind, and yet they're obviously not trying to participate in the nitty gritty technical discussions of the Working Group.

Let us know what scope of work is possible for your group to take on. I know there might be a larger scope of study if there is funding available. My management isn't able to get funding for that, but if another company is interested, there is an opportunity to step up.

mcoblenz commented 2 months ago

Right now I'm looking at doing a small usability study to see what the user experience is like for translators. Do you have a good sense of what sources I could use for identifying appropriate tasks?

sffc commented 2 months ago

But on the other hand, it seems that most of the major decisions (for which research could be impactful) have already been made. So I worry that a research program here would have only limited practical impact.

I don't think a perceived risk of "limited practical impact" should get in the way of the study.

One way I could imagine starting would be to do a usability study of the current design and see what challenges people face when using it.

Yes, this is what I was thinking.

I'd like to focus on the first two constituencies from https://github.com/tc39/tg5/issues/3#issuecomment-2130095739: Web developers and translators. Given that this study is in TC39, Web developers should be the primary focus, although it is good to get feedback from translators, too.

I want to understand:

  1. Does MF2 serve the needs of Web developers?
  2. How does MF2 fit into the workflow of Web developers?
  3. An ECMAScript built-in Intl.MessageFormat is being proposed; is this primitive useful for Web developers?
  4. What roadblocks might prevent Web developers from transitioning from their current solution to the standard MF2 if it were added to ECMAScript?

Basically, there were concerns from TC39-TG1 that we might standardize a syntax that might suboptimally serve the needs of Web developers. If we can show that MF2 serves their needs well, then it would strengthen the Intl.MessageFormat proposal. Likewise, if we get feedback that it doesn't serve their needs, we can bring that back to the MF WG.

aphillips commented 2 months ago

The challenge here is the tension between Web developers and the needs of localizers (which includes translators, but also includes the people responsible for the target language product, which can involve engineering).

Web developers have many tools for inserting values into application strings and user interface elements. Many (maybe even most?) of these DSLs, templating languages, etc. are not well internationalized and do not take into account the need for localized presentation, grammatical matching, and the like.

MF2 is a partial solution to this, in that it is intended to provide a syntax for the creation of messages and an API for developers. It is incomplete because it probably should be implemented as part of a resource management mechanism (such as the proposed MessageResource format). However, it is also intended to work cross platform and support the needs of many programming languages and runtimes, including those that already have a resource management format/APIs.

Tasks to think about in a usability study should thus focus on the problems that Web developers and localizers both face in creating an application. Don't just do English, consider Polish, Japanese, Turkish, French, and Arabic while you're at it, including presentation appropriate for countries other than the USA (e.g. in France they don't use 12 hour time). Here are a few:

Now change formatting details in and/all of the above messages, e.g. making 54.3% into 54% or Sept. 13, 2024 into 09/13/2024.

I agree that the solution has to be attractive to Web developers (they have to want to use it). Compromises in usability for them should be limited to making it harder to write outright I18N bugs, IMO.

mikbar-uib commented 2 months ago

FYI: the agenda of the upcoming TG5 meeting on Wednesday 25th September 2024 has an item to discuss this study.

mikbar-uib commented 3 weeks ago

FYI: at the upcoming TG5 meeting on Wednesday 30th October 2024, @shumbo will share the draft of the task design for the user study.

echeran commented 3 weeks ago

@shumbo Add a link to your slides when you get a chance. Perhaps you want to wait until you collect feedback and can revise accordingly?

As promised, below are some examples from the MFWG of discussions on design decisions that were difficult enough to discuss that they led to balloting to arrive at a decision, as well as their context. Also, keep in mind that you can see a wider set of design decisions in MFWG that warranted a design doc in this folder, whether or not they led to a ballot.

mikbar-uib commented 3 weeks ago

@shumbo Add a link to your slides when you get a chance. Perhaps you want to wait until you collect feedback and can revise accordingly?

A link to the slides is mentioned now in the meeting agenda: https://github.com/tc39/tg5/blob/main/agendas/2024/2024-10-30.md