mlte-team / mlte

An infrastructure for evaluating machine learning models.
http://mlte.rtfd.io/
MIT License
10 stars 3 forks source link

Re-evaluate Report Form structure #463

Open sei-aderr opened 2 months ago

sei-aderr commented 2 months ago

"Take do a side by side look at the negotiation card, and the report form and see if it would make sense to mirror them as there are several inconsistencies.

Some are easy and expected fixes, like ML Task in the negotiation card has helper text, where the field is Task in the report without helper text. But others like ordering of fields and content grouping under certain headers is different. So see if those groupings are still the way that you would like them to be or if they should line up more with each other"

Sub task - Add QAS and Dev Resources to report form as continuation from #459

sei-aderr commented 2 months ago

@GALewis

Please specify where you would like the QAS and dev resources fields form the negotiation card to be added in the report. Then follow what I mentioned above about looking for incostencies between the negotiation card and report forms and see if you want them to stay as-is or become more consistent

GALewis commented 2 months ago

OK, I totally see what you mean about overlap. From a process perspective the report should not be editable. You should only be able to "View" or "Export". For View, the information should be more like a report and less like a form. I recommend the structure to be:

Does this make sense?

sei-aderr commented 2 months ago

I am open to the idea of most of those changes, but several decisions have to be made regarding them.

  1. While nearly all of the report could be imported from other sources, how would the comments field be handled? Currently the only way to add to it is through the form. There is also the quantitative analysis field that is not populated currently but I believe that would also be an import
  2. What is the point of the view if the export exists? There is certainly the future possibility of there being fancy/interactive visualizations in the view that would require it to not be static, but that is currently not the case. So I don't see a reason to have two separate static views unless there is a clear reason to.
  3. When you choose a negotiation card to import, would the data copy to the report and then not change or would it link back to the negotiation card so the data in the report could update as the card changes.
sei-aderr commented 2 months ago

Also, would you view this as crucial to the "initial release" by the end of September? While this needs to be addressed I might backlog it in favor of the frontend work on the catalog store

GALewis commented 2 months ago

The issue here is very similar to what we talked about this week. There is a disconnect between the new MLTE and the old MLTE and we never really revisited the Report. Let me try to answer your questions:

  1. While nearly all of the report could be imported from other sources, how would the comments field be handled? Currently the only way to add to it is through the form. There is also the quantitative analysis field that is not populated currently but I believe that would also be an import

This is a very good question, but I think it will be answered in MLTE 2.0. In the future I think we need to create a better link between the Negotiation Card and Test Results. Some requirements will require qualitative evidence and others will require quantitative evidence. The qualitative evidence will come from the test code and the quantitative evidence will come from developer-ended evidence. While we define how that is done, I recommend to leave fields out for now (See last question).

  1. What is the point of the view if the export exists? There is certainly the future possibility of there being fancy/interactive visualizations in the view that would require it to not be static, but that is currently not the case. So I don't see a reason to have two separate static views unless there is a clear reason to.

You are right, there is no reason to have to have two static views. I am thinking of this as a temporary solution so just displaying the report in the browser would be more than enough.

  1. When you choose a negotiation card to import, would the data copy to the report and then not change or would it link back to the negotiation card so the data in the report could update as the card changes.

The latter. It should go into the Negotiation Card where it could potentially be edited.

  1. Also, would you view this as crucial to the "initial release" by the end of September? While this needs to be addressed I might backlog it in favor of the frontend work on the catalog store

I would simply include the report with all the fields as we talked about before, just adding the test results. Would that work?

sei-aderr commented 2 months ago

I will address 1-3 in the future when I work through those issues.

  1. This should be fine. I will work on these under a separate issue (#464) and then we can return to re-evaluating the report at a later time.
sei-aderr commented 2 months ago

Per #464 and comments from Grace, report should be readonly once it can be properly linked to a negotiation card and populated that way.

sebastian-echeverria commented 2 months ago

Regarding: "When you choose a negotiation card to import, would the data copy to the report and then not change or would it link back to the negotiation card so the data in the report could update as the card changes.

The latter. It should go into the Negotiation Card where it could potentially be edited."

I belive the opposite should be done, not link to the Negotiation Card, since the Report should be a static snapshot of what happened at a specific time. But we can discuss that as part of a bigger set of changes.

GALewis commented 2 months ago

The data should go into the Negotiation Card because it needs to be editable. There is not a one-to-one mapping for all the fields.