neulab / explainaboard_web

MIT License
8 stars 2 forks source link

store `system_output_metada` in the DB #493

Closed lyuyangh closed 1 year ago

lyuyangh commented 1 year ago

blocked by #492 part of #433 Stores system_output_metadata in the DB because it is required in update_overall_statistics() but we don't have access to the raw system output in this method. Now, the only dependency of update_overall_statistics() is self (the SystemModel instance). This makes it very easy to recompute overall statistics.

Note that there is no way of updating system_output_metadata for the systems. The loader code is relatively stable but if the loader (implemented in the SDK) is modified, the changes won't be applied to old systems. A better implementation is to store the raw dataset & output files submitted by the user and use that to regenerate system_output_metadata when update_overall_statistics() is called. That approach requires a bigger change to the current implementation and it also requires all the systems to be re-submitted. That's why I decided to go with the simpler approach for now. I will open an issue for the better approach and we can maybe implement it in the future.

Compatibility