Make submission process backward compatible with the legacy rabinowitz data

[hepcat72]

FEATURE REQUEST

Inspiration

The new submission process made a few column changes necessary for inter-sheet interactions, to support features like dynamic autofill and dynamic drop-down population (based on other sheets' content). This means that if you try to submit old animal/sample sheets to the build-a-submission page or try to load old data using the new StudyLoader, you would run into exceptions.

Description

Given the new ConvertedTableLoader class, it should be fairly straightforward to support legacy data by making the new SamplesLoader and AnimalsLoader inherit from ConvertedTableLoader instead of directly from TableLoader and define the changes necessary to convert version 2 data to version 3. There would also have to be some code added to detect version, but that should similarly be straightforward.

The following is a rough estimate of the changes necessary:

• Create a "Study" sheet and
  • [x] Copy the distinct Study Name and Study Description column values from the Animals sheet to it.
  • [x] The Study Loader will have to know to send the Animals sheet as a DF.
• Animals sheet changes:
  • [x] Delete tracer concentrations column
  • [x] Add concentrations to the infusate column
  • [x] Delete Study description column
  • [x] Copy Study name and description to study sheet
  • [x] Modify the Infusate column to include the tracer concentrations in the name
  • Rename columns whose name changed
  • [x] Animal Genotype -> Genotype
  • [x] Animal Body Weight -> Weight
  • [x] Animal Treatment -> Treatment
  • [x] Study Name -> Study
  • [x] Animal ID -> Animal Name
• Samples sheet changes:
  • Rename columns whose name changed
  • [x] Sample Name -> Sample
  • [x] Animal ID -> Animal
  • [x] Researcher Name -> Researcher
• Tissues sheet changes:
  • Renames:
  • [x] TraceBase Tissue Name -> Tissue

Make StudyLoader able to recognize the versions and call StudiesLoader, even if the sheet is not present. So make StudiesLoader inherit from ConvertedTableLoader as well.

Update DataValidationView to determine the submitted file version and apply that version to the methods that check version, and accept version 2.

Alternatives

One alternative would be to edit the existing rabinowitz repo data to convert v2 data to v3.

Dependencies

This issue cannot be started until the completion of the following issue(s)/ pull request(s):

• 1030

Comment

None

---

ISSUE OWNER SECTION

Assumptions

1. List of assumptions that the code will not explicitly address/check
2. E.g. We will assume input is correct (explaining why there is no validation)

Limitations

1. A list of things this work will specifically not do
2. E.g. This feature will only handle the most frequent use case X

Affected Components

• change: File path or DB table ...
• add: Environment variable or server setting
• delete: External executable or cron job

Requirements

• 1. List of numbered conditions to be met for the feature
• 2. E.g. Every column/row must display a value, i.e. cannot be empty
• 3. Numbers for reference & checkboxes for progress tracking

DESIGN

Interface Change description

None provided

Code Change Description

None provided

Tests

• 1. A description of at least one test for each requirement above.
• 2. E.g. Test for req 2 that there's an exception when display value is ''
• 3. Numbers for reference & checkboxes for progress tracking

[hepcat72]

Possibly the same as #1047

[hepcat72]

I have an error-free version of the "backward_compatibility" branch, but there are still some things I want to do. I might not do all of this in one PR. In fact, some of what I've done could/should be broken up, but I need to organize my thoughts on what's left to do...

• Tests #1148
  • Uncomment all the commented tests and update their setup and calls to load_study
  • Implement these test stubs
    • test_convert_df_v3
    • test_convert_df_v2
    • [x] test_load_study_v3_command
    • [x] test_load_study_v2_command
    • test_name_from_data (Infusate)
• DataValidationView
  • [x] Manually test old study doc versions in the build-a-submission view
  • [x] Make autofill work in addition to validation (so that old study docs get sheets like "Peak Annotation Files")
  • [x] Add version arguments to all the method calls
• ConvertedTableLoader
  • Build in support for the conversion result being a dict of dataframes (instead of a merged single dataframe), meaning that the merge_dict class attribute would be optional and reverting would not involve grabbing data from the merge_dict. It also means that the overrides of convert_df (and other method(s)) would no longer be necessary. #1147
  • Make the methods in StudyLoader and PeakAnnotationsLoader that detect format/version be a part of ConvertedTableLoader as either generic methods or abstract class attributes. #1146

[hepcat72]

Done. Remaining issues were split off to separate issues.