Prevent raw counts requirement on deleting files (SCP-5849)

[bistline]

BACKGROUND & CHANGES

This update prevents two errors from occurring when attempting to relaunch an AnnData extraction that failed due to an OOM exception.

• Raw counts required: When the first file is cloned and then queued for deletion, since its file type is no longer AnnData but it still has an ExpressionFileInfo document, this validation is now enforced (if enabled). This causes an error preventing the new file from being ingested as the entire job fails without successfully launching the retry.
• FrozenError: can't modify frozen BSON::Document: while this doesn't have a user-facing component, this error is nonetheless thrown when trying to delete the original file because the AnnDataFileInfo#data_fragments entries are BSON documents. Now, these documents are removed prior to preparing the file for deletion.

With these two errors addressed, now failed AnnData extractions will correctly retry in the event of an OOM exception.

MANUAL TESTING

1. Boot as normal and sign in
2. Go to the Feature Flags page and set raw_counts_required_backend to True for your email account (or if the flag is enabled, remove any overrides for your account/study)
3. Follow the steps from #2154 and ensure that the retry launches successfully even with raw_counts_required_backend enabled

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 70.02%. Comparing base (1a111d6) to head (c3d4c93). Report is 8 commits behind head on development.

Additional details and impacted files

[](https://app.codecov.io/gh/broadinstitute/single_cell_portal_core/pull/2156?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=broadinstitute) ```diff @@ Coverage Diff @@ ## development #2156 +/- ## ============================================ Coverage 70.02% 70.02% ============================================ Files 331 331 Lines 28015 28021 +6 Branches 2452 2452 ============================================ + Hits 19617 19623 +6 Misses 8252 8252 Partials 146 146 ``` | [Files with missing lines](https://app.codecov.io/gh/broadinstitute/single_cell_portal_core/pull/2156?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=broadinstitute) | Coverage Δ | | |---|---|---| | [app/models/delete\_queue\_job.rb](https://app.codecov.io/gh/broadinstitute/single_cell_portal_core/pull/2156?src=pr&el=tree&filepath=app%2Fmodels%2Fdelete_queue_job.rb&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=broadinstitute#diff-YXBwL21vZGVscy9kZWxldGVfcXVldWVfam9iLnJi) | `64.41% <100.00%> (+0.89%)` | :arrow_up: | | [app/models/expression\_file\_info.rb](https://app.codecov.io/gh/broadinstitute/single_cell_portal_core/pull/2156?src=pr&el=tree&filepath=app%2Fmodels%2Fexpression_file_info.rb&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=broadinstitute#diff-YXBwL21vZGVscy9leHByZXNzaW9uX2ZpbGVfaW5mby5yYg==) | `100.00% <100.00%> (ø)` | | | [app/models/ingest\_job.rb](https://app.codecov.io/gh/broadinstitute/single_cell_portal_core/pull/2156?src=pr&el=tree&filepath=app%2Fmodels%2Fingest_job.rb&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=broadinstitute#diff-YXBwL21vZGVscy9pbmdlc3Rfam9iLnJi) | `55.26% <ø> (ø)` | | | [app/models/study\_file.rb](https://app.codecov.io/gh/broadinstitute/single_cell_portal_core/pull/2156?src=pr&el=tree&filepath=app%2Fmodels%2Fstudy_file.rb&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=broadinstitute#diff-YXBwL21vZGVscy9zdHVkeV9maWxlLnJi) | `81.85% <100.00%> (+0.04%)` | :arrow_up: |

[eweitz]

The last two test failures in the build seem like plausible true positives:

Total elapsed time: 28 minutes, 32 seconds

### FAILURES and ERRORS ###
*** Yarn test failures ***

*** Rails test failures ***

  FireCloudClientTest#test_create_and_manage_user_groups:
  RuntimeError: Wrapped undumpable exception for: RestClient::InternalServerError: 500 Internal Server Error

  StudyValidationTest#test_should_validate_unique_cells_for_expression_matrices:
  RuntimeError: Sleep timeout after 300 seconds when waiting for parse of "expression_matrix_example_2.txt".

  DeleteQueueJobTest#test_should_prepare_file_for_deletion_to_allow_cloning [/home/app/webapp/test/models/delete_queue_job_test.rb:252]:
  Expected false to be truthy.

  IngestJobTest#test_should_automatically_retry_on_OOM_failure:
  NoMethodError: undefined method `delete_nested_associations' for DeleteQueueJob:Class
Exiting with code: 1

[bistline]

The last two test failures in the build seem like plausible true positives:

Total elapsed time: 28 minutes, 32 seconds

### FAILURES and ERRORS ###
*** Yarn test failures ***

*** Rails test failures ***

  FireCloudClientTest#test_create_and_manage_user_groups:
  RuntimeError: Wrapped undumpable exception for: RestClient::InternalServerError: 500 Internal Server Error

  StudyValidationTest#test_should_validate_unique_cells_for_expression_matrices:
  RuntimeError: Sleep timeout after 300 seconds when waiting for parse of "expression_matrix_example_2.txt".

  DeleteQueueJobTest#test_should_prepare_file_for_deletion_to_allow_cloning [/home/app/webapp/test/models/delete_queue_job_test.rb:252]:
  Expected false to be truthy.

  IngestJobTest#test_should_automatically_retry_on_OOM_failure:
  NoMethodError: undefined method `delete_nested_associations' for DeleteQueueJob:Class
Exiting with code: 1

Yeah I did a last-minute refactor and forgot to remove a method reference. d3d183b should fix it.