DDS-1680: AddFailedFiles endpoint to insert files if failed

[valyo]

Read this before submitting the PR

1. Always create a Draft PR first
2. Go through sections 1-5 below, fill them in and check all the boxes
3. Make sure that the branch is updated; if there's an "Update branch" button at the bottom of the PR, rebase or update branch.
4. When all boxes are checked, information is filled in, and the branch is updated: mark as Ready For Review and tag reviewers (top right)
5. Once there is a submitted review, implement the suggestions (if reasonable, otherwise discuss) and request an new review.

If there is a field which you are unsure about, enter the edit mode of this description or go to the PR template; There are invisible comments providing descriptions which may be of help.

1. Description / Summary

On data upload, it can happen that some files uploaded to S3 are not added to the DB (due to failed API request or DB lock). In such cases, DDS CLI creates a log file containing the failed files, which is supposed to be passed to the update-uploaded-file flask command for adding the files in the DB. This change creates an endpoint for updating the failed files upon a request from DDS CLI.

Tested manually in the following way:

• In data_putter.py, def put, line 376 (in the try-except block), I added:

          if "requirements-test" in file_local:
              raise FileNotFoundError

• In data_putter.py, def add_file_db, line 431 (in the try-except block), I added:

          if "test_account_manager.py" in str(file_info["name"]):
              raise dds_cli.exceptions.DDSCLIException(message="Failed to add file to db")

• Then I ran dds data put -p [project] -s tests
  • Without --overwrite
  • With --overwrite

2. Jira task / GitHub issue

DDS-1529

3. Type of change

What type of change(s) does the PR contain?

Check the relevant boxes below. For an explanation of the different sections, enter edit mode of this PR description template.

• [x] New feature
  • [ ] Breaking: Why / How? Add info here.
  • [x] Non-breaking Works with #656
• [ ] Database change: Remember the to include a new migration version, or explain here why it's not needed.
• [ ] Bug fix
• [ ] Security Alert fix
• [ ] Documentation
• [ ] Workflow
• [ ] Tests only

4. Additional information

• [x] Sprintlog
• [ ] Blocking PRs
  • [ ] Merged
• [ ] PR to master branch: _If checked, read the release instructions
  • [ ] I have followed steps 1-8.

5. Actions / Scans

Check the boxes when the specified checks have passed.

For information on what the different checks do and how to fix it if they're failing, enter edit mode of this description or go to the PR template.

• [x] Black
• [x] Prettier
• [x] Yamllint
• [x] Tests
• [x] CodeQL
• [x] Trivy
• [x] Snyk

[codecov[bot]]

Codecov Report

Merging #1472 (9f67d66) into dev (f32c284) will increase coverage by 0.85%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##              dev    #1472      +/-   ##
==========================================
+ Coverage   90.32%   91.17%   +0.85%     
==========================================
  Files          29       29              
  Lines        4484     4542      +58     
==========================================
+ Hits         4050     4141      +91     
+ Misses        434      401      -33     

Files	Coverage Δ
dds_web/api/__init__.py	100.00% <100.00%> (ø)
dds_web/api/files.py	83.75% <100.00%> (+0.47%)	:arrow_up:
dds_web/commands.py	85.62% <100.00%> (+5.35%)	:arrow_up:
dds_web/utils.py	94.65% <100.00%> (+2.04%)	:arrow_up:

[valyo]

The last test in test_utils.py is commented out since I couldn't make it produce the "File not found in S3" error. Suggestion are welcome! We could probably skip this test but I am curious how to make it work.

[rv0lt]

Update branch and set it has ready for review :)