SEAB-6423: Consistently impose SourceFile limits

[svonworl]

Update

Alrighteee... the original PR has been overhauled. Important changes and observations:

• The SourceFile size/binary limiting code is now in LimitedSourceFileBuilder, which demonstrates the "step builder" pattern: http://rdafbn.blogspot.com/2012/10/step-builder-pattern-definition-and.html
• Other helpers have moved to SourceFile.
• We now apply a 10MB per-file download limit in the low-level GitHub download code: the code checks the file size of a Blob before downloading, and if it's >10MB, it is not downloaded at all. This prevents a trivial DOS, but allows us to download large-but-not-entirely-unreasonably-sized files (<10MB) and analyze their content, so we can apply content-specific limits and talk about them in the related error messages.
• Created followup tickets to limit the overall number of bytes downloaded/stored per version, and to add status information to SourceFile that will signal that the file is complete, the content is an error message, etc.: https://ucsc-cgl.atlassian.net/browse/SEAB-6488 https://ucsc-cgl.atlassian.net/browse/SEAB-6489
• Suggest we don't further increase the notebook file size limit, or spend the time to make it configurable, until we see how the above tickets pan out.
• Our code mangles binary file content - input bytes are interpreted as UTF-8, and non-UTF-8 binary sequences get mapped to code point 65533: https://www.fileformat.info/info/unicode/char/0fffd/index.htm If (in the future) we want to accurately interpret or save binary content in the Dockstore database, there is some significant refactoring to be done.
• My initial goal was to limit the scope of this PR so we could hotfix if desired, but in retrospect, hotfixing it feels too risky.
• The reported test coverage is low, but I believe the actual coverage is pretty good. Maybe our system isn't factoring in the new unit test correctly?

Description Whilst diagnosing https://ucsc-cgl.atlassian.net/browse/SEAB-6423, we discovered that when registering a 2.5MB notebook file, the corresponding SourceFile's content is null. This causes the UI "Preview" notebook tab to report that the notebook does not have any content.

Investigation revealed that the webservice only enforces the maximum size and non-binary SourceFile requirements on the old entry registration path. On the .dockstore.yml registration path, SourceFiles are still effectively limited, but only because:

• the GitHub API query used to retrieve a file's contents returns null when the file's size is >= 1MB. This null is stored as the corresponding SourceFile's content.
• Postgres won't store strings containing the nul (0) character. So, when the webservice attempts to create an entry containing a binary SourceFile, a database exception is thrown. (I haven't confirmed the latter but am pretty sure it's true.)

To address the above, this PR:

• adds code to successfully retrieve the content of files larger than 1MB, via the "blob" GitHub API.
• increases the maximum size limit of a Jupyter notebook file to 3MB.
• adds a set of helper methods to create, copy, and duplicate SourceFile instances.
• moves the requirements enforcement code to the create helper method.
• changes existing webservice SourceFile creation/manipulation code to use the helper methods.

If we consistently use these SourceFile helper functions, we make it easy to tweak the SourceFile creation process (to add a new requirement, for example) and ensure that, should we modify the SourceFile object itself, all fields get copied correctly.

As before, when a file doesn't meet the size or non-binary requirements, we set the corresponding SourceFile's content to an error message. For example, "Dockstore does not store binary files". However, code that subsequently uses the SourceFile has no way to differentiate the error message from valid file content. So, after this PR, when a "too large" notebook is registered, the notebook's "Preview" tab will try to parse the error message as a Jupyter notebook, and fail in a different way.

Thus, in a followup PR, I propose to modify SourceFile so that we can differentiate a normal SourceFile from one that didn't meet our requirements. Perhaps we store the error message in a different field, or add a flag or enum that differentiates "COMPLETE" and "ERROR" content. An enum would allow us to support other states, such as a "SUMMARY" state, which would describe an abbreviated form of the full content (perhaps useful if we diversify into indexing AI models, which we might not want [or be able] to store in complete form).

It can be argued that [some of?] the helpers should be methods of SourceFile, etc. If anyone feels super-strongly, we can tweak. There's a rationale behind the current design, but I've already typed too much in this description, so I won't dump it here.

This PR is a little bit scary from a reliability/correctness point of view. The new code passes our existing tests, and appears to be covered. However, given the variety of states and contexts involved, it's very difficult to confirm that we haven't introduced a bug, without spending tons of time auditing the existing tests and filling any holes. Given the way surrounding code is currently architected, it's hard to test, which makes verifying correctness even more challenging. It'll help to visually inspect very carefully and be on the lookout for sourcefile-related problems after we deploy to qa, but it's still a little risky.

Review Instructions Register the notebook which is linked in the ticket, and confirm that Dockstore successfully read it, by visiting the notebook page and confirming that the Preview is correct.

Issue https://ucsc-cgl.atlassian.net/browse/SEAB-6423

Security and Privacy

No concerns.

• [x] Security and Privacy assessed

e.g. Does this change...

• Any user data we collect, or data location?
• Access control, authentication or authorization?
• Encryption features?

Please make sure that you've checked the following before submitting your pull request. Thanks!

• [x] Check that you pass the basic style checks and unit tests by running mvn clean install
• [x] Ensure that the PR targets the correct branch. Check the milestone or fix version of the ticket.
• [x] Follow the existing JPA patterns for queries, using named parameters, to avoid SQL injection
• [x] If you are changing dependencies, check the Snyk status check or the dashboard to ensure you are not introducing new high/critical vulnerabilities
• [x] Assume that inputs to the API can be malicious, and sanitize and/or check for Denial of Service type values, e.g., massive sizes
• [x] Do not serve user-uploaded binary images through the Dockstore API
• [x] Ensure that endpoints that only allow privileged access enforce that with the @RolesAllowed annotation
• [x] Do not create cookies, although this may change in the future
• [x] If this PR is for a user-facing feature, create and link a documentation ticket for this feature (usually in the same milestone as the linked issue). Style points if you create a documentation PR directly and link that instead.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 66.97248% with 36 lines in your changes missing coverage. Please review.

Project coverage is 74.36%. Comparing base (74674f8) to head (ebb3e5a).

Files	Patch %	Lines
...tore/webservice/core/LimitedSourceFileBuilder.java	10.52%	34 Missing :warning:
...re/webservice/languages/LanguagePluginHandler.java	81.81%	1 Missing and 1 partial :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## develop #5893 +/- ## ============================================= - Coverage 74.54% 74.36% -0.19% - Complexity 5364 5368 +4 ============================================= Files 374 375 +1 Lines 19430 19418 -12 Branches 2028 2030 +2 ============================================= - Hits 14485 14441 -44 - Misses 3973 4007 +34 + Partials 972 970 -2 ``` | [Flag](https://app.codecov.io/gh/dockstore/dockstore/pull/5893/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dockstore) | Coverage Δ | | |---|---|---| | [bitbuckettests](https://app.codecov.io/gh/dockstore/dockstore/pull/5893/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dockstore) | `26.93% <28.44%> (-0.09%)` | :arrow_down: | | [hoverflytests](https://app.codecov.io/gh/dockstore/dockstore/pull/5893/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dockstore) | `27.39% <26.60%> (-0.05%)` | :arrow_down: | | [integrationtests](https://app.codecov.io/gh/dockstore/dockstore/pull/5893/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dockstore) | `56.89% <55.96%> (-0.14%)` | :arrow_down: | | [languageparsingtests](https://app.codecov.io/gh/dockstore/dockstore/pull/5893/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dockstore) | `11.09% <0.00%> (+<0.01%)` | :arrow_up: | | [localstacktests](https://app.codecov.io/gh/dockstore/dockstore/pull/5893/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dockstore) | `21.59% <24.77%> (-0.03%)` | :arrow_down: | | [toolintegrationtests](https://app.codecov.io/gh/dockstore/dockstore/pull/5893/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dockstore) | `30.33% <32.11%> (-0.03%)` | :arrow_down: | | [unit-tests_and_non-confidential-tests](https://app.codecov.io/gh/dockstore/dockstore/pull/5893/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dockstore) | `25.99% <20.18%> (-0.01%)` | :arrow_down: | | [workflowintegrationtests](https://app.codecov.io/gh/dockstore/dockstore/pull/5893/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dockstore) | `38.20% <44.03%> (-0.18%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dockstore#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[denis-yuen]

increases the maximum size limit of a Jupyter notebook file to 3MB.

From a different conversation, the limit per file is possibly less important than the limit per version (or even per workflow?). Link follow-up ticket here if spun off

Thus, in a followup PR, I propose to modify SourceFile so that we can differentiate a normal SourceFile from one that didn't meet our requirements.

:+1:

[svonworl]

Ok, I overhauled the PR to address most of the feedback. Please refer to the "Update" section at the top of the PR description for more information.

[sonarcloud[bot]]

Quality Gate failed

Failed conditions
63.2% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud