New database search features added in support of CDM data import, plus private data transfer!

[jeff-cohere]

With the changes in this PR, we've been able to transfer over 10k IMG files from the JGI Data Portal to the KBase CDM object store.

Here's a rundown of the changes:

• A new search parameter allows filtering on the staging status of a file ('staged' or 'unstaged').
• The frictionless package has been updated with some changes to conform to the Frictionless I/O specs, with more appropriate DataResource name fields.
• The DTS now allows database-specific parameters, with several JDP-specific parameters added. Searches with database-specific parameters accept POST requests instead of the usual GET.
  • In addition, the DTS adds an endpoint (/databases/{db}/search-parameters) that allows a client to query the set of all possible search parameters specific to a given database.
• The root and documentation endpoints are no longer protected by authentication.
• Bugs have been fixed in task cancellation and JDP-related file staging.

NOTE: I had to update Huma to get the database-specific parameters to work. This update is actually responsible for most of the update files in this PR.

Update (7/24/2024)

This PR is now a bit long on the tooth and now contains more goodness:

• Database-specific parameters are now a bit fancier, with non-string values available, and an additional endpoint that provides machine-readable descriptions of said parameters
• Support for transferring private data from the JDP has been added (!)
• The data is now placed in a folder named dts-{task-id} instead of {task-id}
• Some issues in dealing with network flakiness have been fixed
• We can now search the JDP for files by taxon OID
• A maximum payload size in gigabytes (default: 100) can now be specified in the configuration file
• File staging checks have been simplified and are database-specific instead of endpoint-specific
• The DTS now uses a special JDP metadata endpoint for querying file metadata, which replaces JAMO (and simplifies our automated testing)

With the above features, the DTS now supports the transfer of private data from the JGI data portal.

Closes #1 Closes #18

[github-actions[bot]]

PR Preview Action v1.4.7 :---: :rocket: Deployed preview to https://kbase.github.io/dts/pr-preview/pr-72/ on branch gh-pages at 2024-07-31 16:37 UTC

[jeff-cohere]

Okay, things are in. We'll have to think about how we want to test the machinery involved in actual transfers, because everything takes so long compared to a typical CI cycle. In any case, I'd like to limit any further growth to this PR.

[jeff-cohere]

@ialarmedalien , I'm super sorry for the mammoth PR here. If you want, I can do some surgery and break this up into several smaller ones. And/or I'm happy to be available to you for Q+A when it's a convenient time to review. Let me know what works best for ya. This has been quite the sprint!

[jeff-cohere]

Okay @ialarmedalien . Ready for your re-review.