LAB-1477 input file handling

[supraja-968]

What type of PR is this?

• 🎮 Feature

Description

This PR is a continuation of #970 with changes related to input file handling. Few cosmetic changes included in the PR: File and Tool CIDs are displayed as File ID and Tool ID because they are SHA 256 hashes and no longer CIDs.

Important changes to note:

• Datafiles are passed in the payload with the S3 URIs. So they are no longer CID/filename, but S3://bucket-name/path/to/file. There is a corresponding new column in the DataFile table to track these S3 URIs.
• There is a custom RayJobResponse unmarshalling, to get uuid, pdb, metrics, and the rest of the files. The x and y axis for these metrics are specified in the tool manifest.

(Note) To-do in the next PR:

• Tool migration to S3 from IPFS
• Create Job UUID before ray job submission and send it with the request body
• Fix the convexity side service logic to use this above UUID throughout all the sub services for a single job
• Checkpoints at the moment throw error with unmarshalling json due to missing job uuid, which is the same issue mentioned above, which can be ignored for now. This doesn't affect the checkpoint plotting after the job returns with the response.

Related Tickets & Documents

This PR can be tested along with PR-111

Steps to Test

Test steps mentioned in PR-111 in Convexity Repo

To test, use this tool manifest: tools/ray/design_with_colabdesign.json

[linear[bot]]

LAB-1477 Input file handling in ray integration

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

| Name | Status | Preview | Comments | Updated (UTC) | | :--- | :----- | :------ | :------- | :------ | | **docs** | ⬜️ Ignored ([Inspect](https://vercel.com/convexitylabs/docs/88MjcG68L3VVn5go79pU1qgA1Ncc)) | [Visit Preview](https://docs-git-lab-1477inputfilehandling-convexitylabs.vercel.app) | | Jun 10, 2024 4:57pm |