search

labdao / plex

Platform for running comp bio applications on distributed compute and storage infrastructure
https://lab.bio
MIT License
54 stars 14 forks source link

Logs to hashS3Object method #997

Closed supraja-968 closed 1 month ago

supraja-968 commented 1 month ago

What type of PR is this?

Remove the categories that do not apply

Description

Added more log messages to the hashs3object method for better debugging. Successful log messages should look like this:

Parsed Ray job response:
{
    "files": {
        "msa": {
            "uri": "s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original.a3m"
        },
        "plots[0]": {
            "uri": "s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_pae.png"
        },
        "plots[1]": {
            "uri": "s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_plddt.png"
        },
        "plots[2]": {
            "uri": "s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_coverage.png"
        },
        "structure_metrics": {
            "uri": "s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_scores_rank_001_alphafold2_multimer_v3_model_5_seed_000.json"
        }
    },
    "pdb": {
        "uri": "s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_unrelaxed_rank_001_alphafold2_multimer_v3_model_5_seed_000.pdb"
    },
    "scores": {
        "i_pae": 1.2,
        "plddt": 86.64271428571428
    },
    "uuid": "ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original"
}
Looping through files in RayJobResponse
 map[msa:{s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original.a3m} plots[0]:{s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_pae.png} plots[1]:{s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_plddt.png} plots[2]:{s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_coverage.png} structure_metrics:{s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_scores_rank_001_alphafold2_multimer_v3_model_5_seed_000.json}]
AddFileToDB for file: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_scores_rank_001_alphafold2_multimer_v3_model_5_seed_000.json, Key: structure_metrics
Processing file: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_scores_rank_001_alphafold2_multimer_v3_model_5_seed_000.json, Type: structure_metrics
Hashing file in AddFileToDB: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_scores_rank_001_alphafold2_multimer_v3_model_5_seed_000.json
Configuring S3 client for local development
Created S3 Client. Parsing S3 URI: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_scores_rank_001_alphafold2_multimer_v3_model_5_seed_000.json
Parsed S3 URI. Bucket: convexity, Key: app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_scores_rank_001_alphafold2_multimer_v3_model_5_seed_000.json
Creating AWS session
Created AWS session. Creating S3 service client
Created S3 service client. Getting object
Got object. Hashing object
Hashed key. Hashing file contents
Hashed file contents. Returning hash
Creating new File record for s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_scores_rank_001_alphafold2_multimer_v3_model_5_seed_000.json with file hash 001aeec1662404545c3bf7a3cdc93534567be2fe587dbc307351fedca8281ebb
AddFileToDB for file: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original.a3m, Key: msa
Processing file: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original.a3m, Type: msa
Hashing file in AddFileToDB: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original.a3m
Configuring S3 client for local development
Created S3 Client. Parsing S3 URI: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original.a3m
Parsed S3 URI. Bucket: convexity, Key: app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original.a3m
Creating AWS session
Created AWS session. Creating S3 service client
Created S3 service client. Getting object
Got object. Hashing object
Hashed key. Hashing file contents
Hashed file contents. Returning hash
Creating new File record for s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original.a3m with file hash ae3e7e246636f5a8feb6f5e81d2800d97439e357d34da690e2bd3b67af210f70
AddFileToDB for file: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_pae.png, Key: plots[0]
Processing file: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_pae.png, Type: plots[0]
Hashing file in AddFileToDB: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_pae.png
Configuring S3 client for local development
Created S3 Client. Parsing S3 URI: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_pae.png
Parsed S3 URI. Bucket: convexity, Key: app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_pae.png
Creating AWS session
Created AWS session. Creating S3 service client
Created S3 service client. Getting object
Got object. Hashing object
Hashed key. Hashing file contents
Hashed file contents. Returning hash
Creating new File record for s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_pae.png with file hash 840a1069e91bd82c4e6b2d6126c4639642df50a7a25d8d3f72441db10f4d9513
AddFileToDB for file: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_plddt.png, Key: plots[1]
Processing file: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_plddt.png, Type: plots[1]
Hashing file in AddFileToDB: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_plddt.png
Configuring S3 client for local development
Created S3 Client. Parsing S3 URI: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_plddt.png
Parsed S3 URI. Bucket: convexity, Key: app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_plddt.png
Creating AWS session
Created AWS session. Creating S3 service client
Created S3 service client. Getting object
Got object. Hashing object
Hashed key. Hashing file contents
Hashed file contents. Returning hash
Creating new File record for s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_plddt.png with file hash a02d7e43f9782dd291a1f50adbf2f02a8b128c9397c75731e7a79156265b5215
AddFileToDB for file: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_coverage.png, Key: plots[2]
Processing file: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_coverage.png, Type: plots[2]
Hashing file in AddFileToDB: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_coverage.png
Configuring S3 client for local development
Created S3 Client. Parsing S3 URI: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_coverage.png
Parsed S3 URI. Bucket: convexity, Key: app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_coverage.png
Creating AWS session
Created AWS session. Creating S3 service client
Created S3 service client. Getting object
Got object. Hashing object
Hashed key. Hashing file contents
Hashed file contents. Returning hash
Creating new File record for s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_coverage.png with file hash 91e0a85bb1b6132f4b8e7b4114253bdc7a3886427510a3ff6d6f7d944a65faac
Adding PDB file to DB
 {s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_unrelaxed_rank_001_alphafold2_multimer_v3_model_5_seed_000.pdb}
Processing file: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_unrelaxed_rank_001_alphafold2_multimer_v3_model_5_seed_000.pdb, Type: pdb
Hashing file in AddFileToDB: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_unrelaxed_rank_001_alphafold2_multimer_v3_model_5_seed_000.pdb
Configuring S3 client for local development
Created S3 Client. Parsing S3 URI: s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_unrelaxed_rank_001_alphafold2_multimer_v3_model_5_seed_000.pdb
Parsed S3 URI. Bucket: convexity, Key: app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_unrelaxed_rank_001_alphafold2_multimer_v3_model_5_seed_000.pdb
Creating AWS session
Created AWS session. Creating S3 service client
Created S3 service client. Getting object
Got object. Hashing object
Hashed key. Hashing file contents
Hashed file contents. Returning hash
Creating new File record for s3://convexity/app/outputs/colabfold/ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca_original_unrelaxed_rank_001_alphafold2_multimer_v3_model_5_seed_000.pdb with file hash c3fbb6e02e20bba854623a5a7cb912b4c7fc0ad63bbabad01af9bb0f590b0ce5
Job 10 completed and added files to DB
Job had id 10
Finished Job with Ray id ca42f8f4-4f90-4cc5-83b4-d99bb0c702ca and status completed
vercel[bot] commented 1 month ago

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment | Name | Status | Preview | Comments | Updated (UTC) | | :--- | :----- | :------ | :------- | :------ | | **docs** | ⬜️ Ignored ([Inspect](https://vercel.com/convexitylabs/docs/6CQKTQ1hKd5KX6t685DAHH3g2enD)) | [Visit Preview](https://docs-git-add-logs-to-hash-s3-convexitylabs.vercel.app) | | Jul 18, 2024 8:59am |