search

DSGT-DLP / Deep-Learning-Playground

Web Application where people new to Deep Learning can input a dataset and toy around with basic Pytorch modules without writing any code
MIT License
22 stars 8 forks source link

[FEATURE]: Upload Result files of Training Requests to S3 #921

Open dwu359 opened 10 months ago

dwu359 commented 10 months ago

Feature Name

Upload Result files of Training Requests to S3

Your Name

Daniel Wu

Description

Part 1 (backend): Once a training request is finished, it should upload its result fields to S3 in dlp-executions-bucket with the location {id}/{filename}. For tabular training specifically, there is dl_results.csv, model.pt, and my_deep_learning_model.onnx. Details on how to generate these files can be found in backend/dl_trainer.py (deprecated) and maybe other files.

Part 2 (full-stack): Add download buttons to download the result files from S3 in the trainspace id page, which is generated by frontend/src/pages/train/[trainspace_id].tsx. Put redux, constants, types, helper components, etc in frontend/src/Train/features/....

A separate SST endpoint may also be needed to generate the presigned url needed to download the files. You may find looking at serverless/packages/functions/src/datasets/user/upload_url.ts useful as an example, as well as instructions from #917.

github-actions[bot] commented 10 months ago

Hello @dwu359! Thank you for submitting the Feature Request Form. We appreciate your contribution. :wave:

We will look into it and provide a response as soon as possible.

To work on this feature request, you can follow these branch setup instructions:

  1. Checkout the main branch:

     git checkout nextjs

  2. Pull the latest changes from the remote main branch:

     git pull origin nextjs

  3. Create a new branch specific to this feature request using the issue number:

     git checkout -b feature-921

    Feel free to make the necessary changes in this branch and submit a pull request when you're ready.

    Best regards, Deep Learning Playground (DLP) Team

karkir0003 commented 10 months ago

For part 2 @dwu359, is this supposed to be doable from the dashboard?

dwu359 commented 10 months ago

@karkir0003 Ig that can be a separate PR, but it probably can't be done anyways until GET /trainspace from #914 is complete.

karkir0003 commented 10 months ago

i see. so what tasks are independent based on the issues u posted yesterday? @dwu359

dwu359 commented 10 months ago

@karkir0003 most of them are independent of each other, it should say on there if another task is needed to be complete beforehand.

karkir0003 commented 10 months ago

ok. got it

abhay-ramesh commented 9 months ago

Hello there,

I'd like to recommend trying out the next-s3-uploader package, which I created specifically for Next.js app directories. It simplifies the file upload process seamlessly.

You can explore the documentation for comprehensive details. Should you encounter any challenges or have questions, don't hesitate to get in touch with me or open a GitHub issue here. I'm here to provide assistance and support.

karkir0003 commented 9 months ago

@abhay-ramesh link to documentaiton doesnt work

abhay-ramesh commented 9 months ago

@abhay-ramesh link to documentaiton doesnt work

https://next-s3-uploader.abhayramesh.com here you go

karkir0003 commented 9 months ago

Link still doesn't work. Also, how is this library different from using AWS JS SDK to do S3 upload? I'm seeing in the docs that only public S3 buckets are supported. However, private S3 support doesn't seem to be present?

abhay-ramesh commented 9 months ago

Link still doesn't work. Also, how is this library different from using AWS JS SDK to do S3 upload? I'm seeing in the docs that only public S3 buckets are supported. However, private S3 support doesn't seem to be present?

Docs hasnt been completed yet, and private bucket works, ryt now this library aims to simplify file uploads have better dev experience

And to use private bucket In generatePresignedUrls function, set privateBucket to true.

generatePresignedUrls(s3Client, keys, bucket, prefix?)

Generates pre-signed URLs for uploading objects to S3.

for now the missing docs are in github readme https://github.com/abhay-ramesh/next-s3-uploader

Sorry for the issues caused, will be updating things soon

abhay-ramesh commented 9 months ago

Link still doesn't work. Also, how is this library different from using AWS JS SDK to do S3 upload? I'm seeing in the docs that only public S3 buckets are supported. However, private S3 support doesn't seem to be present?

Docs hasnt been completed yet, and private bucket works, ryt now this library aims to simplify file uploads have better dev experience

And to use private bucket In generatePresignedUrls function, set privateBucket to true.

generatePresignedUrls(s3Client, keys, bucket, prefix?)

Generates pre-signed URLs for uploading objects to S3.

  • s3Client: An instance of the S3 client.
  • keys: An array of object keys to upload.
  • bucket: The S3 bucket name.
  • prefix: (Optional) Prefix for object keys.
  • privateBucket: (Optional) Whether the bucket is private. Default is false.

for now the missing docs are in github readme https://github.com/abhay-ramesh/next-s3-uploader

Sorry for the issues caused, will be updating things soon

Have updated the docs