search

Cloud-CV / evalai-cli

:cloud: :rocket: Official EvalAI Command Line Tool
https://cli.eval.ai
BSD 3-Clause "New" or "Revised" License
55 stars 63 forks source link

Add presigned url multi part upload #291

Closed Ram81 closed 3 years ago

Ram81 commented 3 years ago

Description

This PR uses S3 multipart upload for large file uploaded using --large flags. It will allow us to show upload progress using progress bars. EvalAI backend PR

Ram81 commented 3 years ago

@RishabhJain2018 This PR is ready for review, I have tested the feature

RishabhJain2018 commented 3 years ago

Also, can you please check if we can add tests using mock for S3?

Ram81 commented 3 years ago

@RishabhJain2018 Adding tests look like more work at this point. Let's pick it up after deployment as a separate task. We can hardcode responses like it is being done for other APIs (but I am not sure if it is correct way to do it), but we can't mock S3 client because we are not actually using boto3 client on cli side.

RishabhJain2018 commented 3 years ago

Adding tests look like more work at this point. Let's pick it up after deployment as a separate task.

Let's not drop the unit tests and add them in a separate PR now itself.

We can hardcode responses like it is being done for other APIs (but I am not sure if it is correct way to do it) though but we can't mock S3 client because we are not actually using boto3 client.

okay, hardcoding and testing should be fine as it will make sure that if any API changes, our CLI client will also change accordingly.

Ram81 commented 3 years ago

@RishabhJain2018 I've added the tests