search

IBM / jupyterlab-s3-browser

A JupyterLab extension for browsing S3-compatible object storage
Apache License 2.0
119 stars 41 forks source link

File upload function do not upload complete file #63

Open mohittalele opened 2 years ago

mohittalele commented 2 years ago

Describe the bug I am trying to upload csv file which contains million rows. Using the s3-browser functionality I can upload file - However not complete file is uploaded and about 50% of the data is lost. The s3-broswer do not show any logs or error message To Reproduce Steps to reproduce the behavior:

  1. Take fairly large csv file
  2. Click on upload button in s3 browser
  3. s3 browser Uploads the data.
  4. Check the number of rows in the file. Not all the rows are present since whole file is not uploaded

Expected behavior Large file should be uploaded completely to s3 endpoint

Screenshots

Desktop (please complete the following information):

I am using standalone Minio version - minio/minio:RELEASE.2020-06-14T18-32-17Z as my s3 endpoint

reevejd commented 2 years ago

Thanks for reporting the issue! I have been able to reproduce it. I'm pretty busy at the moment but I should have time to investigate and fix it within the next couple of weeks.

mohittalele commented 1 year ago

Hi Reeve, thanks for info.

If it's not that complicated, you can leave some instructions on how to fix it. Maybe I can create draft pr.

On Mon, 4 Apr 2022, 16:00 James Reeve, @.***> wrote:

Thanks for reporting the issue! I have been able to reproduce it. I'm pretty busy at the moment but I should have time to investigate and fix it within the next couple of weeks.

— Reply to this email directly, view it on GitHub https://github.com/IBM/jupyterlab-s3-browser/issues/63#issuecomment-1087595044, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIVOMX6KPV2ON6TIWNE3H3VDLYV5ANCNFSM5R23JOKQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

reevejd commented 1 year ago

Hi @mohittalele, thanks for your interest and sorry for the lack of updates on this issue. I'm guessing the issue is that the serverextension doesn't handle the case where jupyterlab breaks a file up into chunks for uploading. The extension is probably just overwriting the whole file with the latest chunk every time. Are you able to confirm that hypothesis? Specifically I'm thinking you'll see the size of the file stays the same throughout the upload (maybe 1MB? or however large the chunk size is) but the content will continue changing.

inteloid commented 1 year ago

I don't see the upload button guys, where is it supposed to be? I see the small video in the readme, I don't see it on my jupyter installation