search

fsspec / ossfs

fsspec filesystem for Alibaba Cloud (Aliyun) Object Storage System (OSS)
Apache License 2.0
21 stars 11 forks source link

[to#127]Fix ossfile upload_chunk location bug #129

Open idling11 opened 1 year ago

idling11 commented 1 year ago

This PR is a bug fix for ossfile location bug in upload_chunk function.

idling11 commented 1 year ago

UT faild. I will debug it in local env

idling11 commented 1 year ago

@karajan1001 I try to mock a env to run your test_file.py UT by using my own oss. If I have no get_bucket_info auth to this bucket, result is PositionNotEqualToLength error as well. Pls check your oss permission in env. Aprreciated!

karajan1001 commented 1 year ago

@karajan1001 I try to mock a env to run your test_file.py UT by using my own oss. If I have no get_bucket_info auth to this bucket, result is PositionNotEqualToLength error as well. Pls check your oss permission in env. Aprreciated!

Hello, First thank you to do the research. I also looked into the code and found that the problem comes from the difference between self.loc and self.offset . The error all comes from when we want to append something to an existing object. In this case self.loc will point the end of the file while self.offset always point to 0. So this is where the PositionNotEqualToLength comes from. I'm also confused with the usage of self.offset and had already created a ticket in https://github.com/fsspec/filesystem_spec/issues/1285.

zigmengineer commented 10 months ago

@idling11 @karajan1001 Can this PR be merged please?

karajan1001 commented 7 months ago

@dependabot rebase

Wangsongming commented 2 months ago

I also have this problem, it seems that large files are uploaded using sharding when transferring OSS, but it is directly ended without waiting for the return, resulting in the situation of large files that have been unable to pass up, I hope to solve it as soon as possible

Collecting |2.00 [00:00, 250entry/s] Pushing D:\python\lib\site-packages\ossfs\async_oss.py:388: RuntimeWarning: coroutine 'resumable_upload' was never awaitedile/s] await self._call_oss( RuntimeWarning: Enable tracemalloc to get the object allocation traceback Pushing 2 files pushed