search

fsspec / ossfs

fsspec filesystem for Alibaba Cloud (Aliyun) Object Storage System (OSS)
Apache License 2.0
21 stars 11 forks source link

append object logging hangs #148

Open DeoLeung opened 11 months ago

DeoLeung commented 11 months ago

I'm using AioOSSFileSystem

in my code

from  anyio import to_thread

fs = AioOSSFileSystem(...)

exists = await to_thread.run_sync(fs.exists, key)
if not exists:
  with fs.open(key, 'wb') as f:
    f.write(image_res)

when debug is on

api 146 2023-11-20 19:54:24,161 - DEBUG - Responses from the server, req_id: 655B48F0A38DE83432969092, status_code: 403
api 154 2023-11-20 19:54:24,161 - INFO - Exception: {'status': 403, 'x-oss-request-id': '655B48F0A38DE83432969092', 'details': {'Code': 'AccessDenied', 'Message': 'The bucket you access does not belong to you.', 'RequestId': '655B48F0A38DE83432969092', 'HostId': 'mdt-prod-public.oss-cn-hangzhou.aliyuncs.com', 'EC': '0003-00000905', 'RecommendDoc': 'https://api.aliyun.com/troubleshoot?q=0003-00000905'}}
async_oss 111 2023-11-20 19:54:24,162 - DEBUG - Connect AioSession instance
api 118 2023-11-20 19:54:24,162 - DEBUG - Init endpoint: http://oss-cn-hangzhou.aliyuncs.com, isCname: False, connect_timeout: None, app_name: ossfs, enabled_crc: True, proxies: None
async_oss 161 2023-11-20 19:54:24,162 - DEBUG - CALL: append_object - ('xxx.png', 0, b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x08\x00\x00\x00\x08\x00\x08\x02\x00\x00\x00=\xc5Dg\x00\x01\x00\x00IDATx\x01\x84\xfd\xcb\x82,\xc9\x8e\xa5\xe9E\xc4\x8es2\xb3\xaa\xbb\xd9= \xdf\xff\x05\xc9AUv^N\xdc\xf8\x7f\x0bb\xbew\x16\x07T7W\x13\x85\x00\x0b\x0b\x10\x88\xa8\x9a\x9a\x99\xfb\xcf\xff\xdb\x7f\xfb\x97\x9f~\xfa\xf9\xaf\x9f\xfe\xfa\xf9\x97_~\xf9\xe9\xe7\x9f\x7f\xf9\xe9\xa7\xbf\xfe\xfay\xdbO?\xf7\x93\xa8\xfd/\xfd\xd4N\xe5\xaf?\xff\xac\xf3\x17\xbf?\xfd\xf5\xd3\x8c\xfeJ\xf2\xd3\xb7\x8e\x7ff\xfaK\xfa\xa9\x1dH\xc8\x7f\xfd\xf5\xed\x1b\xf5\xcc\xf3\xc2\xc9\x9f\x7f\xfc\xf2\xcb7\x02\xb26\xae\xff\xfa\xeb\x8f\xa9\x10y\\\x17\xb0)L\xf2g\x00\xeb%\x9b\x02/\x0e2Y\x08c\xf9\x

this hangs my terminal which keeps logging forever

DeoLeung commented 11 months ago

currently do logging.getlogger('ossfs') and set logging level to disable it

also I'm curious about

  1. for exists check, isn't object check enough? I saw it check bucket first, which may require extra permission, like getbucketinfo
  2. for write, wb, I saw logging it does some head, append job, should it be a simple putObject?