pfnet / pfio

IO library to access various filesystems with unified API
https://pfio.readthedocs.io/
MIT License
52 stars 21 forks source link

S3 + zipfile on fork-server multiprocessing #292

Closed y1r closed 1 year ago

y1r commented 1 year ago

Current PFIO doesn't support S3 + zipfile on fork-server based multiprocessing execution model. This model requires parameters given to Process should be pickled to move object from child to parent, who has responsibility to fork an new process.

https://github.com/pfnet/pfio/pull/292/commits/dffff08c83c3fb52147ab53029d212625828a1c3 solves this issue by avoiding pickling of fileobj and zipobj, then after fork, it will be re-created by zip and S3's _reset() functions.

y1r commented 1 year ago

/test

pfn-ci-bot commented 1 year ago

Successfully created a job for commit dffff08:

pfn-ci-bot commented 1 year ago

Successfully created a job for commit dffff08:

y1r commented 1 year ago

/test

pfn-ci-bot commented 1 year ago

Successfully created a job for commit e77521d:

pfn-ci-bot commented 1 year ago

Successfully created a job for commit e77521d:

y1r commented 1 year ago

/test

pfn-ci-bot commented 1 year ago

Successfully created a job for commit d407139:

pfn-ci-bot commented 1 year ago

Successfully created a job for commit d407139:

y1r commented 1 year ago

/test

pfn-ci-bot commented 1 year ago

Successfully created a job for commit 612ad31:

pfn-ci-bot commented 1 year ago

Successfully created a job for commit 612ad31:

y1r commented 1 year ago

https://github.com/pfnet/pfio/pull/292/commits/612ad312c01199f336fbbb115072b9fc48483d10 fixes the test.

@kuenishi Please review this PR.