Remove reset_on_fork parameter

pfnet / pfio

IO library to access various filesystems with unified API

https://pfio.readthedocs.io/

MIT License

52 stars 21 forks source link

Remove reset_on_fork parameter #297

Closed y1r closed 1 year ago

y1r commented 1 year ago

Fixes https://github.com/pfnet/pfio/issues/296

Removed reset_on_fork parameter from all FS implementations
~Removed ForkedError because we don't need it now~
Added reset_on_fork parameter check on from_url to notify the deprecation

I've found that, in zip + SparseFileCache case, we always use MPCachedWrapper instead of CachedWrapper because we don't have prior information of fork existence by reset_on_fork parameter. This may introduce performance decrease by flock(2).

y1r commented 1 year ago

@kuenishi Thank you for your review.

https://github.com/pfnet/pfio/pull/297/commits/0ee12550773c2c483ec6a49816db5c03c0f2d4d5 addresses your review comments:

Added ForkedError again
Checked multiprocessing.get_start_method() in the constructor of HDFS to inform users early
Also, raises ForkedError in _reset() if start_method is not forkserver.