search

Unleash / unleash-client-python

Unleash client SDK for Python 💡💡💡
http://unleash.github.io/unleash-client-python
MIT License
84 stars 60 forks source link

please ditch fcache as it does not work with multiprocessing on Windows #303

Open masariello opened 8 months ago

masariello commented 8 months ago

Describe the bug

fcache has a known issue with multiprocessing (tsroten/fcache#26) that has been ignored tor 5 years.

Here's an example of the errors

2024-03-06 10:54:33.478 - ERROR - apscheduler.executors.unleash_executor_YTIN7P - run_job - Job "fetch_and_load_features (trigger: interval[0:01:00], next run at: 2024-03-06 10:55:32 CST)" raised an exception
Traceback (most recent call last):
  File "C:\Users\userName\AppData\Roaming\Python\Python39\site-packages\apscheduler\executors\base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
  File "C:\Users\dchuserNameoi\AppData\Roaming\Python\Python39\site-packages\UnleashClient\periodic_tasks\fetch_and_load.py", line 43, in fetch_and_load_features
    cache.set(ETAG, etag)
  File "C:\Users\userName\AppData\Roaming\Python\Python39\site-packages\UnleashClient\cache.py", line 129, in set
    self._cache[key] = value
  File "C:\Users\userName\AppData\Roaming\Python\Python39\site-packages\fcache\cache.py", line 267, in __setitem__
    self._write_to_file(filename, value)
  File "C:\Users\userName\AppData\Roaming\Python\Python39\site-packages\fcache\cache.py", line 254, in _write_to_file
    os.chmod(filename, self._mode)
OSError: [WinError 6800] The function attempted to use a name that is reserved for use by another transaction: 'C:\\Users\\userName\\AppData\\Local\\PROD\\PROD\\Cache\\cache\\65746167'

To Reproduce

Please see StackOverflow linked in tsroten/fcache#26

sighphyre commented 8 months ago

Hey @masariello, this probably isn't something that'll get resolved in the next few weeks. But we do have tentative plans to rework the caching anyway due to an upcoming project on this SDK

Is this actually affecting you, outside of causing trouble in the logs? Writing to backup, shouldn't be a process that needs to happen to ensure the SDK works correctly

masariello commented 8 months ago

Yes. I hit the problem reproduced in the linked fcache issues

In fact, fcache has numerous issues when running with multiprocessing. It just opens and writes files on disk without any locking or retry logic, so every now and then we do get exceptions caused by those concurrent accesses.

The only way to avoid those issues is replacing fcache with a dict-based BaseCache

calum-pledge-io commented 6 months ago

As an extra motivation, fcache also isn't compatible with many serverless environments out-of-the-box. For example, in AWS lambda, the home directory is mounted on a read-only file system and fcache defaults to $HOME/.cache/ as the base path for cache files. This can be worked around by setting XDG_CACHE_HOME to something like /tmp/cache/, but it makes this package slightly more cumbersome to use in a serverless deployment.