search

stefan-jansen / zipline-reloaded

Zipline, a Pythonic Algorithmic Trading Library
https://zipline.ml4trading.io
Apache License 2.0
1.13k stars 210 forks source link

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: #169

Closed markgahagan1 closed 1 year ago

markgahagan1 commented 1 year ago

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

i ran this: import os import zipline from zipline.utils.run_algo import load_extensions from zipline.data import bundles from norgatedata import StockPriceAdjustmentType from zipline_norgatedata import ( register_norgatedata_equities_bundle, register_norgatedata_futures_bundle ) load_extensions( default=True, extensions=[], strict=True, environ=os.environ, )

bundles.ingest('norgatedata-fang', show_progress = False)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "g:/My Drive/Projects/Notebooks/zipline/zipline_ingest.py", line 18, in bundles.ingest('norgatedata-fang', show_progress = False) File "C:\Users\markg\anaconda3\envs\zip38\lib\site-packages\zipline\data\bundles\core.py", line 469, in ingest downgrade(wf.path, version) File "C:\Users\markg\anaconda3\envs\zip38\lib\contextlib.py", line 525, in exit raise exc_details[1] File "C:\Users\markg\anaconda3\envs\zip38\lib\contextlib.py", line 510, in exit if cb(*exc_details): File "C:\Users\markg\anaconda3\envs\zip38\lib\site-packages\zipline\utils\cache.py", line 385, in exit rmtree(self.path) File "C:\Users\markg\anaconda3\envs\zip38\lib\shutil.py", line 740, in rmtree return _rmtree_unsafe(path, onerror) File "C:\Users\markg\anaconda3\envs\zip38\lib\shutil.py", line 613, in _rmtree_unsafe _rmtree_unsafe(fullname, onerror) File "C:\Users\markg\anaconda3\envs\zip38\lib\shutil.py", line 613, in _rmtree_unsafe _rmtree_unsafe(fullname, onerror) File "C:\Users\markg\anaconda3\envs\zip38\lib\shutil.py", line 618, in _rmtree_unsafe onerror(os.unlink, fullname, sys.exc_info()) File "C:\Users\markg\anaconda3\envs\zip38\lib\shutil.py", line 616, in _rmtree_unsafe os.unlink(fullname) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\markg\AppData\Local\Temp\tmpbzksn37a\tmp029o9sqq\norgatedata-fang\2023-02-19T11;02;14.903658\assets-7.sqlite' Error in atexit._run_exitfuncs: Traceback (most recent call last): File "C:\Users\markg\anaconda3\envs\zip38\lib\shutil.py", line 616, in _rmtree_unsafe os.unlink(fullname) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\markg\AppData\Local\Temp\tmpbzksn37a\tmp029o9sqq\norgatedata-fang\2023-02-19T11;02;14.903658\assets-7.sqlite'

Here is how you can reproduce this issue on your machine:

Reproduction Steps

1.Load code as above 2.Run code 3. ...

What steps have you taken to resolve this already?

Searched all github issues and google but not found similar issue

...

Anything else?

I have tried closing every program that has anything to do with zipline being open, including explorer, jupyter etc

I have also tried saving the py file locally and running from there (not google drive) but same error occurs

...

Sincerely, windowspc\markg

MBounouar commented 1 year ago

Hi, I have no idea what's the real issue here (maybe Windows or the python context manager doesn't work properly) . For some reason windows doesn't allow the folder to be moved from the temporary directory to the final location.

I have seen it happen once on a windows machine installation, but unfortunately can't replicate it, as I don't have a windows machine at hand.

Also the error conn = txn.connect() AttributeError: 'Connection' object has no attribute 'connect' is quite suspicious.

I can only suggest to maybe try to use a python 3.9 environment to see if the error persists. Or, if there is data under the temporary norgatedata-fang\2023-02-19T11;02;14.903658 to simply move it you the zipline/data folder

Cheers, MBounouar

markgahagan1 commented 1 year ago

Thanks for coming back to me. I tried completely uninstalling anaconda, installing miniconda and creating a python 3.9 environment. I then refollowed the instructions to install zipline reloaded and zipline norgatedata. Unfortunately exactly the same error occurs under 3.9. As you suggested there is data in the Temp folder which is not being moved to the correct folder. So the issue is somewhere in between the two operations.

Further update. You were right on the connection error being suspicious. I saw that had been reported before on here so i did as instructed in that solution: pip install sqlalchemy<2. It worked. Thanks for your help.

MBounouar commented 1 year ago

I suspected SQLAlchemy version early on, but I didn't mention it explicitly because I assumed from the packages versions extract shown that it was "sqlalchemy 1.4.39".

Feel free to close the issue if you think it's solved.