Windows - "all pipe instances are busy" error

[domatka78]

Hi - thanks a lot for the great scripts. I am trying to run the batch exporter (export_all.sh) in git bash (Windows 7, 64 bit) and get the following error: File "export_blocks_and_transactions.py", line 57, in job.run() File "D:\Git\ethereum-etl\ethereumetl\jobs\base_job.py", line 30, in run self._end() File "D:\Git\ethereum-etl\ethereumetl\jobs\export_blocks_job.py", line 80, in _end super()._end() File "D:\Git\ethereum-etl\ethereumetl\jobs\batch_export_job.py", line 79, in _end self.executor.shutdown() File "D:\Git\ethereum-etl\ethereumetl\executors\fail_safe_executor.py", line 39, in shutdown self._check_completed_futures() File "D:\Git\ethereum-etl\ethereumetl\executors\fail_safe_executor.py", line 47, in _check_completed_futures future.result() File "C:\Python35\lib\concurrent\futures_base.py", line 398, in result return self.get_result() File "C:\Python35\lib\concurrent\futures_base.py", line 357, in get_result raise self._exception File "C:\Python35\lib\concurrent\futures\thread.py", line 55, in run result = self.fn(*self.args, **self.kwargs) File "D:\Git\ethereum-etl\ethereumetl\jobs\batch_export_job.py", line 73, in _fail_safe_export_batch self._export_batch(block_number, block_number) File "D:\Git\ethereum-etl\ethereumetl\jobs\export_blocks_job.py", line 65, in _export_batch response = self.ipc_wrapper.make_request(json.dumps(blocks_rpc)) File "D:\Git\ethereum-etl\ethereumetl\ipc.py", line 57, in make_request with self._lock, self._socket as sock: File "C:\Python35\lib\site-packages\web3\providers\ipc.py", line 45, in enter self.sock = self._open() File "C:\Python35\lib\site-packages\web3\providers\ipc.py", line 58, in _open return get_ipc_socket(self.ipc_path) File "C:\Python35\lib\site-packages\web3\providers\ipc.py", line 26, in get_ipc_socket return NamedPipe(ipc_path) File "C:\Python35\lib\site-packages\web3\utils\windows.py", line 19, in init raise IOError(err) OSError: (231, 'CreateFile', 'All pipe instances are busy.')

Any ideas how to fix? thanks a lot!!!

[medvedev1088]

Try adding -w 1 option to python scripts in export_all.sh. Also try restarting geth/parity node and see if it works.

[domatka78]

the -w 1 option worked, thanks a lot!

[medvedev1088]

Could you please also check if -w 2 works? I assume the issue is that Windows pipes can not be connected from multiple threads/processes.

[domatka78]

I checked with -w 2. Worked one time but then I tried again with larger block range and it failed. So I guess you're right about pipes and Windows

Interestingly if I run the export_blocks_and_transactions script directly (without specifying -w) it works no matter how long block range I give it. The issue is then probably in the loop in the .sh script. I can help investigate further, just tell me what to do

[medvedev1088]

Thank you for checking this! I think the issue is that the pipe file is not closed when the export job is finished. So when running export_all.sh it will open new connections with the pipe file while the old ones are still open from the previous python script invocation. This issue can also be submitted to web3.py since there they don't handle socket/pipe file closing either.

Could you please also try with -p http://localhost:8545 parameter for export_all.sh, and see if the same issue happens. This will use the RPC instead of IPC.

Thank you!

[domatka78]

sorry for the late reply, was busy with sth else. Using -p http://localhost:8545 didn't help (still same error if using more than 1 worker on Windows). But using infura.io works great with default workers