santosjorge / cufflinks

Productivity Tools for Plotly + Pandas
MIT License
3.03k stars 676 forks source link

Can't run multiple processes that import Cufflinks and set config file #192

Open rwolst opened 5 years ago

rwolst commented 5 years ago

When running multiple python processes in parallel that import cufflinks and set the config file, I get an incorrect exception that I don't have file permissions.

To reproduce this, can use the following Python script

# test.py

import cufflinks as cf
cf.set_config_file(offline=True, world_readable=True, theme='white')

and run a bash script

# test.sh

python test.py &
python test.py &
python test.py &
python test.py &

to run the processes in parallel.

The reason this is causing an incorrect error (I do have file permissions) is due to this snippet in auth.py, that creates a directory and removes it to check for the correct file permissions.

The problem is that all the processes are trying to create and remove the same directory. Hence we get an incorrect error leading us into the except clause that then thinks we don't have correct file permissions.

An easy fix would be to name the test directory based on the PID of the process running it i.e. change

TEST_DIR = os.path.join(os.path.expanduser("~"), ".test")

to

TEST_DIR = os.path.join(os.path.expanduser("~"), ".test_" + str(os.getpid()))
antoniomolram commented 3 years ago

just to overcome the issue just do this:


import cufflinks as cf
# Overcome problems when executing in parallel
cf.auth.TEST_DIR = os.path.join(os.path.expanduser("~"), ".test_" + str(os.getpid()))
cf.set_config_file(sharing='public',theme='white',offline=True)