world.py and usa.py are exported directly from world.ipynb and usa.ipynb. The Jupyter notebooks are the "source of truth".
On macOS and Windows I can run the Jupyter notebooks without any issues.
On Linux (Ubuntu 16.04) I'm trying to run python world.py. It's a headless system, but I've installed Xvfb. I've installed all dependencies via conda, except for fuzzyset which is installed via pip.
Most of world.py works fine on Linux. Then it tries to spawn 4 workers, one for each CPU:
if __name__ == '__main__':
# one worker per CPU
workers = psutil.cpu_count(logical = False)
# MP voodoo. spawn seems to work on Mac and Win.
# Other methods fail in bizarre ways.
mp.set_start_method('spawn')
# ... stuff here ...
for map_bundle in [(c19g, 'plain numbers'), (c19relg, 'per capita')]:
# ... stuff here ...
# Start the pool, then break it down after each data type.
# Probably safer this way.
p = Pool(processes = workers)
work_out = p.map(make_map, warglist)
p.close()
make_map() is in workers.py. It works on macOS / Win from the Jupyter notebook - it spawns multiple workers, multiple Orca processes. It fails on Linux (see error below). For Linux, I've tried to add plotly.io.orca.ensure_server() - no luck.
Do I need to start one Orca server per worker? On separate ports? Or what's causing the error?
Traceback (most recent call last):
File "/home/c19/miniconda3/lib/python3.7/site-packages/urllib3/connection.py", line 157, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/home/c19/miniconda3/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/home/c19/miniconda3/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/c19/miniconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/home/c19/miniconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/home/c19/miniconda3/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/home/c19/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/home/c19/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/home/c19/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/home/c19/miniconda3/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/home/c19/miniconda3/lib/python3.7/site-packages/urllib3/connection.py", line 184, in connect
conn = self._new_conn()
File "/home/c19/miniconda3/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fd77670b690>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/c19/miniconda3/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/c19/miniconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/c19/miniconda3/lib/python3.7/site-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=36915): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd77670b690>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/c19/miniconda3/lib/python3.7/site-packages/plotly/io/_orca.py", line 1562, in to_image
figure=fig_dict, format=format, scale=scale, width=width, height=height
File "/home/c19/miniconda3/lib/python3.7/site-packages/retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "/home/c19/miniconda3/lib/python3.7/site-packages/retrying.py", line 212, in call
raise attempt.get()
File "/home/c19/miniconda3/lib/python3.7/site-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/home/c19/miniconda3/lib/python3.7/site-packages/six.py", line 703, in reraise
raise value
File "/home/c19/miniconda3/lib/python3.7/site-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/home/c19/miniconda3/lib/python3.7/site-packages/plotly/io/_orca.py", line 1471, in request_image_with_retrying
response = post(server_url + "/", data=json_str)
File "/home/c19/miniconda3/lib/python3.7/site-packages/requests/api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "/home/c19/miniconda3/lib/python3.7/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/home/c19/miniconda3/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/home/c19/miniconda3/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/home/c19/miniconda3/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=36915): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd77670b690>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/c19/miniconda3/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/home/c19/miniconda3/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/home/c19/git/c19/workers.py", line 211, in make_map
fig.write_image(map_full_path, scale=1.0)
File "/home/c19/miniconda3/lib/python3.7/site-packages/plotly/basedatatypes.py", line 3193, in write_image
return pio.write_image(self, *args, **kwargs)
File "/home/c19/miniconda3/lib/python3.7/site-packages/plotly/io/_orca.py", line 1770, in write_image
fig, format=format, scale=scale, width=width, height=height, validate=validate
File "/home/c19/miniconda3/lib/python3.7/site-packages/plotly/io/_orca.py", line 1594, in to_image
info=status_str
ValueError:
For some reason plotly.py was unable to communicate with the
local orca server process, even though the server process seems to be running.
Please review the process and connection information below:
orca status
-----------
state: running
executable: /usr/bin/xvfb-run --auto-servernum --server-args -screen 0 640x480x24 +extension RANDR +extension GLX /home/c19/miniconda3/bin/orca
version: 1.3.1
port: 36915
pid: 8509
command: ['/usr/bin/xvfb-run', '--auto-servernum', '--server-args', '-screen 0 640x480x24 +extension RANDR +extension GLX', '/home/c19/miniconda3/bin/orca', 'serve', '-p', '36915', '--plotly', '/home/c19/miniconda3/lib/python3.7/site-packages/plotly/package_data/plotly.min.js', '--graph-only', '--mathjax', 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js']
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "world.py", line 362, in <module>
work_out = p.map(make_map, warglist)
File "/home/c19/miniconda3/lib/python3.7/multiprocessing/pool.py", line 268, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/home/c19/miniconda3/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
ValueError:
For some reason plotly.py was unable to communicate with the
local orca server process, even though the server process seems to be running.
Please review the process and connection information below:
orca status
-----------
state: running
executable: /usr/bin/xvfb-run --auto-servernum --server-args -screen 0 640x480x24 +extension RANDR +extension GLX /home/c19/miniconda3/bin/orca
version: 1.3.1
port: 36915
pid: 8509
command: ['/usr/bin/xvfb-run', '--auto-servernum', '--server-args', '-screen 0 640x480x24 +extension RANDR +extension GLX', '/home/c19/miniconda3/bin/orca', 'serve', '-p', '36915', '--plotly', '/home/c19/miniconda3/lib/python3.7/site-packages/plotly/package_data/plotly.min.js', '--graph-only', '--mathjax', 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js']
code: https://github.com/FlorinAndrei/c19 (clone this repo)
world.py and usa.py are exported directly from world.ipynb and usa.ipynb. The Jupyter notebooks are the "source of truth".
On macOS and Windows I can run the Jupyter notebooks without any issues.
On Linux (Ubuntu 16.04) I'm trying to run
python world.py. It's a headless system, but I've installed Xvfb. I've installed all dependencies via conda, except for fuzzyset which is installed via pip.Most of world.py works fine on Linux. Then it tries to spawn 4 workers, one for each CPU:
make_map() is in workers.py. It works on macOS / Win from the Jupyter notebook - it spawns multiple workers, multiple Orca processes. It fails on Linux (see error below). For Linux, I've tried to add
plotly.io.orca.ensure_server()- no luck.Do I need to start one Orca server per worker? On separate ports? Or what's causing the error?