search

ericmandel / pyjs9

Python connection to JS9
http://js9.si.edu
Other
16 stars 17 forks source link

Loading data via pyjs9 #11

Closed mgckind closed 5 years ago

mgckind commented 5 years ago

Hi @ericmandel, We have some time to revisit the whole Jupyter + JS9 again. We are running a local sever via node and we are trying to load data using pyjs9. Every call seems to work, but loading file or even setting numpy array doesn't. I'm using python3.

Do you have any insights? Below is the piece of code and error

from astropy.io import fits
F = fits.open(name)
data_numpy = F[0].data
data_numpy.dtype
j = pyjs9.JS9(host='http://localhost:2718', id='{}JS9'.format(jid))
j.SetNumpy(data_numpy)

However, I can open file using the window File--> open option, and I can manipulate the display, just is loading the data via pyjs9 is when it fails

I think this is related to chunking (as I've seen this error before in different debugging cases), related to urllib3 and requets Using SetFITS gives a similar error (ChunkedEncodingError)

Here is the error:

ConnectionResetError Traceback (most recent call last) ~/miniconda3/lib/python3.6/site-packages/urllib3/response.py in _error_catcher(self) 396 try: --> 397 yield 398

~/miniconda3/lib/python3.6/site-packages/urllib3/response.py in read(self, amt, decode_content, cache_content) 478 cache_content = False --> 479 data = self._fp.read(amt) 480 if amt != 0 and not data: # Platform-specific: Buggy versions of Python.

~/miniconda3/lib/python3.6/http/client.py in read(self, amt) 448 b = bytearray(amt) --> 449 n = self.readinto(b) 450 return memoryview(b)[:n].tobytes()

~/miniconda3/lib/python3.6/http/client.py in readinto(self, b) 492 # (for example, reading in 1k chunks) --> 493 n = self.fp.readinto(b) 494 if not n and b:

~/miniconda3/lib/python3.6/socket.py in readinto(self, b) 585 try: --> 586 return self._sock.recv_into(b) 587 except timeout:

ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

ProtocolError Traceback (most recent call last) ~/miniconda3/lib/python3.6/site-packages/requests/models.py in generate() 749 try: --> 750 for chunk in self.raw.stream(chunk_size, decode_content=True): 751 yield chunk

~/miniconda3/lib/python3.6/site-packages/urllib3/response.py in stream(self, amt, decode_content) 530 while not is_fp_closed(self._fp): --> 531 data = self.read(amt=amt, decode_content=decode_content) 532

~/miniconda3/lib/python3.6/site-packages/urllib3/response.py in read(self, amt, decode_content, cache_content) 495 # Content-Length are caught. --> 496 raise IncompleteRead(self._fp_bytes_read, self.length_remaining) 497

~/miniconda3/lib/python3.6/contextlib.py in exit(self, type, value, traceback) 98 try: ---> 99 self.gen.throw(type, value, traceback) 100 except StopIteration as exc:

~/miniconda3/lib/python3.6/site-packages/urllib3/response.py in _error_catcher(self) 414 # This includes IncompleteRead. --> 415 raise ProtocolError('Connection broken: %r' % e, e) 416

ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

ChunkedEncodingError Traceback (most recent call last) ~/miniconda3/lib/python3.6/site-packages/pyjs9/init.py in send(self, obj, msg) 316 try: --> 317 url = requests.get(host + '/' + msg, params=jstr) 318 except IOError as e:

~/miniconda3/lib/python3.6/site-packages/requests/api.py in get(url, params, kwargs) 74 kwargs.setdefault('allow_redirects', True) ---> 75 return request('get', url, params=params, kwargs) 76

~/miniconda3/lib/python3.6/site-packages/requests/api.py in request(method, url, kwargs) 59 with sessions.Session() as session: ---> 60 return session.request(method=method, url=url, kwargs) 61

~/miniconda3/lib/python3.6/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json) 532 send_kwargs.update(settings) --> 533 resp = self.send(prep, **send_kwargs) 534

~/miniconda3/lib/python3.6/site-packages/requests/sessions.py in send(self, request, **kwargs) 685 if not stream: --> 686 r.content 687

~/miniconda3/lib/python3.6/site-packages/requests/models.py in content(self) 827 else: --> 828 self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b'' 829

~/miniconda3/lib/python3.6/site-packages/requests/models.py in generate() 752 except ProtocolError as e: --> 753 raise ChunkedEncodingError(e) 754 except DecodeError as e:

ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

OSError Traceback (most recent call last)

in () 1 #j = pyjs9.JS9(host='http://localhost:2718', id='{}JS9'.format(jid)) ----> 2 j.SetNumpy(data_numpy) ~/miniconda3/lib/python3.6/site-packages/pyjs9/__init__.py in SetNumpy(self, arr, filename, dtype) 505 hdu['filename'] = filename 506 # send encoded file to JS9 for display --> 507 return self.Load(hdu) 508 509 else: ~/miniconda3/lib/python3.6/site-packages/pyjs9/__init__.py in Load(self, *args) 560 >>> j.Load('png/m13.png', {'scale':'linear', 'colormap':'sls'}) 561 """ --> 562 return self.send({'cmd': 'Load', 'args': args}) 563 564 def LoadProxy(self, *args): ~/miniconda3/lib/python3.6/site-packages/pyjs9/__init__.py in send(self, obj, msg) 317 url = requests.get(host + '/' + msg, params=jstr) 318 except IOError as e: --> 319 raise IOError('Cannot connect to {0}: {1}'.format(host, e)) 320 urtn = url.text 321 if 'ERROR:' in urtn: OSError: Cannot connect to http://localhost:2718: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
ericmandel commented 5 years ago

Hi @mgckind, glad to hear from you! Coincidentally, this morning we updated pyjs9 to run Python3 only (no more support for Python 2), so if your pyjs9 was pulled before this morning, would you please update again and see if its better? The newest version uses a different socketio library, as it was pointed out that the old one did not work properly with Python3.

If upgrading does not fix this, I'll get into it ... I have to install numpy and everything ...

ericmandel commented 5 years ago

It would be good to know the value of pyjs9.js9Globals["transport"]. I guessing its "html" but it should be "socketio", and it probably will be "socketio" once you upgrade and use the new socketio library.

mgckind commented 5 years ago

Thanks for the quick answer, I've upgraded and pyjs9.js9Globals["transport"] is still "html" and I still get the same error. I changed it to "scoketio" and I get "WARNING:root:socketio connect failed: name 'socketio' is not defined, using html" . What socketio library are you using now?

mgckind commented 5 years ago

Sorry, I've installed python-socketio and now the transport is via websockets and everything is working fine :) . Maybe it should be added to the requirements in the setup.py file. I can load numpy and fits object fine. Thanks! I'll keep you posted of this new development. Feel free to close this issue. Best, Matias

ericmandel commented 5 years ago

If "transport" is "html", then something is wrong with the socketio install and pyjs9 is dropping back to "html" transport, which is not what we want. So we need to concentrate on that.

Here is my test, which displayed the image correctly:

>>> from astropy.io import fits
>>> import pyjs9
>>> F = fits.open('/Users/eric/data/m13.fits')
>>> data_numpy = F[0].data
>>> data_numpy.dtype
dtype('>i2')
>>> j = pyjs9.JS9()
>>> j.SetNumpy(data_numpy)

I have to go out to my little town's Annual Town Meeting (yes, the whole town gets together to vote on the annual budget, its a pretty neat exercise in participatory democracy). But we need to get the socketio working for you. I assume you have the js9Helper running?

ericmandel commented 5 years ago

Great ... the "html" transport should work fine, I'll check it again tomorrow ... the chunking probably is a problem ...

mgckind commented 5 years ago

Thanks again, yes with socketio it works fine (it defaults to it after restarting the server). If I change that to html it fails. But using socketio is ok. and yes, I'm using the js9Helper

ericmandel commented 5 years ago

Maybe it should be added to the requirements in the setup.py file.

Yes, we can consider this after gaining some confidence that the python-socketio library is going to be stable and long-lived.

ericmandel commented 5 years ago

Maybe it should be added to the requirements in the setup.py file.

Yes, we can consider this after gaining some confidence that the python-socketio library is going to be stable and long-lived.

ericmandel commented 5 years ago

@mgckind re-opening this for a bit ... I more or less fixed the "html" transport problem by switching the method from GET to POST. The GET method works for small images, while POST works for larger images. But there still seems to be a Python-specific limitation on the amount of data that can be transferred between Python and JS9. For example, I can now send a 4096x4096x4byte numpy array to JS9 but I can't retrieve it using Python (though it can be retrieved using the js9 script in the shell), either using "html" or "socketio" transport. I'll need help if we want to deal with this ... for now, I look for the transport failure and throw and error. Let me know if/how you want to proceed ...

The new pyjs9 code is updated in Github.

ericmandel commented 5 years ago

@mgckind I changed the retrieveAs method from "base64" to "array". The former was faster in the early days, but the latter allows larger images to be transported. This helps, esp. with "html" transport, but unfortunately, the "socketio" method still seems to have more severe limitations: I can now retrieve a 4096x4096x4byte data array using "html" transport, but not "socketio", which throws the error mentioned above.

mgckind commented 5 years ago

I can confirm now that the html transport works for us with latest version, we are not working with super lar images so we haven't faced that problem. Probably that would be a separate issue, how to load/retrieve large images via pyja9. Thanks for looking into this