counsyl / stor

A cross-compatible CLI and Python API for accessing block and object storage
https://counsyl.github.io/stor/
Other
35 stars 12 forks source link

stor needs to pass 'rb' flag to dxpy #108

Open ying-w opened 5 years ago

ying-w commented 5 years ago

Currently on python3 if you're trying to stor.open() a file a DNAnexus, it will give a unicode error if you're opening a non-text (binary file).

Recently a PR was merged that adds an 'rb' mode for dxpy. Thus read_object() function in dx.py needs to be updated to reflect this.

For example, given a .png file that you want to view within jupyter notebook with DNAnexus id: project-abc:file-xyz

import stor
import dxpy
from PIL import Image

img = stor.Path('dx://project-abc:file-xyz')
with stor.open(img, "rb") as f:
    display(Image.open(io.BytesIO(f.read()))) # this will give UnicodeDecodeError

with dxpy.DXFile(dxid='file-xyz', project='project-abc', mode='rb') as dp:
    display(Image.open(io.BytesIO(dp.read()))) # this will work

dxpy.__version__ # '0.277.0+g18cd5634'
jtratner commented 5 years ago

@anujkumar93 - looks like they updated the underlying API - we should update for Python 3 so we can support text files!