onedata / fs-onedatafs

OnedataFS is a PyFilesystem interface to Onedata virtual file system
MIT License
0 stars 1 forks source link

ArgumentError when using numpy types for reading files #2

Open enolfc opened 5 years ago

enolfc commented 5 years ago

Reading one large netCDF file using xarray results in errors due to wrong arguments to read:

ArgumentError: Python argument types in
    OnedataFileHandle.read(OnedataFileHandle, numpy.int64, int)
did not match C++ signature:
    read(OnedataFileHandle {lvalue}, long, unsigned long)

The self.pos value was updated before by a call to seek() with a numpy.int64.

Code to reproduce:

import numpy as np

from fs.onedatafs import OnedataFS
onedata_provider_host = "plg-cyfronet-01.datahub.egi.eu"
onedata_access_token = "MDA.."

# Create connection to Oneprovider
odfs = OnedataFS(onedata_provider_host, onedata_access_token)
h = np.int64(100)
f = odfs.open('/EGI Foundation/Untitled Folder/tas_1991_2016_NLD.csv', 'rb')
f.seek(h)
f.read(10)

The original code was doing this:

import xarray as xr
f = odfs.open('/EGI Foundation/Untitled Folder/data2/ERA5_rainfall_and_temperature.nc', 'rb')
ds = xr.open_dataset(f)
bkryza commented 5 years ago

@enolfc This looks like a problem of the C++ binding which is picky about types passed to the methods - I will fix this by adding the cast inside the seek and read, but can you check for now that it works if you change the code to:

f.seek(int(h))
f.read(10)
enolfc commented 5 years ago

I have changed the code in the seek function to do a the casting, it manages to progress, so it seems the right approach

enolfc commented 5 years ago

3 should fix this issue