Closed tjcrone closed 3 years ago
tjcrone
commented
3 years ago I should point out that this appeared to work for some of my students intermittently. Which is odd. I don't think I managed to get the entire HTML repr of the dataset printed out properly.
martindurant
commented
3 years ago Perhaps you want to keep the file open explicitly
f = gdfs.open('CRND0103-2017-NY_Millbrook_3_W.nc').open()instead of using with - but I don't know why the h5py driver needs to read additional metadata after the initial parse. Actually, I am assuming the driver backend - maybe you have different "engines" available to xarray than the students, causing the different behaviour.
tjcrone
commented
3 years ago We are all working on the same Pangeo JupyterHub and the same notebook image, so we should have the same engines.
Anyway, tried your suggestion and got this:
f = gdfs.open('CRND0103-2017-NY_Millbrook_3_W.nc').open()
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-11-076798fa5f88> in <module>
----> 1 f = gdfs.open('CRND0103-2017-NY_Millbrook_3_W.nc').open()
AttributeError: 'GoogleDriveFile' object has no attribute 'open'And it appears to be correct about that:
dir(f)
['DEFAULT_BLOCK_SIZE',
'__abstractmethods__',
'__class__',
'__del__',
'__delattr__',
'__dict__',
'__dir__',
'__doc__',
'__enter__',
'__eq__',
'__exit__',
'__format__',
'__ge__',
'__getattribute__',
'__gt__',
'__hash__',
'__init__',
'__init_subclass__',
'__iter__',
'__le__',
'__lt__',
'__module__',
'__ne__',
'__new__',
'__next__',
'__reduce__',
'__reduce_ex__',
'__repr__',
'__setattr__',
'__sizeof__',
'__str__',
'__subclasshook__',
'_abc_impl',
'_checkClosed',
'_checkReadable',
'_checkSeekable',
'_checkWritable',
'_closed',
'_fetch_range',
'_initiate_upload',
'_upload_chunk',
'autocommit',
'blocksize',
'cache',
'close',
'closed',
'commit',
'details',
'discard',
'end',
'file_id',
'fileno',
'flush',
'fs',
'info',
'isatty',
'kwargs',
'loc',
'mode',
'path',
'read',
'readable',
'readinto',
'readinto1',
'readline',
'readlines',
'readuntil',
'seek',
'seekable',
'size',
'start',
'tell',
'truncate',
'writable',
'write',
'writelines']Thanks for fielding these dumb questions. Once I have a good sense of which of these issues are mine and which are issues with the back-end, I will start working on fixes.
martindurant
commented
3 years ago Sorry, my mistake: I was referring to fsspec.open.
Your code could be
gdfs = gdrivefs.GoogleDriveFileSystem(root_file_id='1PCBDhk5f3v5PoPCY3Rdcqgy4S_Yj2kCC', token='cache')
f = gdfs.open('CRND0103-2017-NY_Millbrook_3_W.nc')
ds = xr.open_dataset(f)
ds
...
f.close()This suggestion works great. Thank you @martindurant much appreciated.
I am trying to open an Xarray dataset on Google Drive. No problem listing the contents of a directory or reading a Pandas dataframe with read_csv(). However xr.open_dataset() causes a strange "I/O operation on closed file" error, not when it is read opened, but when I try to print out the details. It appears to print some of the details and then error out with "closed file". Any idea what I am doing wrong? Thanks. I'm at HEAD on gdrivefs and 0.8.4 on fsspec. I believe this would be a working example for anyone: