Downloading large file to directory throws IsADirectoryError

chStaiger commented 2 months ago

We are trying to download a 50M file with

session.data_objects.get("/zone/home/user/50Mfile.dat", "/scratch/user/downloaded/'")

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home2/user/.venv/ibridges_venv_1/lib64/python3.11/site-packages/irods/manager/data_object_manager.py", line 165, in get
    self._download(path, local_path, num_threads = num_threads, **options)
  File "/home2/user/.venv/ibridges_venv_1/lib64/python3.11/site-packages/irods/manager/data_object_manager.py", line 145, in _download
    if not self.parallel_get( (obj,o), local_path, num_threads = num_threads,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home2/user/.venv/ibridges_venv_1/lib64/python3.11/site-packages/irods/manager/data_object_manager.py", line 269, in parallel_get
    return parallel.io_main( self.sess, data_or_path_, parallel.Oper.GET | (parallel.Oper.NONBLOCKING if async_ else 0), file_,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home2/user/.venv/ibridges_venv_1/lib64/python3.11/site-packages/irods/parallel.py", line 469, in io_main
    retval = _io_multipart_threaded (Operation, (Data, Io), replica_token, resc_hier, session, fname, total_bytes,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home2/user/.venv/ibridges_venv_1/lib64/python3.11/site-packages/irods/parallel.py", line 357, in _io_multipart_threaded
    File = gen_file_handle()
           ^^^^^^^^^^^^^^^^^
  File "/home2/user/.venv/ibridges_venv_1/lib64/python3.11/site-packages/irods/parallel.py", line 356, in <lambda>
    gen_file_handle = lambda: open(fname, Operation.disk_file_mode(initial_open = (counter == 1)))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
IsADirectoryError: [Errno 21] Is a directory: '/scratch/user/downloaded/'

When we download the same file to a distinct file destination, everything works fine:

session.data_objects.get("/zone/home/user/50Mfile.dat", "/scratch/user/downloaded/50Mfile.dat")

The error lies in this line https://github.com/irods/python-irodsclient/blob/1c9ca0315001e6f05a103bb9d95ae4de02d304fb/irods/manager/data_object_manager.py#L145

local_path should be local_file

PR for a fix will follow immediately :)

d-w-moore commented 2 months ago

@chStaiger good find!

alanking commented 2 months ago

Will assign @qubixes as soon as collaboration status is established.

irods / python-irodsclient

Downloading large file to directory throws IsADirectoryError #543