search

Isilon / isilon_sdk_python

Official generated source of the Isilon SDK Python language bindings.
36 stars 33 forks source link

copying multiple files at once #11

Closed isantana-bio closed 1 year ago

isantana-bio commented 5 years ago

I recently became aware of this python sdk. I need to copy 600-3000 files (300MB-100GB per file) to various NFS destinations at the same time on the same cluster about twice or so per week. Is there a method to be able to do this in parallel or/and is it ok to have multiple threads using the same api user ?

Also, I was told by our isilon IT that there is an issue with the clone feature. Can someone confirm?

fr3man1 commented 5 years ago

Hi, What about native synciq policy? Have you checked that?

isantana-bio commented 5 years ago

we have tried synciq, generally for copying entire folders, it requires source/target (?) read only and it's not possible in this case. Another complication is that for the case in question, source files are many and scattered across multiple folders, the same goes for their respective copy targets.

fr3man1 commented 5 years ago

Read only lock is only for the final sync phase, while most of the replication makes only the destination read - only.

isantana-bio commented 5 years ago

Unfortunately, my sources/destinations can not be read only at any point. Other processes can be adding more files in those areas at the same time.

nalinkanwar commented 1 year ago

Closing as inactive; Please file this as a feature request/enhancement to OneFS using official support channels. This Issues page is for SDK related issues only.