irods-contrib / idrop

iDrop desktop and web GUI
Other
16 stars 9 forks source link

Data Not Found doing idrop desktop download from file in public via softlink #54

Closed michael-conway closed 10 years ago

michael-conway commented 10 years ago

I'm GET'ing from a mounted collection in public. When I do an iget from the command-line tools it works fine but iDrop produces zero-length files.

schroeder@pivo:/tbox/IRODS_BUILD/iRODS$ ils
/tempZone/home/rods:
  wsf1
  C- /tempZone/home/rods/wsd1
schroeder@pivo:/tbox/IRODS_BUILD/iRODS$ ils ..
/tempZone/home:
  C- /tempZone/home/public
  C- /tempZone/home/rods
  C- /tempZone/home/wayne
schroeder@pivo:/tbox/IRODS_BUILD/iRODS$ ils ../public
/tempZone/home/public:
  C- /tempZone/home/public/linktowsd1
schroeder@pivo:/tbox/IRODS_BUILD/iRODS$ icd ../public
schroeder@pivo:/tbox/IRODS_BUILD/iRODS$ ils
/tempZone/home/public:
  C- /tempZone/home/public/linktowsd1
schroeder@pivo:/tbox/IRODS_BUILD/iRODS$ icd linktowsd1
schroeder@pivo:/tbox/IRODS_BUILD/iRODS$ ils
/tempZone/home/public/linktowsd1:
  wsf2
  wsf3
schroeder@pivo:/tbox/IRODS_BUILD/iRODS$ ipwd
/tempZone/home/public/linktowsd1
schroeder@pivo:/tbox/IRODS_BUILD/iRODS$ icd
schroeder@pivo:/tbox/IRODS_BUILD/iRODS$ ipwd
/tempZone/home/rods
schroeder@pivo:/tbox/IRODS_BUILD/iRODS$ icd wsd1
schroeder@pivo:/tbox/IRODS_BUILD/iRODS$ ils
/tempZone/home/rods/wsd1:
  wsf2
  wsf3
schroeder@pivo:/tbox/IRODS_BUILD/iRODS$ 

log info


2014-08-08 10:18:30,879 50714 [AWT-EventQueue-0] INFO  org.irods.jargon.conveyor.basic.BasicQueueManagerServiceImpl  - dequeueNextOperation()
2014-08-08 10:18:30,879 50714 [AWT-EventQueue-0] INFO  org.irods.jargon.conveyor.core.ConveyorExecutorServiceImpl  - set busy
2014-08-08 10:18:30,916 50751 [AWT-EventQueue-0] INFO  org.irods.jargon.conveyor.basic.BasicQueueManagerServiceImpl  - have transfer to run... setting up the new attempt:Transfer
   id:458752
   transferState:ENQUEUED
     resorceName:
   transferStatus:OK
   transferType:GET
   localAbsolutePath:/Users/hellyj
   irodsAbsolutePath:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
   createdAt:Fri Aug 08 10:18:30 PDT 2014
   updatedAt:Fri Aug 08 10:18:30 PDT 2014
2014-08-08 10:18:30,916 50751 [AWT-EventQueue-0] INFO  org.irods.jargon.conveyor.core.ConveyorExecutorServiceImpl  - processTransferAndHandleReturn
2014-08-08 10:18:30,916 50751 [AWT-EventQueue-0] INFO  org.irods.jargon.conveyor.core.ConveyorExecutorServiceImpl  - submitting transferAttempt:TransferAttempt:
   id:491520
   seq:1407518310879
   attemptStart:2014-08-08 10:18:30.916
   attemptEnd:null
   attemptStatus:OK
   errorMessage:null
   globalException:
   lastSuccessfulPath:
   totalFilesCount:0
   totalFilesTransferredSoFar:0
2014-08-08 10:18:30,917 50752 [AWT-EventQueue-0] INFO  org.irods.jargon.conveyor.core.callables.ConveyorCallableFactory  - instanceCallableForOperation()
2014-08-08 10:18:30,917 50752 [AWT-EventQueue-0] INFO  org.irods.jargon.conveyor.core.callables.ConveyorCallableFactory  - transferAttempt for callable:TransferAttempt:
   id:491520
   seq:1407518310879
   attemptStart:2014-08-08 10:18:30.916
   attemptEnd:null
   attemptStatus:OK
   errorMessage:null
   globalException:
   lastSuccessfulPath:
   totalFilesCount:0
   totalFilesTransferredSoFar:0
2014-08-08 10:18:30,918 50753 [AWT-EventQueue-0] INFO  org.irods.jargon.conveyor.basic.BasicQueueManagerServiceImpl  - transfer added:Transfer
   id:458752
   transferState:PROCESSING
     resorceName:
   transferStatus:OK
   transferType:GET
   localAbsolutePath:/Users/hellyj
   irodsAbsolutePath:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
   createdAt:Fri Aug 08 10:18:30 PDT 2014
   updatedAt:2014-08-08 10:18:30.916
2014-08-08 10:18:30,919 50754 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.basic.GridAccountServiceImpl  - irodsAccountForGridAccount()
2014-08-08 10:18:30,921 50756 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.basic.ConfigurationServiceImpl  - buildDefaultTransferControlBlockBasedOnConfiguration()
2014-08-08 10:18:30,922 50757 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.basic.ConfigurationServiceImpl  - getCachedConveyorConfigurationProperties
2014-08-08 10:18:30,923 50758 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.core.callables.AbstractConveyorCallable  - retrieveCandidateFlowSpecs()
2014-08-08 10:18:30,923 50758 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.basic.BasicFlowManagerService  - retrieveCandidateFlowSpecs()
2014-08-08 10:18:30,925 50760 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.core.callables.AbstractConveyorCallable  - getIRODSAccountForGridAccount()
2014-08-08 10:18:30,926 50761 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.basic.GridAccountServiceImpl  - irodsAccountForGridAccount()
2014-08-08 10:18:30,926 50761 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.core.callables.PutConveyorCallable  - processCallForThisTransfer()
2014-08-08 10:18:30,928 50763 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.AbstractIRODSMidLevelProtocolFactory  - instance() method...calling connection life cycle
2014-08-08 10:18:30,928 50763 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.AbstractIRODSMidLevelProtocolFactory  - create connection....
2014-08-08 10:18:30,928 50763 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.IRODSTCPConnectionFactoryImpl  - instance()
2014-08-08 10:18:30,928 50763 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.AbstractConnection  - AbstractConnection()
2014-08-08 10:18:30,928 50763 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.AbstractConnection  - using internal cache buffer of size:65535
2014-08-08 10:18:30,928 50763 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.AbstractConnection  - opening irods socket
2014-08-08 10:18:30,942 50777 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.AbstractIRODSMidLevelProtocolFactory  - ...have connection, now authenticate given the auth scheme in the iRODS account...
2014-08-08 10:18:30,942 50777 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.AbstractIRODSMidLevelProtocolFactory  - authenticate()
2014-08-08 10:18:30,942 50777 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.AbstractIRODSMidLevelProtocolFactory  - get auth mechanism
2014-08-08 10:18:30,942 50777 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.AuthenticationFactoryImpl  - instanceAuthMechanism()
2014-08-08 10:18:30,942 50777 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.AuthenticationFactoryImpl  - account is anonymous, use default auth scheme
2014-08-08 10:18:30,942 50777 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.AuthenticationFactoryImpl  - authScheme:STANDARD
2014-08-08 10:18:30,942 50777 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.AuthenticationFactoryImpl  - using standard auth
2014-08-08 10:18:30,942 50777 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.AbstractIRODSMidLevelProtocolFactory  - authenticate...
2014-08-08 10:18:31,236 51071 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.AuthMechanism  - startup response:StartupResponseData:
   status:0
   relVersion:rods3.2
   apiVersion:d
   reconnPort:0
   reconnAddr:
   cookie:0
2014-08-08 10:18:31,236 51071 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.StandardIRODSAuth  - authenticate
2014-08-08 10:18:31,236 51071 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.StandardIRODSAuth  - sending standard irods password
2014-08-08 10:18:31,277 51112 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.StandardIRODSAuth  - auth was successful
2014-08-08 10:18:31,277 51112 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.StandardIRODSAuth  - auth response was:org.irods.jargon.core.connection.auth.AuthResponse@246ef157
2014-08-08 10:18:31,277 51112 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.AbstractIRODSMidLevelProtocolFactory  - ..authenticated...now decorate and return...
2014-08-08 10:18:31,277 51112 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.AbstractIRODSMidLevelProtocolFactory  - decorate()
2014-08-08 10:18:31,277 51112 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.EnvironmentalInfoAccessor  - getting irods server properties
2014-08-08 10:18:31,277 51112 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.DiscoveredServerPropertiesCache  - now retriving server properties from cache with zone:cmmap001
2014-08-08 10:18:31,277 51112 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.EnvironmentalInfoAccessor  - returning cached props:IRODS server properties
    icat enabled?:ICAT_ENABLED
    Server boot time:1407516167
    Rel version:rods3.2
    API version:d
    zone:cmmap001
    eirods:false

2014-08-08 10:18:31,277 51112 [pool-1-thread-1] INFO  org.irods.jargon.core.connection.AbstractIRODSMidLevelProtocolFactory  - IRODS server properties
    icat enabled?:ICAT_ENABLED
    Server boot time:1407516167
    Rel version:rods3.2
    API version:d
    zone:cmmap001
    eirods:false

2014-08-08 10:18:31,286 51121 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataTransferOperationsImpl  - get operation, irods source file is: /cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
2014-08-08 10:18:31,286 51121 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataTransferOperationsImpl  -   local file for get: /Users/hellyj
2014-08-08 10:18:31,286 51121 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataTransferOperationsImpl  -    specifiying resource:
2014-08-08 10:18:31,286 51121 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.io.IRODSFileImpl  - setting file name, given path = /cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
2014-08-08 10:18:31,286 51121 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.io.IRODSFileImpl  - detected local separator = /
2014-08-08 10:18:31,286 51121 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.io.IRODSFileImpl  - file name was set as: GATE_IDEAL_S_00450_127_to_140.nc
2014-08-08 10:18:31,286 51121 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataTransferOperationsImpl  - getOperation()
2014-08-08 10:18:31,286 51121 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataTransferOperationsImpl  - get operation, irods source file is: /cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
2014-08-08 10:18:31,286 51121 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataTransferOperationsImpl  -   local file for get: /Users/hellyj
2014-08-08 10:18:31,287 51122 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataTransferOperationsImpl  - file name normalized:/Users/hellyj/GATE_IDEAL_S_00450_127_to_140.nc
2014-08-08 10:18:31,287 51122 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataTransferOperationsImpl  - am I rerouting?
2014-08-08 10:18:31,287 51122 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.io.IRODSFileImpl  - isDirectory() for path:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
2014-08-08 10:18:31,287 51122 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.io.IRODSFileImpl  - looking up objStat, not cached in file
2014-08-08 10:18:31,287 51122 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.IRODSFileSystemAOImpl  - getObjStat(final String irodsAbsolutePath)
2014-08-08 10:18:31,343 51178 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.CollectionAndDataObjectListAndSearchAOImpl  - objStat:
  absolutePath:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
   dataId:30625
   specColType:LINKED_COLL
   objectType:DATA_OBJECT
   collectionPath:/cmmap001/home/public/GigaLES_010
   objectPath:/cmmap001/home/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
   checksum:68d4604756f76ab958be1540a5428fb6
   ownerName:hellyj
   ownerZone:cmmap001
  objSize:1879066636
   cacheDir:
   cacheDirty:false
   createdAt:replNumber:0Mon Jun 16 16:56:25 PDT 2014
   modifiedAt:Mon Jun 16 16:56:25 PDT 2014
2014-08-08 10:18:31,344 51179 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.core.callables.AbstractConveyorCallable  - overall status callback signalling completion:transfer status:
   transferState:OVERALL_INITIATION
   transferType:GET
   sourceFileAbsolutePath:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
   targetFileAbsolutePath:/Users/hellyj
   targetResource:
   totalSize:0
   bytesTransferred:0
   totalFilesTransferredSoFar:0
   totalFilesSkippedSoFar:0
   totalFilesToTransfer:1
   transferException:null
   intraFileStatusReportfalse
   transferHost:cmmap001.sdsc.edu
   transferZone:cmmap001
2014-08-08 10:18:31,344 51179 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.core.callables.AbstractConveyorCallable  - handleOverallInitiation()
2014-08-08 10:18:31,344 51179 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.core.callables.AbstractConveyorCallable  - No flows to inspect
2014-08-08 10:18:31,344 51179 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataTransferOperationsImpl  - get of single file
2014-08-08 10:18:31,344 51179 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.TransferOperationsHelper  - get of single file...filtered?
2014-08-08 10:18:31,344 51179 [pool-1-thread-1] INFO  org.irods.jargon.core.transfer.DefaultTransferControlBlock  - filtering: /cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
2014-08-08 10:18:31,344 51179 [pool-1-thread-1] INFO  org.irods.jargon.core.transfer.DefaultTransferControlBlock  - no filter
2014-08-08 10:18:31,344 51179 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.TransferOperationsHelper  - filter passed, process...
2014-08-08 10:18:31,344 51179 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.io.IRODSFileImpl  - length() for path:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
2014-08-08 10:18:31,344 51179 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.core.callables.AbstractConveyorCallable  - status callback:transfer status:
   transferState:IN_PROGRESS_START_FILE
   transferType:GET
   sourceFileAbsolutePath:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
   targetFileAbsolutePath:/Users/hellyj
   targetResource:
   totalSize:1879066636
   bytesTransferred:1879066636
   totalFilesTransferredSoFar:0
   totalFilesSkippedSoFar:0
   totalFilesToTransfer:1
   transferException:null
   intraFileStatusReportfalse
   transferHost:cmmap001.sdsc.edu
   transferZone:cmmap001
2014-08-08 10:18:31,345 51180 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.core.callables.AbstractConveyorCallable  - file initiation, this is just passed on by conveyor
2014-08-08 10:18:31,345 51180 [pool-1-thread-1] INFO  org.irods.jargon.idrop.desktop.systraygui.iDrop  - transfer status callback to iDROP:transfer status:
   transferState:IN_PROGRESS_START_FILE
   transferType:GET
   sourceFileAbsolutePath:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
   targetFileAbsolutePath:/Users/hellyj
   targetResource:
   totalSize:1879066636
   bytesTransferred:1879066636
   totalFilesTransferredSoFar:0
   totalFilesSkippedSoFar:0
   totalFilesToTransfer:1
   transferException:null
   intraFileStatusReportfalse
   transferHost:cmmap001.sdsc.edu
   transferZone:cmmap001
2014-08-08 10:18:31,345 51180 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataObjectAOImpl  - getDataObjectFromIrods()
2014-08-08 10:18:31,345 51180 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataObjectAOImpl  - transferStatusCallbackListener present for getDataObjectFromIrods() method
2014-08-08 10:18:31,345 51180 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataObjectAOImpl  - irodsFileToGet:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
2014-08-08 10:18:31,345 51180 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataObjectAOImpl  - localFileToHoldData:/Users/hellyj
2014-08-08 10:18:31,345 51180 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataObjectAOImpl  - a put to a directory, just use the source file name and accept the directory as a target
2014-08-08 10:18:31,345 51180 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataObjectAOImpl  - target file name will be:/Users/hellyj/GATE_IDEAL_S_00450_127_to_140.nc
2014-08-08 10:18:31,346 51181 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.io.IRODSFileImpl  - length() for path:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
2014-08-08 10:18:31,346 51181 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataObjectAOImpl  - testing file length to set parallel transfer options
2014-08-08 10:18:31,346 51181 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataObjectAOImpl  - setting max threads cap to:4
2014-08-08 10:18:31,346 51181 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataObjectAOImpl  - target local file: /Users/hellyj/GATE_IDEAL_S_00450_127_to_140.nc
2014-08-08 10:18:31,346 51181 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataObjectAOImpl  - from source file: /cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
2014-08-08 10:18:31,347 51182 [pool-1-thread-1] INFO  org.irods.jargon.core.pub.DataObjectAOImpl  - process get after resource determined
2014-08-08 10:18:31,347 51182 [pool-1-thread-1] INFO  org.irods.jargon.core.utils.LocalFileUtils  - local file does not exist, will attempt to create local file: /Users/hellyj/GATE_IDEAL_S_00450_127_to_140.nc
2014-08-08 10:18:31,364 51199 [pool-1-thread-1] ERROR org.irods.jargon.core.pub.TransferOperationsHelper  - exception in transfer
org.irods.jargon.core.exception.DataNotFoundException: No data found
    at org.irods.jargon.core.connection.IRODSErrorScanner.checkSpecificCodesAndThrowIfExceptionLocated(IRODSErrorScanner.java:144)
    at org.irods.jargon.core.connection.IRODSErrorScanner.inspectAndThrowIfNeeded(IRODSErrorScanner.java:117)
    at org.irods.jargon.core.connection.AbstractIRODSMidLevelProtocol.processMessageInfoLessThanZero(AbstractIRODSMidLevelProtocol.java:1136)
    at org.irods.jargon.core.connection.AbstractIRODSMidLevelProtocol.readMessage(AbstractIRODSMidLevelProtocol.java:589)
    at org.irods.jargon.core.connection.AbstractIRODSMidLevelProtocol.readMessage(AbstractIRODSMidLevelProtocol.java:555)
    at org.irods.jargon.core.connection.IRODSMidLevelProtocol.irodsFunction(IRODSMidLevelProtocol.java:208)
    at org.irods.jargon.core.connection.AbstractIRODSMidLevelProtocol.irodsFunction(AbstractIRODSMidLevelProtocol.java:113)
    at org.irods.jargon.core.connection.AbstractIRODSMidLevelProtocol.irodsFunction(AbstractIRODSMidLevelProtocol.java:497)
    at org.irods.jargon.core.pub.DataObjectAOImpl.processGetAfterResourceDetermined(DataObjectAOImpl.java:1276)
    at org.irods.jargon.core.pub.DataObjectAOImpl.getDataObjectFromIrods(DataObjectAOImpl.java:1040)
    at org.irods.jargon.core.pub.TransferOperationsHelper.processGetOfSingleFile(TransferOperationsHelper.java:292)
    at org.irods.jargon.core.pub.DataTransferOperationsImpl.processGetOfSingleFile(DataTransferOperationsImpl.java:800)
    at org.irods.jargon.core.pub.DataTransferOperationsImpl.processGetAfterAnyConnectionRerouting(DataTransferOperationsImpl.java:597)
    at org.irods.jargon.core.pub.DataTransferOperationsImpl.getOperation(DataTransferOperationsImpl.java:384)
    at org.irods.jargon.core.pub.DataTransferOperationsImpl.getOperation(DataTransferOperationsImpl.java:664)
    at org.irods.jargon.conveyor.core.callables.GetConveyorCallable.processCallForThisTransfer(GetConveyorCallable.java:47)
    at org.irods.jargon.conveyor.core.callables.AbstractConveyorCallable.call(AbstractConveyorCallable.java:208)
    at org.irods.jargon.conveyor.core.callables.AbstractConveyorCallable.call(AbstractConveyorCallable.java:53)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
2014-08-08 10:18:31,364 51199 [pool-1-thread-1] WARN  org.irods.jargon.core.pub.TransferOperationsHelper  - exception will be passed back to existing callback listener
2014-08-08 10:18:31,364 51199 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.core.callables.AbstractConveyorCallable  - status callback:transfer status:
   transferState:FAILURE
   transferType:GET
   sourceFileAbsolutePath:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
   targetFileAbsolutePath:/Users/hellyj
   targetResource:
   totalSize:5984
   bytesTransferred:5984
   totalFilesTransferredSoFar:0
   totalFilesSkippedSoFar:0
   totalFilesToTransfer:1
   transferException:org.irods.jargon.core.exception.DataNotFoundException: No data found
   intraFileStatusReportfalse
   transferHost:cmmap001.sdsc.edu
   transferZone:cmmap001
2014-08-08 10:18:31,411 51246 [pool-1-thread-1] INFO  org.irods.jargon.idrop.desktop.systraygui.iDrop  - transfer status callback to iDROP:transfer status:
   transferState:FAILURE
   transferType:GET
   sourceFileAbsolutePath:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
   targetFileAbsolutePath:/Users/hellyj
   targetResource:
   totalSize:5984
   bytesTransferred:5984
   totalFilesTransferredSoFar:0
   totalFilesSkippedSoFar:0
   totalFilesToTransfer:1
   transferException:org.irods.jargon.core.exception.DataNotFoundException: No data found
   intraFileStatusReportfalse
   transferHost:cmmap001.sdsc.edu
   transferZone:cmmap001
2014-08-08 10:18:31,411 51246 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.core.callables.AbstractConveyorCallable  - overall status callback signalling completion:transfer status:
   transferState:OVERALL_COMPLETION
   transferType:GET
   sourceFileAbsolutePath:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
   targetFileAbsolutePath:/Users/hellyj
   targetResource:
   totalSize:0
   bytesTransferred:0
   totalFilesTransferredSoFar:0
   totalFilesSkippedSoFar:0
   totalFilesToTransfer:1
   transferException:null
   intraFileStatusReportfalse
   transferHost:cmmap001.sdsc.edu
   transferZone:cmmap001
2014-08-08 10:18:31,411 51246 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.core.callables.AbstractConveyorCallable  - overall completion...updating status of transfer...
2014-08-08 10:18:31,411 51246 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.core.callables.AbstractConveyorCallable  - processOverallCompletionOfTransfer default version in AbstractConveyorCallable
2014-08-08 10:18:31,411 51246 [AWT-EventQueue-0] ERROR org.irods.jargon.idrop.desktop.systraygui.iDrop  - error occurred in transfer: transfer status:
   transferState:FAILURE
   transferType:GET
   sourceFileAbsolutePath:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
   targetFileAbsolutePath:/Users/hellyj
   targetResource:
   totalSize:5984
   bytesTransferred:5984
   totalFilesTransferredSoFar:0
   totalFilesSkippedSoFar:0
   totalFilesToTransfer:1
   transferException:org.irods.jargon.core.exception.DataNotFoundException: No data found
   intraFileStatusReportfalse
   transferHost:cmmap001.sdsc.edu
   transferZone:cmmap001
2014-08-08 10:18:31,411 51246 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.core.callables.AbstractConveyorCallable  - evaluating transfer status by inspecting items for any file level errors
2014-08-08 10:18:31,412 51247 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.core.callables.AbstractConveyorCallable  - transfer had errors and no files were successful, so mark as an error
2014-08-08 10:18:31,412 51247 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.core.callables.AbstractConveyorCallable  - status was:ERROR
2014-08-08 10:18:31,413 51248 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.basic.TransferAccountingManagementServiceImpl  - updateTransferStatusAfterOverallWarning()
2014-08-08 10:18:31,413 51248 [pool-1-thread-1] INFO  org.irods.jargon.conveyor.basic.TransferAccountingManagementServiceImpl  - transferUpdateOverall()
2014-08-08 10:18:31,413 51248 [pool-1-thread-1] INFO  org.iro
michael-conway commented 10 years ago

ran unit test successfully


package org.irods.jargon.core.unittest.functionaltest;

import java.io.File;
import java.util.Properties;

import junit.framework.Assert;

import org.irods.jargon.core.connection.IRODSAccount;
import org.irods.jargon.core.protovalues.FilePermissionEnum;
import org.irods.jargon.core.pub.CollectionAO;
import org.irods.jargon.core.pub.DataTransferOperations;
import org.irods.jargon.core.pub.IRODSAccessObjectFactory;
import org.irods.jargon.core.pub.IRODSFileSystem;
import org.irods.jargon.core.pub.MountedCollectionAO;
import org.irods.jargon.core.pub.io.IRODSFile;
import org.irods.jargon.core.pub.io.IRODSFileFactory;
import org.irods.jargon.testutils.IRODSTestSetupUtilities;
import org.irods.jargon.testutils.TestingPropertiesHelper;
import org.irods.jargon.testutils.filemanip.FileGenerator;
import org.irods.jargon.testutils.filemanip.ScratchFileUtils;
import org.junit.After;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;

/**
 * Test Case for Bug #38 https://github.com/DICE-UNC/jargon/issues/38 Data Not
 * Found doing idrop desktop download from file in public via softlink
 * 
 * @author Mike Conway - DICE
 * 
 */
public class TestBug38GetFileFromSoftLinkedPublicCollection {

    private static Properties testingProperties = new Properties();
    private static TestingPropertiesHelper testingPropertiesHelper = new TestingPropertiesHelper();
    private static ScratchFileUtils scratchFileUtils = null;
    public static final String IRODS_TEST_SUBDIR_PATH = "TestBug38GetFileFromSoftLinkedPublicCollection";
    private static IRODSTestSetupUtilities irodsTestSetupUtilities = null;
    private static IRODSFileSystem irodsFileSystem;

    @BeforeClass
    public static void setUpBeforeClass() throws Exception {
        TestingPropertiesHelper testingPropertiesLoader = new TestingPropertiesHelper();
        testingProperties = testingPropertiesLoader.getTestProperties();
        scratchFileUtils = new ScratchFileUtils(testingProperties);
        irodsFileSystem = IRODSFileSystem.instance();

        scratchFileUtils
                .clearAndReinitializeScratchDirectory(IRODS_TEST_SUBDIR_PATH);
        irodsTestSetupUtilities = new IRODSTestSetupUtilities();
        irodsTestSetupUtilities.initializeIrodsScratchDirectory();
        irodsTestSetupUtilities
                .initializeDirectoryForTest(IRODS_TEST_SUBDIR_PATH);

    }

    @AfterClass
    public static void tearDownAfterClass() throws Exception {
    }

    @Before
    public void setUp() throws Exception {
    }

    @After
    public void tearDown() throws Exception {
        irodsFileSystem.closeAndEatExceptions();
    }

    @Test
    public void testBugCase() throws Exception {

        // make a dir as test1

        String testDir = "softlinkme";
        String softLinkSubdir = "softlinkhere";
        String localRetrievedDir = "localRetrievedDir";
        String targetIrodsCollection = testingPropertiesHelper
                .buildIRODSCollectionAbsolutePathFromTestProperties(
                        testingProperties, IRODS_TEST_SUBDIR_PATH);

        IRODSAccount irodsAccount = testingPropertiesHelper
                .buildIRODSAccountFromTestProperties(testingProperties);
        IRODSAccessObjectFactory accessObjectFactory = irodsFileSystem
                .getIRODSAccessObjectFactory();
        CollectionAO collectionAO = accessObjectFactory
                .getCollectionAO(irodsAccount);
        IRODSFile irodsFile = collectionAO
                .instanceIRODSFileForCollectionPath(targetIrodsCollection + "/"
                        + testDir);
        irodsFile.mkdirs();

        // make some subfiles

        String subfile1 = "subfile1.txt";
        String subfile2 = "subfile2.txt";

        // use a local file and put it

        String absPath = scratchFileUtils
                .createAndReturnAbsoluteScratchPath(IRODS_TEST_SUBDIR_PATH);
        String localFileName = FileGenerator
                .generateFileOfFixedLengthGivenName(absPath, subfile1, 10);
        String localFileName2 = FileGenerator
                .generateFileOfFixedLengthGivenName(absPath, subfile2, 100);

        File localFile1 = new File(localFileName);
        File localFile2 = new File(localFileName2);

        DataTransferOperations dataTransferOperations = irodsFileSystem
                .getIRODSAccessObjectFactory().getDataTransferOperations(
                        irodsAccount);

        IRODSFileFactory fileFactory = irodsFileSystem
                .getIRODSFileFactory(irodsAccount);
        IRODSFile subFile = fileFactory.instanceIRODSFile(
                irodsFile.getAbsolutePath(), subfile1);

        IRODSFile subFile2 = fileFactory.instanceIRODSFile(
                irodsFile.getAbsolutePath(), subfile2);

        // TransferControlBlock tcb =

        dataTransferOperations.putOperation(localFile1, subFile, null, null);

        subFile = fileFactory.instanceIRODSFile(irodsFile.getAbsolutePath(),
                subfile2);

        dataTransferOperations.putOperation(localFile2, subFile2, null, null);
        // now make a soft link under public

        String publicDir = "/" + irodsAccount.getZone() + "/home/public";
        IRODSFile publicFile = fileFactory.instanceIRODSFile(publicDir);

        if (!publicFile.exists()) {
            publicFile.mkdirs();
            collectionAO.setAccessPermission(irodsAccount.getZone(), publicDir,
                    "public", true, FilePermissionEnum.READ);
        }

        // do an initial unmount
        MountedCollectionAO mountedCollectionAO = irodsFileSystem
                .getIRODSAccessObjectFactory().getMountedCollectionAO(
                        irodsAccount);

        String mountedDir = publicDir + "/" + softLinkSubdir;

        mountedCollectionAO.unmountACollection(mountedDir,
                irodsAccount.getDefaultStorageResource());

        // create a soft link between the 'irodsFile' that is the top subdir in
        // the user directory to the public directory

        mountedCollectionAO.createASoftLink(irodsFile.getAbsolutePath(),
                mountedDir);

        // now get one of the files

        String localCollectionAbsolutePath = scratchFileUtils
                .createAndReturnAbsoluteScratchPath(IRODS_TEST_SUBDIR_PATH
                        + '/' + localRetrievedDir);

        dataTransferOperations.getOperation(mountedDir,
                localCollectionAbsolutePath, "", null, null);

        File returnDirFile = new File(localCollectionAbsolutePath);
        Assert.assertTrue("local dir not created", returnDirFile.exists());

        Assert.assertFalse("no files returned",
                returnDirFile.listFiles().length == 0);

    }
}
hellyj commented 10 years ago

Ok, Mike.

michael-conway commented 10 years ago

Hi John,

I've updated the iRODS issue at the Consortium, it is a server issue, it appears. We probably should start thinking about short term work arouns, such as doing a copy to public?

hellyj commented 10 years ago

Unfortunately, we cannot afford the space to have multiple copies. These model runs are on the order of 10-100 TB collections.

michael-conway commented 10 years ago

Hey John,

One thing that might be too obvious. but do you have permission on the public folder that is 'inherited' and has the anonymous group? Did Wayne talk to you about that fact?

hellyj commented 10 years ago

Hi Mike. I'll check on that. I don't remember whether Wayne and I spoke about that. Why would that explain why it works with icommands but not iDrop? J.

On 9/9/14, 1:19 AM, Mike Conway wrote:

Hey John,

One thing that might be too obvious. but do you have permission on the public folder that is 'inherited' and has the anonymous group? Did Wayne talk to you about that fact?

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-54955205.

michael-conway commented 10 years ago

Hi John,

What I found was that it worked in icommands because my own irodsEnv was set to my user name, when I switched the irodsEnv to be the actual anonymous user it failed.

Does that match your experience?

On Tue, Sep 9, 2014 at 1:56 PM, hellyj notifications@github.com wrote:

Hi Mike. I'll check on that. I don't remember whether Wayne and I spoke about that. Why would that explain why it works with icommands but not iDrop? J.

On 9/9/14, 1:19 AM, Mike Conway wrote:

Hey John,

One thing that might be too obvious. but do you have permission on the public folder that is 'inherited' and has the anonymous group? Did Wayne talk to you about that fact?

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-54955205.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55009283.

hellyj commented 10 years ago

I'll do that test tonight. Wayne and I used the export command to set the username. I'll try it with the .irods/.irodsEnv file and report back. J.

On 9/9/14, 9:06 AM, Mike Conway wrote:

Hi John,

What I found was that it worked in icommands because my own irodsEnv was set to my user name, when I switched the irodsEnv to be the actual anonymous user it failed.

Does that match your experience?

On Tue, Sep 9, 2014 at 1:56 PM, hellyj notifications@github.com wrote:

Hi Mike. I'll check on that. I don't remember whether Wayne and I spoke about that. Why would that explain why it works with icommands but not iDrop? J.

On 9/9/14, 1:19 AM, Mike Conway wrote:

Hey John,

One thing that might be too obvious. but do you have permission on the public folder that is 'inherited' and has the anonymous group? Did Wayne talk to you about that fact?

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-54955205.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55009283.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55017707.

michael-conway commented 10 years ago

that would be great. I can see that same effect when I do that in jargon (using a named user versus public)

On Tue, Sep 9, 2014 at 4:03 PM, hellyj notifications@github.com wrote:

I'll do that test tonight. Wayne and I used the export command to set the username. I'll try it with the .irods/.irodsEnv file and report back. J.

On 9/9/14, 9:06 AM, Mike Conway wrote:

Hi John,

What I found was that it worked in icommands because my own irodsEnv was set to my user name, when I switched the irodsEnv to be the actual anonymous user it failed.

Does that match your experience?

On Tue, Sep 9, 2014 at 1:56 PM, hellyj notifications@github.com wrote:

Hi Mike. I'll check on that. I don't remember whether Wayne and I spoke about that. Why would that explain why it works with icommands but not iDrop? J.

On 9/9/14, 1:19 AM, Mike Conway wrote:

Hey John,

One thing that might be too obvious. but do you have permission on the public folder that is 'inherited' and has the anonymous group? Did Wayne talk to you about that fact?

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-54955205.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55009283.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55017707.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55025055.

hellyj commented 10 years ago

You mean versus 'anonymous', correct? J.

On 9/9/14, 10:09 AM, Mike Conway wrote:

that would be great. I can see that same effect when I do that in jargon (using a named user versus public)

On Tue, Sep 9, 2014 at 4:03 PM, hellyj notifications@github.com wrote:

I'll do that test tonight. Wayne and I used the export command to set the username. I'll try it with the .irods/.irodsEnv file and report back. J.

On 9/9/14, 9:06 AM, Mike Conway wrote:

Hi John,

What I found was that it worked in icommands because my own irodsEnv was set to my user name, when I switched the irodsEnv to be the actual anonymous user it failed.

Does that match your experience?

On Tue, Sep 9, 2014 at 1:56 PM, hellyj notifications@github.com wrote:

Hi Mike. I'll check on that. I don't remember whether Wayne and I spoke about that. Why would that explain why it works with icommands but not iDrop? J.

On 9/9/14, 1:19 AM, Mike Conway wrote:

Hey John,

One thing that might be too obvious. but do you have permission on the public folder that is 'inherited' and has the anonymous group? Did Wayne talk to you about that fact?

— Reply to this email directly or view it on GitHub

https://github.com/DICE-UNC/idrop/issues/54#issuecomment-54955205.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55009283.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55017707.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55025055.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55025800.

michael-conway commented 10 years ago

yes. User anonymous in the group public, with the public dir having inheritance turned on and the group public having rights.

I think an issue is that iRODS, with soft links, might be making the wrong database query, and using the source collection as the source of ACL rights, rather than the soft link.

In that case, we can think about a shim where we set the access rights in the source (original location) to allow public and try that again.

On Tue, Sep 9, 2014 at 4:18 PM, hellyj notifications@github.com wrote:

You mean versus 'anonymous', correct? J.

On 9/9/14, 10:09 AM, Mike Conway wrote:

that would be great. I can see that same effect when I do that in jargon (using a named user versus public)

On Tue, Sep 9, 2014 at 4:03 PM, hellyj notifications@github.com wrote:

I'll do that test tonight. Wayne and I used the export command to set the username. I'll try it with the .irods/.irodsEnv file and report back. J.

On 9/9/14, 9:06 AM, Mike Conway wrote:

Hi John,

What I found was that it worked in icommands because my own irodsEnv was set to my user name, when I switched the irodsEnv to be the actual anonymous user it failed.

Does that match your experience?

On Tue, Sep 9, 2014 at 1:56 PM, hellyj notifications@github.com wrote:

Hi Mike. I'll check on that. I don't remember whether Wayne and I spoke about that. Why would that explain why it works with icommands but not iDrop? J.

On 9/9/14, 1:19 AM, Mike Conway wrote:

Hey John,

One thing that might be too obvious. but do you have permission on the public folder that is 'inherited' and has the anonymous group? Did Wayne talk to you about that fact?

— Reply to this email directly or view it on GitHub

https://github.com/DICE-UNC/idrop/issues/54#issuecomment-54955205.

— Reply to this email directly or view it on GitHub <https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55009283 .

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55017707.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55025055.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55025800.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55027651.

hellyj commented 10 years ago

Hi Mike.

The icommand test works with user 'anonymous' and the directory 'public'. I can 'iget' files that are linked to actually copies. Here is a copy of the .irods/.irodEnv that I used. Feel free to try it yourself.

iRODS personal configuration file.

#

This file was automatically created during iRODS installation.

Created Sun Jul 18 13:06:23 2010

#

iRODS server host name:

irodsHost 'cmmap001.sdsc.edu'

iRODS server port number:

irodsPort 1247

Default storage resource name:

irodsDefResource 'cmmap001'

Home directory in iRODS:

irodsHome '/cmmap001/home/hellyj'

irodsHome '/cmmap001/home/public'

Current directory in iRODS:

irodsCwd '/cmmap001/home/hellyj'

irodsCwd '/cmmap001/home/public'

Account name:

irodsUserName 'hellyj'

irodsUserName 'anonymous'

Zone:

irodsZone 'cmmap001'

On 9/9/14, 10:33 AM, Mike Conway wrote:

yes. User anonymous in the group public, with the public dir having inheritance turned on and the group public having rights.

I think an issue is that iRODS, with soft links, might be making the wrong database query, and using the source collection as the source of ACL rights, rather than the soft link.

In that case, we can think about a shim where we set the access rights in the source (original location) to allow public and try that again.

On Tue, Sep 9, 2014 at 4:18 PM, hellyj notifications@github.com wrote:

You mean versus 'anonymous', correct? J.

On 9/9/14, 10:09 AM, Mike Conway wrote:

that would be great. I can see that same effect when I do that in jargon (using a named user versus public)

On Tue, Sep 9, 2014 at 4:03 PM, hellyj notifications@github.com wrote:

I'll do that test tonight. Wayne and I used the export command to set the username. I'll try it with the .irods/.irodsEnv file and report back. J.

On 9/9/14, 9:06 AM, Mike Conway wrote:

Hi John,

What I found was that it worked in icommands because my own irodsEnv was set to my user name, when I switched the irodsEnv to be the actual anonymous user it failed.

Does that match your experience?

On Tue, Sep 9, 2014 at 1:56 PM, hellyj notifications@github.com wrote:

Hi Mike. I'll check on that. I don't remember whether Wayne and I spoke about that. Why would that explain why it works with icommands but not iDrop? J.

On 9/9/14, 1:19 AM, Mike Conway wrote:

Hey John,

One thing that might be too obvious. but do you have permission on the public folder that is 'inherited' and has the anonymous group? Did Wayne talk to you about that fact?

— Reply to this email directly or view it on GitHub

https://github.com/DICE-UNC/idrop/issues/54#issuecomment-54955205.

— Reply to this email directly or view it on GitHub

<https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55009283 .

— Reply to this email directly or view it on GitHub

https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55017707.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55025055.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55025800.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55027651.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55029984.

michael-conway commented 10 years ago

OK thanks John, I’ll take a look this morning MC

On Sep 10, 2014, at 2:56 AM, hellyj notifications@github.com wrote:

Hi Mike.

The icommand test works with user 'anonymous' and the directory 'public'. I can 'iget' files that are linked to actually copies. Here is a copy of the .irods/.irodEnv that I used. Feel free to try it yourself.

iRODS personal configuration file.

#

This file was automatically created during iRODS installation.

Created Sun Jul 18 13:06:23 2010

#

iRODS server host name:

irodsHost 'cmmap001.sdsc.edu'

iRODS server port number:

irodsPort 1247

Default storage resource name:

irodsDefResource 'cmmap001'

Home directory in iRODS:

irodsHome '/cmmap001/home/hellyj'

irodsHome '/cmmap001/home/public'

Current directory in iRODS:

irodsCwd '/cmmap001/home/hellyj'

irodsCwd '/cmmap001/home/public'

Account name:

irodsUserName 'hellyj'

irodsUserName 'anonymous'

Zone:

irodsZone 'cmmap001'

On 9/9/14, 10:33 AM, Mike Conway wrote:

yes. User anonymous in the group public, with the public dir having inheritance turned on and the group public having rights.

I think an issue is that iRODS, with soft links, might be making the wrong database query, and using the source collection as the source of ACL rights, rather than the soft link.

In that case, we can think about a shim where we set the access rights in the source (original location) to allow public and try that again.

On Tue, Sep 9, 2014 at 4:18 PM, hellyj notifications@github.com wrote:

You mean versus 'anonymous', correct? J.

On 9/9/14, 10:09 AM, Mike Conway wrote:

that would be great. I can see that same effect when I do that in jargon (using a named user versus public)

On Tue, Sep 9, 2014 at 4:03 PM, hellyj notifications@github.com wrote:

I'll do that test tonight. Wayne and I used the export command to set the username. I'll try it with the .irods/.irodsEnv file and report back. J.

On 9/9/14, 9:06 AM, Mike Conway wrote:

Hi John,

What I found was that it worked in icommands because my own irodsEnv was set to my user name, when I switched the irodsEnv to be the actual anonymous user it failed.

Does that match your experience?

On Tue, Sep 9, 2014 at 1:56 PM, hellyj notifications@github.com wrote:

Hi Mike. I'll check on that. I don't remember whether Wayne and I spoke about that. Why would that explain why it works with icommands but not iDrop? J.

On 9/9/14, 1:19 AM, Mike Conway wrote:

Hey John,

One thing that might be too obvious. but do you have permission on the public folder that is 'inherited' and has the anonymous group? Did Wayne talk to you about that fact?

— Reply to this email directly or view it on GitHub

https://github.com/DICE-UNC/idrop/issues/54#issuecomment-54955205.

— Reply to this email directly or view it on GitHub

<https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55009283 .

— Reply to this email directly or view it on GitHub

https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55017707.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55025055.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55025800.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55027651.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55029984.

— Reply to this email directly or view it on GitHub.

michael-conway commented 10 years ago

Hi John,

I've got iDrop web running locally and they are currently downloading using the guest account here, I'll update you when they complete...they are rather large.

I'm currently downloading GATE_IDEAL_S_00450_127_to_140.nc

If this works here, I'll deploy this to another server instance and have you try it

michael-conway commented 10 years ago

iDrop desktop is also working. DId someone update the ACLs on the public directory or similar? I have successfully downloaded from /home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF using iDrop desktop and iDrop web.

michael-conway commented 10 years ago

Log from a successful iDrop web transfer running on my desktop


2014-09-12 09:46:29,735 [http-bio-8080-exec-4] INFO  io.IRODSFileImpl  - length() for path:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_001_to_014.nc
2014-09-12 09:46:29,736 [http-bio-8080-exec-4] INFO  pub.IRODSFileSystemAOImpl  - getObjStat(final String irodsAbsolutePath)
2014-09-12 09:46:29,816 [http-bio-8080-exec-4] INFO  pub.CollectionAndDataObjectListAndSearchAOImpl  - objStat:
  absolutePath:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_001_to_014.nc
   dataId:30546
   specColType:LINKED_COLL
   objectType:DATA_OBJECT
   collectionPath:/cmmap001/home/public/GigaLES_010
   objectPath:/cmmap001/home/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_001_to_014.nc
   checksum:6a1764e035513da94a1bfa228b804589
   ownerName:hellyj
   ownerZone:cmmap001
  objSize:1879066636
   cacheDir:
   cacheDirty:false
   createdAt:replNumber:0Mon Jun 16 19:34:54 EDT 2014
   modifiedAt:Mon Jun 16 19:34:54 EDT 2014
2014-09-12 09:46:29,958 [http-bio-8080-exec-4] INFO  io.IRODSFileInputStream  - closing file stream and file
2014-09-12 09:46:29,958 [http-bio-8080-exec-4] INFO  io.IRODSFileImpl  - closing irodsFile:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_001_to_014.nc
2014-09-12 09:46:29,959 [http-bio-8080-exec-4] INFO  pub.IRODSFileSystemAOImpl  - ileClose(final int fileDescriptor) :3
2014-09-12 09:46:30,030 [http-bio-8080-exec-4] INFO  pub.Stream2StreamAOImpl  - transfer stats:TransferStatistics
     totalByte:1879066636
     seconds:655
     kbPerSecond:2868804
2014-09-12 09:46:30,031 [http-bio-8080-exec-4] INFO  controller.FileController  - transferStats:TransferStatistics
     totalByte:1879066636
     seconds:655
     kbPerSecond:2868804
2014-09-12 09:46:30,032 [http-bio-8080-exec-4] DEBUG controller.FileController  - closing the session
2014-09-12 09:49:30,854 [http-bio-8080-exec-9] INFO  io.IRODSFileImpl  - length() for path:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
2014-09-12 09:49:30,855 [http-bio-8080-exec-9] INFO  pub.IRODSFileSystemAOImpl  - getObjStat(final String irodsAbsolutePath)
2014-09-12 09:49:30,934 [http-bio-8080-exec-9] INFO  pub.CollectionAndDataObjectListAndSearchAOImpl  - objStat:
  absolutePath:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
   dataId:30625
   specColType:LINKED_COLL
   objectType:DATA_OBJECT
   collectionPath:/cmmap001/home/public/GigaLES_010
   objectPath:/cmmap001/home/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
   checksum:68d4604756f76ab958be1540a5428fb6
   ownerName:hellyj
   ownerZone:cmmap001
  objSize:1879066636
   cacheDir:
   cacheDirty:false
   createdAt:replNumber:0Mon Jun 16 19:56:25 EDT 2014
   modifiedAt:Mon Jun 16 19:56:25 EDT 2014
2014-09-12 09:49:31,076 [http-bio-8080-exec-9] INFO  io.IRODSFileInputStream  - closing file stream and file
2014-09-12 09:49:31,077 [http-bio-8080-exec-9] INFO  io.IRODSFileImpl  - closing irodsFile:/cmmap001/home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF/GATE_IDEAL_S_00450_127_to_140.nc
2014-09-12 09:49:31,077 [http-bio-8080-exec-9] INFO  pub.IRODSFileSystemAOImpl  - ileClose(final int fileDescriptor) :3
2014-09-12 09:49:31,148 [http-bio-8080-exec-9] INFO  pub.Stream2StreamAOImpl  - transfer stats:TransferStatistics
     totalByte:1879066636
     seconds:644
     kbPerSecond:2917805
2014-09-12 09:49:31,148 [http-bio-8080-exec-9] INFO  controller.FileController  - transferStats:TransferStatistics
     totalByte:1879066636
     seconds:644
     kbPerSecond:2917805
2014-09-12 09:49:31,149 [http-bio-8080-exec-9] DEBUG controller.FileController  - closing the session
hellyj commented 10 years ago

Great. Thanks. J.

On 9/12/14, 3:42 AM, Mike Conway wrote:

Hi John,

I've got iDrop web running locally and they are currently downloading using the guest account here, I'll update you when they complete...they are rather large.

I'm currently downloading GATE_IDEAL_S_00450_127_to_140.nc

If this works here, I'll deploy this to another server instance and have you try it

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55404854.

hellyj commented 10 years ago

No, nothing has changed. J.

On 9/12/14, 3:59 AM, Mike Conway wrote:

iDrop desktop is also working. DId someone update the ACLs on the public directory or similar? I have successfully downloaded from /home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF using iDrop desktop and iDrop web.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55406907.

michael-conway commented 10 years ago

Hi John,

I was able to complete downloads in idrop web and idrop desktop to that /home/public dir, so that is somewhat confusing! I'm trying to set up an alternate server for you to log into to sign into idrop web.

MC

On Fri, Sep 12, 2014 at 2:00 PM, hellyj notifications@github.com wrote:

No, nothing has changed. J.

On 9/12/14, 3:59 AM, Mike Conway wrote:

iDrop desktop is also working. DId someone update the ACLs on the public directory or similar? I have successfully downloaded from /home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF using iDrop desktop and iDrop web.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55406907.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55439042.

hellyj commented 10 years ago

Most of the data are on a resource server that is federated with the primary. Sometimes that server hangs due to a firmware problem. Is it possible for iRODS to 'think' it is downloading something and then produce a zero-length file on the target if a resource server is down without throwing an error?

This could explain the inconsistent results.

J.

On 9/12/14, 8:39 AM, Mike Conway wrote:

Hi John,

I was able to complete downloads in idrop web and idrop desktop to that /home/public dir, so that is somewhat confusing! I'm trying to set up an alternate server for you to log into to sign into idrop web.

MC

On Fri, Sep 12, 2014 at 2:00 PM, hellyj notifications@github.com wrote:

No, nothing has changed. J.

On 9/12/14, 3:59 AM, Mike Conway wrote:

iDrop desktop is also working. DId someone update the ACLs on the public directory or similar? I have successfully downloaded from /home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF using iDrop desktop and iDrop web.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55406907.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55439042.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55443640.

michael-conway commented 10 years ago

Hi John,

I was getting, in my testing, an error making a particular request to irods (getting an ObjStat, which is a summary set of data about a file).

I would get a DataNotFound exception, which can happen when there is an access problem due to security. I WAS getting that! You can page up in that log and see where it was occurring. In my tests today it DID NOT OCCUR. That is very odd. I'm sorry, this has been such a confusing issue. I'm working on getting a server up and running to have you test it out.

On Fri, Sep 12, 2014 at 2:48 PM, hellyj notifications@github.com wrote:

Most of the data are on a resource server that is federated with the primary. Sometimes that server hangs due to a firmware problem. Is it possible for iRODS to 'think' it is downloading something and then produce a zero-length file on the target if a resource server is down without throwing an error?

This could explain the inconsistent results.

J.

On 9/12/14, 8:39 AM, Mike Conway wrote:

Hi John,

I was able to complete downloads in idrop web and idrop desktop to that /home/public dir, so that is somewhat confusing! I'm trying to set up an alternate server for you to log into to sign into idrop web.

MC

On Fri, Sep 12, 2014 at 2:00 PM, hellyj notifications@github.com wrote:

No, nothing has changed. J.

On 9/12/14, 3:59 AM, Mike Conway wrote:

iDrop desktop is also working. DId someone update the ACLs on the public directory or similar? I have successfully downloaded from /home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF using iDrop desktop and iDrop web.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55406907.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55439042.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55443640.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55444698.

hellyj commented 10 years ago

Hi Mike.

I just re-tested and get the same results: iDrop produces only zero-length files while iget produces a proper copy.

Maybe we should schedule a shared desktop session for next week?

J.

On 9/12/14, 9:05 AM, Mike Conway wrote:

Hi John,

I was getting, in my testing, an error making a particular request to irods (getting an ObjStat, which is a summary set of data about a file).

I would get a DataNotFound exception, which can happen when there is an access problem due to security. I WAS getting that! You can page up in that log and see where it was occurring. In my tests today it DID NOT OCCUR. That is very odd. I'm sorry, this has been such a confusing issue. I'm working on getting a server up and running to have you test it out.

On Fri, Sep 12, 2014 at 2:48 PM, hellyj notifications@github.com wrote:

Most of the data are on a resource server that is federated with the primary. Sometimes that server hangs due to a firmware problem. Is it possible for iRODS to 'think' it is downloading something and then produce a zero-length file on the target if a resource server is down without throwing an error?

This could explain the inconsistent results.

J.

On 9/12/14, 8:39 AM, Mike Conway wrote:

Hi John,

I was able to complete downloads in idrop web and idrop desktop to that /home/public dir, so that is somewhat confusing! I'm trying to set up an alternate server for you to log into to sign into idrop web.

MC

On Fri, Sep 12, 2014 at 2:00 PM, hellyj notifications@github.com wrote:

No, nothing has changed. J.

On 9/12/14, 3:59 AM, Mike Conway wrote:

iDrop desktop is also working. DId someone update the ACLs on the public directory or similar? I have successfully downloaded from /home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF using iDrop desktop and iDrop web.

— Reply to this email directly or view it on GitHub

https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55406907.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55439042.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55443640.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55444698.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55446908.

michael-conway commented 10 years ago

yes, it makes no sense…

I can do Monday all day or Tuesday morning?

MC

On Sep 13, 2014, at 7:33 PM, hellyj notifications@github.com wrote:

Hi Mike.

I just re-tested and get the same results: iDrop produces only zero-length files while iget produces a proper copy.

Maybe we should schedule a shared desktop session for next week?

J.

On 9/12/14, 9:05 AM, Mike Conway wrote:

Hi John,

I was getting, in my testing, an error making a particular request to irods (getting an ObjStat, which is a summary set of data about a file).

I would get a DataNotFound exception, which can happen when there is an access problem due to security. I WAS getting that! You can page up in that log and see where it was occurring. In my tests today it DID NOT OCCUR. That is very odd. I'm sorry, this has been such a confusing issue. I'm working on getting a server up and running to have you test it out.

On Fri, Sep 12, 2014 at 2:48 PM, hellyj notifications@github.com wrote:

Most of the data are on a resource server that is federated with the primary. Sometimes that server hangs due to a firmware problem. Is it possible for iRODS to 'think' it is downloading something and then produce a zero-length file on the target if a resource server is down without throwing an error?

This could explain the inconsistent results.

J.

On 9/12/14, 8:39 AM, Mike Conway wrote:

Hi John,

I was able to complete downloads in idrop web and idrop desktop to that /home/public dir, so that is somewhat confusing! I'm trying to set up an alternate server for you to log into to sign into idrop web.

MC

On Fri, Sep 12, 2014 at 2:00 PM, hellyj notifications@github.com wrote:

No, nothing has changed. J.

On 9/12/14, 3:59 AM, Mike Conway wrote:

iDrop desktop is also working. DId someone update the ACLs on the public directory or similar? I have successfully downloaded from /home/public/GigaLES_010/CMMAP_SAM6.7SR_OUT3D_netCDF using iDrop desktop and iDrop web.

— Reply to this email directly or view it on GitHub

https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55406907.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55439042.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55443640.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55444698.

— Reply to this email directly or view it on GitHub https://github.com/DICE-UNC/idrop/issues/54#issuecomment-55446908.

— Reply to this email directly or view it on GitHub.

michael-conway commented 10 years ago

issue was default storage resource...resolved.