search

cyverse / gocommands

iRODS Command-line Tools written in Go
Other
29 stars 18 forks source link

put / sync not working (maybe its our instance) #34

Closed jjkoehorst closed 2 months ago

jjkoehorst commented 3 months ago

When trying to upload data to a (relatively) new irods instance I have the following issue

➜  snellius /scratch/jasperk/gocmd -c irods_environment.json sync ./meta i:/unlock/home/wur.gutbrain_axis/data/ --progress --clear --single_threaded --thread_num 1 --max_file_num 1
bundle 0 - Bundling                      ... done! [1 in 190ms]
bundle 0 - Cleaning & making dirs        ... done! [1 in 190ms]
bundle 1 - Cleaning & making dirs        ... done! [1 in 30ms]
bundle 1 - Bundling                      ... done! [1 in 20ms]
bundle 2 - Cleaning & making dirs        ... done! [1 in 20ms]
bundle 2 - Bundling                      ... done! [1 in 0s]
bundle 3 - Bundling                      ... done! [1 in 0s]
bundle 4 - Bundling                      ... done! [1 in 0s]
bundle 5 - Bundling                      ... done! [1 in 0s]
bundle 6 - Bundling                      ... done! [1 in 0s]
bundle 3 - Cleaning & making dirs        ... done! [1 in 30ms]
bundle 4 - Cleaning & making dirs        ... done! [1 in 30ms]
bundle 5 - Cleaning & making dirs        ... done! [1 in 20ms]
bundle 0 - Uploading                     ...  0.0% [..................] [0B in 100ms]
bundle 6 - Cleaning & making dirs        ...  0.0% [..................] [0 in 0s]
iRODS Error (code: '-1', message: '')

Error Trace:
  - failed to perform sync (from local to iRODS):
    github.com/cyverse/gocommands/cmd/subcmd.processSyncCommand
        /github/workspace/cmd/subcmd/sync.go:92
  - failed to perform bput ./meta to /unlock/home/wur.gutbrain_axis/data:
    github.com/cyverse/gocommands/cmd/subcmd.processBputCommand
        /github/workspace/cmd/subcmd/bput.go:175
  - failed to walk for /scratch/snellius/meta:
    github.com/cyverse/gocommands/cmd/subcmd.bputOne
        /github/workspace/cmd/subcmd/bput.go:241
  - failed to schedule /scratch/snellius/meta/DNA16-18314_94133-H11_ATCCAGGTATGCTGACGTTG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz:
    github.com/cyverse/gocommands/cmd/subcmd.bputOne.func1
        /github/workspace/cmd/subcmd/bput.go:234
  - failed to upload file /scratch/snellius/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz in bundle 0 to /unlock/home/wur.gutbrain_axis/data/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz:
    github.com/cyverse/gocommands/commons.(*BundleTransferManager).processBundleUpload
        /github/workspace/commons/bundle_transfer.go:1156
  - failed to open data object /unlock/home/wur.gutbrain_axis/data/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz:
    github.com/cyverse/go-irodsclient/irods/fs.UploadDataObject
        /go/pkg/mod/github.com/cyverse/go-irodsclient@v0.14.6/irods/fs/data_object_bulk.go:143
  - failed to open data object:
    github.com/cyverse/go-irodsclient/irods/fs.OpenDataObjectWithOperation
        /go/pkg/mod/github.com/cyverse/go-irodsclient@v0.14.6/irods/fs/data_object.go:1540

The issue might be on our side but iRODS Error (code: '-1', message: '') does not tell me much what goes wrong... any ideas?

jjkoehorst commented 3 months ago

/scratch/jasperk/gocmd -c irods_environment.json mkdir /unlock/home/wur.gutbrain_axis/bla works perfectly its just uploading files?

jjkoehorst commented 3 months ago

debug

➜  snellius /scratch/jasperk/gocmd -c irods_environment.json sync ./meta i:/unlock/home/wur.gutbrain_axis/data/ --progress --clear --single_threaded --thread_num 1 --max_file_num 1 --no_hash --no_bulk_reg --clear --debug
DEBU[2024-05-22 08:01:55.620] use sessionID - 847426                        function=ProcessCommonFlags package=flag
DEBU[2024-05-22 08:01:55.620] reading config file/dir - /scratch/snellius/irods_environment.json  function=LoadConfigFromFile package=commons
DEBU[2024-05-22 08:01:55.620] reading icommands environment file - /scratch/snellius/irods_environment.json  function=LoadConfigFromFile package=commons
DEBU[2024-05-22 08:01:55.620] reading environment file /scratch/snellius/irods_environment.json  function=Load package=icommands struct=ICommandsEnvironmentManager
DEBU[2024-05-22 08:01:55.621] reading environment session file /scratch/snellius/irods_environment.json.cwd  function=Load package=icommands struct=ICommandsEnvironmentManager
DEBU[2024-05-22 08:01:55.621] reading environment password file /home/jasperk/.irods/.irodsA  function=Load package=icommands struct=ICommandsEnvironmentManager
DEBU[2024-05-22 08:01:55.621] reading config from environment variables     function=LoadAndOverwriteConfigFromEnv package=commons
DEBU[2024-05-22 08:01:55.621] run bput with args: [-c irods_environment.json --diff ./meta i:/unlock/home/wur.gutbrain_axis/data/ --progress --clear --single_threaded --thread_num 1 --max_file_num 1 --no_hash --no_bulk_reg --clear --debug]  function=syncFromLocalToIRODS package=subcmd
DEBU[2024-05-22 08:01:55.621] use sessionID - 847426                        function=ProcessCommonFlags package=flag
DEBU[2024-05-22 08:01:55.621] reading config file/dir - /scratch/snellius/irods_environment.json  function=LoadConfigFromFile package=commons
DEBU[2024-05-22 08:01:55.621] reading icommands environment file - /scratch/snellius/irods_environment.json  function=LoadConfigFromFile package=commons
DEBU[2024-05-22 08:01:55.621] reading environment file /scratch/snellius/irods_environment.json  function=Load package=icommands struct=ICommandsEnvironmentManager
DEBU[2024-05-22 08:01:55.621] reading environment session file /scratch/snellius/irods_environment.json.cwd  function=Load package=icommands struct=ICommandsEnvironmentManager
DEBU[2024-05-22 08:01:55.621] reading environment password file /home/jasperk/.irods/.irodsA  function=Load package=icommands struct=ICommandsEnvironmentManager
DEBU[2024-05-22 08:01:55.621] reading config from environment variables     function=LoadAndOverwriteConfigFromEnv package=commons
DEBU[2024-05-22 08:01:55.621] clearing local bundle files in /tmp           function=CleanUpOldLocalBundles package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.657] Connecting to scomp1789.wurnet.nl:1247        function=Connect package=connection struct=IRODSConnection
DEBU[2024-05-22 08:01:55.660] Start up a connection with CS Negotiation     function=connectWithCSNegotiation package=connection struct=IRODSConnection
DEBU[2024-05-22 08:01:55.813] Start up CS Negotiation                       function=connectWithCSNegotiation package=connection struct=IRODSConnection
DEBU[2024-05-22 08:01:55.814] Client policy - CS_NEG_REQUIRE, server policy - CS_NEG_REQUIRE  function=connectWithCSNegotiation package=connection struct=IRODSConnection
DEBU[2024-05-22 08:01:55.814] Start up SSL                                  function=sslStartup package=connection struct=IRODSConnection
DEBU[2024-05-22 08:01:55.850] Logging in using pam authentication method    function=loginPAMWithToken package=connection struct=IRODSConnection
DEBU[2024-05-22 08:01:55.859] Created a new connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:55.868] failed to commit transaction                  error=SYS_NO_API_PRIV function=endTransaction package=session struct=IRODSSession
DEBU[2024-05-22 08:01:55.870] Returning a connection                        function=Return package=session struct=ConnectionPool
INFO[2024-05-22 08:01:55.870] determining staging dir...                    function=processBputCommand package=subcmd
DEBU[2024-05-22 08:01:55.870] get default staging dir                       function=processBputCommand package=subcmd
INFO[2024-05-22 08:01:55.870] use staging dir - /unlock/home/wur.gutbrain_axis/data/.gocmd_staging  function=processBputCommand package=subcmd
DEBU[2024-05-22 08:01:55.870] clearing irods temp dir /unlock/home/wur.gutbrain_axis/data/.gocmd_staging  function=processBputCommand package=subcmd
DEBU[2024-05-22 08:01:55.871] clearing old irods bundle files in /unlock/home/wur.gutbrain_axis/data/.gocmd_staging  function=CleanUpOldIRODSBundles package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.871] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:55.876] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:55.876] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:55.888] Returning a connection                        function=Return package=session struct=ConnectionPool
deleted 0 old irods bundles in /unlock/home/wur.gutbrain_axis/data/.gocmd_staging
DEBU[2024-05-22 08:01:55.888] deleted 0 old irods bundles in /unlock/home/wur.gutbrain_axis/data/.gocmd_staging  function=CleanUpOldIRODSBundles package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.888] bundle-uploading a local directory /scratch/snellius/meta  function=bputOne package=subcmd
DEBU[2024-05-22 08:01:55.888] assigned a new bundle 0                       function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.888] checking if target /unlock/home/wur.gutbrain_axis/data/meta for source /scratch/snellius/meta exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.888] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:55.888] start input thread                            function=Start package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.888] start extract thread 1                        function=Start package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.888] start extract thread 0                        function=Start package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.888] start extract thread 2                        function=Start package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.888] start transfer thread 0                       function=Start package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.888] start stale file remove and dir create thread  function=Start package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.888] start bundle thread                           function=Start package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.894] Returning a connection                        function=Return package=session struct=ConnectionPool
skip adding a dir /scratch/snellius/meta to the bundle. The dir already exists!
DEBU[2024-05-22 08:01:55.894] skip adding a dir /scratch/snellius/meta to the bundle. The dir already exists!  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.897] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.897] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:55.904] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:55.904] adding a file /scratch/snellius/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.904] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.904] assigned a new bundle 1                       function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.904] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R2_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R2_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.904] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:55.910] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:55.910] adding a file /scratch/snellius/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R2_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.910] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R2_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.910] creating a tarball for bundle 0 to /tmp/bundle_e7e6f3c3e6179c20b9bc8b15337b1558.tar  function=processBundleTar package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.910] deleting exising data objects and creating new collections in the bundle 0  function=processBundleRemoveFilesAndMakeDirs package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.910] assigned a new bundle 2                       function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.910] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-14697_94133-G07_TTGTATCAGGTGGCCTCTGT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-14697_94133-G07_TTGTATCAGGTGGCCTCTGT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.910] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:55.915] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:55.915] adding a file /scratch/snellius/meta/DNA16-14697_94133-G07_TTGTATCAGGTGGCCTCTGT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.915] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-14697_94133-G07_TTGTATCAGGTGGCCTCTGT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.915] assigned a new bundle 3                       function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.915] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-14697_94133-G07_TTGTATCAGGTGGCCTCTGT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-14697_94133-G07_TTGTATCAGGTGGCCTCTGT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.915] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:55.921] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:55.921] adding a file /scratch/snellius/meta/DNA16-14697_94133-G07_TTGTATCAGGTGGCCTCTGT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.921] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-14697_94133-G07_TTGTATCAGGTGGCCTCTGT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.921] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:55.922] assigned a new bundle 4                       function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.922] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-14708_94133-G08_TCCGGACTAGATTGTTCGTC_L001_R1_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-14708_94133-G08_TCCGGACTAGATTGTTCGTC_L001_R1_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:55.922] Connecting to scomp1789.wurnet.nl:1247        function=Connect package=connection struct=IRODSConnection
DEBU[2024-05-22 08:01:55.925] Start up a connection with CS Negotiation     function=connectWithCSNegotiation package=connection struct=IRODSConnection
bundle 0 - Bundling                      ...  0.0% [..................] [0 in 200ms]
bundle 0 - Cleaning & making dirs        ...  0.0% [..................] [0 in 200ms]
DEBU[2024-05-22 08:01:56.108] Start up CS Negotiation                       function=connectWithCSNegotiation package=connection struct=IRODSConnection
DEBU[2024-05-22 08:01:56.108] Client policy - CS_NEG_REQUIRE, server policy - CS_NEG_REQUIRE  function=connectWithCSNegotiation package=connection struct=IRODSConnection
DEBU[2024-05-22 08:01:56.109] Start up SSL                                  function=sslStartup package=connection struct=IRODSConnection
DEBU[2024-05-22 08:01:56.118] Logging in using pam authentication method    function=loginPAMWithToken package=connection struct=IRODSConnection
DEBU[2024-05-22 08:01:56.128] Created a new connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.132] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.132] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.142] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.142] adding a file /scratch/snellius/meta/DNA16-14708_94133-G08_TCCGGACTAGATTGTTCGTC_L001_R1_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.142] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-14708_94133-G08_TCCGGACTAGATTGTTCGTC_L001_R1_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.142] assigned a new bundle 5                       function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.142] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-14708_94133-G08_TCCGGACTAGATTGTTCGTC_L001_R2_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-14708_94133-G08_TCCGGACTAGATTGTTCGTC_L001_R2_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.146] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.146] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.152] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.152] adding a file /scratch/snellius/meta/DNA16-14708_94133-G08_TCCGGACTAGATTGTTCGTC_L001_R2_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.153] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-14708_94133-G08_TCCGGACTAGATTGTTCGTC_L001_R2_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.153] deleted exising data objects in the bundle 0  function=processBundleRemoveFilesAndMakeDirs package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.153] deleting exising data objects and creating new collections in the bundle 1  function=processBundleRemoveFilesAndMakeDirs package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.153] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.153] assigned a new bundle 6                       function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.153] skip - creating a tarball for bundle 0 to /tmp/bundle_e7e6f3c3e6179c20b9bc8b15337b1558.tar  function=processBundleTar package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.153] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-15071_94133-G09_AGCCTATGATGTAGGTGGTG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-15071_94133-G09_AGCCTATGATGTAGGTGGTG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.153] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.159] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.159] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.163] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.163] adding a file /scratch/snellius/meta/DNA16-15071_94133-G09_AGCCTATGATGTAGGTGGTG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.163] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-15071_94133-G09_AGCCTATGATGTAGGTGGTG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.163] uploading bundle 0 to /unlock/home/wur.gutbrain_axis/data/.gocmd_staging/bundle_e7e6f3c3e6179c20b9bc8b15337b1558.tar  function=processBundleUpload package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.163] creating a tarball for bundle 1 to /tmp/bundle_fe8c3d6a85dd8c6091e1d7679699e285.tar  function=processBundleTar package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.163] assigned a new bundle 7                       function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.163] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-15071_94133-G09_AGCCTATGATGTAGGTGGTG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-15071_94133-G09_AGCCTATGATGTAGGTGGTG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.165] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.165] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.172] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.172] adding a file /scratch/snellius/meta/DNA16-15071_94133-G09_AGCCTATGATGTAGGTGGTG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.172] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-15071_94133-G09_AGCCTATGATGTAGGTGGTG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.172] assigned a new bundle 8                       function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.172] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-15099_94133-G10_GGACCAGTGGTAGGCTCGCG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-15099_94133-G10_GGACCAGTGGTAGGCTCGCG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.172] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.178] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.178] adding a file /scratch/snellius/meta/DNA16-15099_94133-G10_GGACCAGTGGTAGGCTCGCG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.178] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-15099_94133-G10_GGACCAGTGGTAGGCTCGCG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.178] deleted exising data objects in the bundle 1  function=processBundleRemoveFilesAndMakeDirs package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.178] deleting exising data objects and creating new collections in the bundle 2  function=processBundleRemoveFilesAndMakeDirs package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.178] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.178] skip - creating a tarball for bundle 1 to /tmp/bundle_fe8c3d6a85dd8c6091e1d7679699e285.tar  function=processBundleTar package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.178] creating a tarball for bundle 2 to /tmp/bundle_d5d638eec06dea875e713dfe72ba0944.tar  function=processBundleTar package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.178] assigned a new bundle 9                       function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.178] upload data object /scratch/snellius/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz  function=UploadDataObject package=fs
DEBU[2024-05-22 08:01:56.178] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-15099_94133-G10_GGACCAGTGGTAGGCTCGCG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-15099_94133-G10_GGACCAGTGGTAGGCTCGCG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.178] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.178] Connecting to scomp1789.wurnet.nl:1247        function=Connect package=connection struct=IRODSConnection
DEBU[2024-05-22 08:01:56.182] Start up a connection with CS Negotiation     function=connectWithCSNegotiation package=connection struct=IRODSConnection
bundle 0 - Bundling                      ... done! [1 in 230ms]
bundle 0 - Cleaning & making dirs        ... done! [1 in 230ms]
bundle 1 - Cleaning & making dirs        ... done! [1 in 30ms]
bundle 1 - Bundling                      ... done! [1 in 0s]
bundle 0 - Uploading                     ...  0.0% [..................] [0B in 0s]
bundle 2 - Cleaning & making dirs        ...  0.0% [..................] [0 in 0s]
DEBU[2024-05-22 08:01:56.190] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.190] adding a file /scratch/snellius/meta/DNA16-15099_94133-G10_GGACCAGTGGTAGGCTCGCG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.190] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-15099_94133-G10_GGACCAGTGGTAGGCTCGCG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.190] assigned a new bundle 10                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.191] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-15783_94134-B12_AGAATTCGCCGAGCTTGCCG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-15783_94134-B12_AGAATTCGCCGAGCTTGCCG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.193] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.193] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.200] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.200] adding a file /scratch/snellius/meta/DNA16-15783_94134-B12_AGAATTCGCCGAGCTTGCCG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.200] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-15783_94134-B12_AGAATTCGCCGAGCTTGCCG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.200] deleted exising data objects in the bundle 2  function=processBundleRemoveFilesAndMakeDirs package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.200] deleting exising data objects and creating new collections in the bundle 3  function=processBundleRemoveFilesAndMakeDirs package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.200] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.200] assigned a new bundle 11                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.200] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-15783_94134-B12_AGAATTCGCCGAGCTTGCCG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-15783_94134-B12_AGAATTCGCCGAGCTTGCCG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.200] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.207] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.207] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.211] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.211] adding a file /scratch/snellius/meta/DNA16-15783_94134-B12_AGAATTCGCCGAGCTTGCCG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.211] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-15783_94134-B12_AGAATTCGCCGAGCTTGCCG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.211] assigned a new bundle 12                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.211] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-16025_94133-G11_TTCCTACAGCAGAATTCGCC_L001_R1_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-16025_94133-G11_TTCCTACAGCAGAATTCGCC_L001_R1_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.213] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.214] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.219] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.219] adding a file /scratch/snellius/meta/DNA16-16025_94133-G11_TTCCTACAGCAGAATTCGCC_L001_R1_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.219] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-16025_94133-G11_TTCCTACAGCAGAATTCGCC_L001_R1_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.219] deleted exising data objects in the bundle 3  function=processBundleRemoveFilesAndMakeDirs package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.219] assigned a new bundle 13                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.219] skip - creating a tarball for bundle 2 to /tmp/bundle_d5d638eec06dea875e713dfe72ba0944.tar  function=processBundleTar package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.219] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-16025_94133-G11_TTCCTACAGCAGAATTCGCC_L001_R2_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-16025_94133-G11_TTCCTACAGCAGAATTCGCC_L001_R2_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.219] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.225] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.225] adding a file /scratch/snellius/meta/DNA16-16025_94133-G11_TTCCTACAGCAGAATTCGCC_L001_R2_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.225] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-16025_94133-G11_TTCCTACAGCAGAATTCGCC_L001_R2_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.225] deleting exising data objects and creating new collections in the bundle 4  function=processBundleRemoveFilesAndMakeDirs package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.225] creating a tarball for bundle 3 to /tmp/bundle_5f7e41cb519bde14e7c18aca9e6eb8e0.tar  function=processBundleTar package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.225] assigned a new bundle 14                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.225] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-16027_94134-A01_TCTCATGATATCCACGTGTT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-16027_94134-A01_TCTCATGATATCCACGTGTT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.225] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.231] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.231] adding a file /scratch/snellius/meta/DNA16-16027_94134-A01_TCTCATGATATCCACGTGTT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.231] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-16027_94134-A01_TCTCATGATATCCACGTGTT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.231] assigned a new bundle 15                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.231] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-16027_94134-A01_TCTCATGATATCCACGTGTT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-16027_94134-A01_TCTCATGATATCCACGTGTT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.231] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.238] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.238] adding a file /scratch/snellius/meta/DNA16-16027_94134-A01_TCTCATGATATCCACGTGTT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.238] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-16027_94134-A01_TCTCATGATATCCACGTGTT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.238] skip - creating a tarball for bundle 3 to /tmp/bundle_5f7e41cb519bde14e7c18aca9e6eb8e0.tar  function=processBundleTar package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.238] creating a tarball for bundle 4 to /tmp/bundle_688c0385d97dcdd589771394876fb86b.tar  function=processBundleTar package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.238] skip - creating a tarball for bundle 4 to /tmp/bundle_688c0385d97dcdd589771394876fb86b.tar  function=processBundleTar package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.238] creating a tarball for bundle 5 to /tmp/bundle_0946bb3b94dcc0c47822d8cf2b0fb4ca.tar  function=processBundleTar package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.238] skip - creating a tarball for bundle 5 to /tmp/bundle_0946bb3b94dcc0c47822d8cf2b0fb4ca.tar  function=processBundleTar package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.238] assigned a new bundle 16                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.238] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-16028_94133-H01_GGTCTATTAACGTGTCGTGG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-16028_94133-H01_GGTCTATTAACGTGTCGTGG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.238] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.238] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.244] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.244] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.247] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.247] adding a file /scratch/snellius/meta/DNA16-16028_94133-H01_GGTCTATTAACGTGTCGTGG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.247] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-16028_94133-H01_GGTCTATTAACGTGTCGTGG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.247] creating a tarball for bundle 6 to /tmp/bundle_3c4358053f13e4268cd370379c2ac8c5.tar  function=processBundleTar package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.247] assigned a new bundle 17                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.247] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-16028_94133-H01_GGTCTATTAACGTGTCGTGG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-16028_94133-H01_GGTCTATTAACGTGTCGTGG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.250] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.250] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.258] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.258] adding a file /scratch/snellius/meta/DNA16-16028_94133-H01_GGTCTATTAACGTGTCGTGG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.258] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-16028_94133-H01_GGTCTATTAACGTGTCGTGG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.258] assigned a new bundle 18                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.258] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-16029_94133-H02_TGATGGCTACATGTTGTTGG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-16029_94133-H02_TGATGGCTACATGTTGTTGG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.258] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.264] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.264] adding a file /scratch/snellius/meta/DNA16-16029_94133-H02_TGATGGCTACATGTTGTTGG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.264] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-16029_94133-H02_TGATGGCTACATGTTGTTGG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.264] deleted exising data objects in the bundle 4  function=processBundleRemoveFilesAndMakeDirs package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.264] deleting exising data objects and creating new collections in the bundle 5  function=processBundleRemoveFilesAndMakeDirs package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.264] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.264] assigned a new bundle 19                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.264] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-16029_94133-H02_TGATGGCTACATGTTGTTGG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-16029_94133-H02_TGATGGCTACATGTTGTTGG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.264] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.271] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.271] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.275] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.275] adding a file /scratch/snellius/meta/DNA16-16029_94133-H02_TGATGGCTACATGTTGTTGG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.275] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-16029_94133-H02_TGATGGCTACATGTTGTTGG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.275] assigned a new bundle 20                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.276] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-16030_94133-H03_AGGACAGGCCAGCTGGAATG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-16030_94133-H03_AGGACAGGCCAGCTGGAATG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.278] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.278] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.284] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.284] adding a file /scratch/snellius/meta/DNA16-16030_94133-H03_AGGACAGGCCAGCTGGAATG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.284] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-16030_94133-H03_AGGACAGGCCAGCTGGAATG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.284] deleted exising data objects in the bundle 5  function=processBundleRemoveFilesAndMakeDirs package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.284] deleting exising data objects and creating new collections in the bundle 6  function=processBundleRemoveFilesAndMakeDirs package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.284] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.284] assigned a new bundle 21                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.284] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-16030_94133-H03_AGGACAGGCCAGCTGGAATG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-16030_94133-H03_AGGACAGGCCAGCTGGAATG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.284] skip - creating a tarball for bundle 6 to /tmp/bundle_3c4358053f13e4268cd370379c2ac8c5.tar  function=processBundleTar package=commonbundle 2 - Cleaning & making dirs        ... done! [1 in 20ms]
bundle 2 - Bundling                      ... done! [1 in 20ms]
bundle 3 - Cleaning & making dirs        ... done! [1 in 20ms]
bundle 4 - Cleaning & making dirs        ... done! [1 in 30ms]
bundle 3 - Bundling                      ... done! [1 in 0s]
bundle 4 - Bundling                      ... done! [1 in 0s]
bundle 5 - Bundling                      ... done! [1 in 0s]
bundle 6 - Bundling                      ... done! [1 in 20ms]
bundle 5 - Cleaning & making dirs        ... done! [1 in 20ms]
bundle 0 - Uploading                     ...  0.0% [..................] [0B in 100ms]
bundle 6 - Cleaning & making dirs        ...  0.0% [..................] [0 in 0s]
DEBU[2024-05-22 08:01:56.289] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.289] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.296] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.296] adding a file /scratch/snellius/meta/DNA16-16030_94133-H03_AGGACAGGCCAGCTGGAATG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.296] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-16030_94133-H03_AGGACAGGCCAGCTGGAATG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.296] assigned a new bundle 22                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.296] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-16648_94133-H04_CTAATAACCGCGATGCGGTT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-16648_94133-H04_CTAATAACCGCGATGCGGTT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.298] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.298] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.305] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.305] adding a file /scratch/snellius/meta/DNA16-16648_94133-H04_CTAATAACCGCGATGCGGTT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.305] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-16648_94133-H04_CTAATAACCGCGATGCGGTT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.305] assigned a new bundle 23                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.305] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-16648_94133-H04_CTAATAACCGCGATGCGGTT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-16648_94133-H04_CTAATAACCGCGATGCGGTT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.305] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.312] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.312] adding a file /scratch/snellius/meta/DNA16-16648_94133-H04_CTAATAACCGCGATGCGGTT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.312] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-16648_94133-H04_CTAATAACCGCGATGCGGTT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.312] deleted exising data objects in the bundle 6  function=processBundleRemoveFilesAndMakeDirs package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.312] deleting exising data objects and creating new collections in the bundle 7  function=processBundleRemoveFilesAndMakeDirs package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.312] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.312] assigned a new bundle 24                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.312] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-16735_94133-H05_CCAGGCACCATAATTAGCGT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-16735_94133-H05_CCAGGCACCATAATTAGCGT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.312] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.317] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.317] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.321] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.321] adding a file /scratch/snellius/meta/DNA16-16735_94133-H05_CCAGGCACCATAATTAGCGT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.321] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-16735_94133-H05_CCAGGCACCATAATTAGCGT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.321] assigned a new bundle 25                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.321] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-16735_94133-H05_CCAGGCACCATAATTAGCGT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-16735_94133-H05_CCAGGCACCATAATTAGCGT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.324] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.324] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.331] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.331] adding a file /scratch/snellius/meta/DNA16-16735_94133-H05_CCAGGCACCATAATTAGCGT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.331] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-16735_94133-H05_CCAGGCACCATAATTAGCGT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.331] assigned a new bundle 26                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.331] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-16986_94133-H06_AATCCAATTGCCTCTACATG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-16986_94133-H06_AATCCAATTGCCTCTACATG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.331] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.337] Start up CS Negotiation                       function=connectWithCSNegotiation package=connection struct=IRODSConnection
DEBU[2024-05-22 08:01:56.337] Client policy - CS_NEG_REQUIRE, server policy - CS_NEG_REQUIRE  function=connectWithCSNegotiation package=connection struct=IRODSConnection
DEBU[2024-05-22 08:01:56.338] Start up SSL                                  function=sslStartup package=connection struct=IRODSConnection
DEBU[2024-05-22 08:01:56.338] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.338] adding a file /scratch/snellius/meta/DNA16-16986_94133-H06_AATCCAATTGCCTCTACATG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.338] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-16986_94133-H06_AATCCAATTGCCTCTACATG_L001_R1_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.338] deleted exising data objects in the bundle 7  function=processBundleRemoveFilesAndMakeDirs package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.338] assigned a new bundle 27                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.338] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-16986_94133-H06_AATCCAATTGCCTCTACATG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-16986_94133-H06_AATCCAATTGCCTCTACATG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.338] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.344] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.344] adding a file /scratch/snellius/meta/DNA16-16986_94133-H06_AATCCAATTGCCTCTACATG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.344] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-16986_94133-H06_AATCCAATTGCCTCTACATG_L001_R2_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.345] assigned a new bundle 28                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.345] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-17474_94133-H07_CACAGCGGTCCAATAGGAAT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-17474_94133-H07_CACAGCGGTCCAATAGGAAT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.345] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.347] Logging in using pam authentication method    function=loginPAMWithToken package=connection struct=IRODSConnection
DEBU[2024-05-22 08:01:56.350] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.350] adding a file /scratch/snellius/meta/DNA16-17474_94133-H07_CACAGCGGTCCAATAGGAAT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.350] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-17474_94133-H07_CACAGCGGTCCAATAGGAAT_L001_R1_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.350] assigned a new bundle 29                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.350] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-17474_94133-H07_CACAGCGGTCCAATAGGAAT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-17474_94133-H07_CACAGCGGTCCAATAGGAAT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.350] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.354] Created a new connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.356] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.356] adding a file /scratch/snellius/meta/DNA16-17474_94133-H07_CACAGCGGTCCAATAGGAAT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.356] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-17474_94133-H07_CACAGCGGTCCAATAGGAAT_L001_R2_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.356] assigned a new bundle 30                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.356] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-17584_94133-H08_GTGACGGAGCTGGACCGCCA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-17584_94133-H08_GTGACGGAGCTGGACCGCCA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.356] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.362] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.363] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.363] adding a file /scratch/snellius/meta/DNA16-17584_94133-H08_GTGACGGAGCTGGACCGCCA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.363] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-17584_94133-H08_GTGACGGAGCTGGACCGCCA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.363] assigned a new bundle 31                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.363] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-17584_94133-H08_GTGACGGAGCTGGACCGCCA_L001_R2_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-17584_94133-H08_GTGACGGAGCTGGACCGCCA_L001_R2_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.363] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.368] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.369] adding a file /scratch/snellius/meta/DNA16-17584_94133-H08_GTGACGGAGCTGGACCGCCA_L001_R2_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.369] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-17584_94133-H08_GTGACGGAGCTGGACCGCCA_L001_R2_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.369] assigned a new bundle 32                      function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.369] checking if target /unlock/home/wur.gutbrain_axis/data/meta/DNA16-18030_94133-H09_TATGACAATCATACAAGCAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz for source /scratch/snellius/meta/DNA16-18030_94133-H09_TATGACAATCATACAAGCAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz exists  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.369] Reuse an idle connection                      function=Get package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.376] Returning a connection                        function=Return package=session struct=ConnectionPool
DEBU[2024-05-22 08:01:56.376] adding a file /scratch/snellius/meta/DNA16-18030_94133-H09_TATGACAATCATACAAGCAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz to the bundle as it doesn't exist  function=Schedule package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.376] > scheduled a local file bundle-upload /scratch/snellius/meta/DNA16-18030_94133-H09_TATGACAATCATACAAGCAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz  function=Schedule package=commons struct=BundleTransferManager
ERRO[2024-05-22 08:01:56.376] failed to upload file /scratch/snellius/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz in bundle 0 to /unlock/home/wur.gutbrain_axis/data/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz: failed to open data object /unlock/home/wur.gutbrain_axis/data/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz: failed to open data object:   function=Start package=commons struct=BundleTransferManager
DEBU[2024-05-22 08:01:56.377] Disconnecting the connection                  function=Disconnect package=connection struct=IRODSConnection
DEBU[2024-05-22 08:01:56.377] Disconnecting the connection                  function=Disconnect package=connection struct=IRODSConnection
DEBU[2024-05-22 08:01:56.377] Disconnecting the connection                  function=Disconnect package=connection struct=IRODSConnection
ERRO[2024-05-22 08:01:56.377] failed to perform sync (from local to iRODS):
    github.com/cyverse/gocommands/cmd/subcmd.processSyncCommand
        /github/workspace/cmd/subcmd/sync.go:92
  - failed to perform bput ./meta to /unlock/home/wur.gutbrain_axis/data:
    github.com/cyverse/gocommands/cmd/subcmd.processBputCommand
        /github/workspace/cmd/subcmd/bput.go:175
  - failed to walk for /scratch/snellius/meta:
    github.com/cyverse/gocommands/cmd/subcmd.bputOne
        /github/workspace/cmd/subcmd/bput.go:241
  - failed to schedule /scratch/snellius/meta/DNA16-18030_94133-H09_TATGACAATCATACAAGCAA_L001_R2_001_BH3MLWDSX5.filt.fastq.gz:
    github.com/cyverse/gocommands/cmd/subcmd.bputOne.func1
        /github/workspace/cmd/subcmd/bput.go:234
  - failed to upload file /scratch/snellius/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz in bundle 0 to /unlock/home/wur.gutbrain_axis/data/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz:
    github.com/cyverse/gocommands/commons.(*BundleTransferManager).processBundleUpload
        /github/workspace/commons/bundle_transfer.go:1156
  - failed to open data object /unlock/home/wur.gutbrain_axis/data/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz:
    github.com/cyverse/go-irodsclient/irods/fs.UploadDataObject
        /go/pkg/mod/github.com/cyverse/go-irodsclient@v0.14.6/irods/fs/data_object_bulk.go:143
  - failed to open data object:
    github.com/cyverse/go-irodsclient/irods/fs.OpenDataObjectWithOperation
        /go/pkg/mod/github.com/cyverse/go-irodsclient@v0.14.6/irods/fs/data_object.go:1540
  -   function=main package=main
iRODS Error (code: '-1', message: '')

Error Trace:
  - failed to perform sync (from local to iRODS):
    github.com/cyverse/gocommands/cmd/subcmd.processSyncCommand
        /github/workspace/cmd/subcmd/sync.go:92
  - failed to perform bput ./meta to /unlock/home/wur.gutbrain_axis/data:
    github.com/cyverse/gocommands/cmd/subcmd.processBputCommand
        /github/workspace/cmd/subcmd/bput.go:175
  - failed to walk for /scratch/snellius/meta:
    github.com/cyverse/gocommands/cmd/subcmd.bputOne
        /github/workspace/cmd/subcmd/bput.go:241
  - failed to schedule /scratch/snellius/meta/DNA16-18030_94133-H09_TATGACAATCATACAAGCAA_L001_R2_001_BH3MLWDSX5.filt.fastq.gz:
    github.com/cyverse/gocommands/cmd/subcmd.bputOne.func1
        /github/workspace/cmd/subcmd/bput.go:234
  - failed to upload file /scratch/snellius/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz in bundle 0 to /unlock/home/wur.gutbrain_axis/data/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz:
    github.com/cyverse/gocommands/commons.(*BundleTransferManager).processBundleUpload
        /github/workspace/commons/bundle_transfer.go:1156
  - failed to open data object /unlock/home/wur.gutbrain_axis/data/meta/DNA16-14185_94133-G06_GTTGGATGAATTACCTGGAA_L001_R1_001_BH3MLWDSX5.filt.fastq.gz:
    github.com/cyverse/go-irodsclient/irods/fs.UploadDataObject
        /go/pkg/mod/github.com/cyverse/go-irodsclient@v0.14.6/irods/fs/data_object_bulk.go:143
  - failed to open data object:
    github.com/cyverse/go-irodsclient/irods/fs.OpenDataObjectWithOperation
        /go/pkg/mod/github.com/cyverse/go-irodsclient@v0.14.6/irods/fs/data_object.go:1540
  - 
➜  snellius
jjkoehorst commented 3 months ago

Ah, I think I got it...

Level 0: #### !!RDM WARNING!! pep_api_data_obj_put_pre: You cannot upload the file without calculating and verifying the checksum. You need to upload with -K (not -k) to guarantee integrity on your filetransfer to irods

This is from iCommands, should have checked that first... Is this option available in gocmd?

iychoi commented 3 months ago

There's no such flag supported in GoCommands yet. I'll figure out what that is and add it in GoCommands soon.

iychoi commented 3 months ago

FYI, working on it now. But it will take few weeks to finish it due to other tasks.

jjkoehorst commented 2 months ago

@iychoi, If in one way or another I can help please let me know :)

iychoi commented 2 months ago

@jjkoehorst Thanks. Actually the latest commit includes -k and -K flags. I just need more time to test before releasing. If you can build from code, can you try it?

jjkoehorst commented 2 months ago

Looks to be working!

bundle 0 - Bundling                      ... done! [458.00K in 420ms]
bundle 0 - Cleaning & making dirs        ... done! [50 in 480ms]
bundle 1 - Cleaning & making dirs        ... done! [50 in 490ms]
bundle 2 - Cleaning & making dirs        ... done! [50 in 230ms]
bundle 1 - Bundling                      ... done! [66.87M in 560ms]
bundle 0 - Uploading                     ... done! [1.75GB in 15.14s]
bundle 2 - Bundling                      ... done! [76.96M in 150ms]
bundle 3 - Cleaning & making dirs        ... done! [50 in 230ms]
bundle 0 - Extracting                    ... done! [50 in 2.96s]
bundle 3 - Bundling                      ... done! [38.56M in 380ms]
bundle 4 - Cleaning & making dirs        ... done! [50 in 1.16s]
bundle 4 - Bundling                      ... done! [42.73M in 90ms]
bundle 5 - Cleaning & making dirs        ... done! [50 in 880ms]
bundle 5 - Bundling                      ... done! [39.81M in 100ms]
bundle 6 - Cleaning & making dirs        ... done! [50 in 840ms]
bundle 6 - Bundling                      ... done! [41.02M in 90ms]
bundle 7 - Cleaning & making dirs        ... done! [50 in 890ms]
bundle 7 - Bundling                      ... done! [37.42M in 80ms]
bundle 8 - Cleaning & making dirs        ... done! [50 in 760ms]
bundle 8 - Bundling                      ... done! [40.49M in 90ms]
bundle 9 - Cleaning & making dirs        ... done! [50 in 780ms]
bundle 9 - Bundling                      ... done! [72.96M in 140ms]
bundle 10 - Cleaning & making dirs       ... done! [50 in 810ms]
bundle 10 - Bundling                     ... done! [28.12M in 60ms]
bundle 11 - Cleaning & making dirs       ... done! [50 in 880ms]
bundle 11 - Bundling                     ... done! [91.83M in 170ms]
bundle 12 - Cleaning & making dirs       ... done! [50 in 830ms]
bundle 1 - Uploading                     ... 95.2% [#################.] [1.76GB in 1m9.2s; ~ETA: 4s]
bundle 2 - Uploading                     ... 75.7% [#############.....] [1.52GB in 1m5.5s; ~ETA: 22s]
bundle 3 - Uploading                     ... 96.4% [#################.] [1.74GB in 1m2.4s; ~ETA: 3s]
bundle 4 - Uploading                     ... 81.8% [##############....] [1.58GB in 59.2s; ~ETA: 14s]
bundle 5 - Uploading                     ... 73.1% [#############.....] [1.49GB in 55.9s; ~ETA: 21
iychoi commented 2 months ago

Thanks for testing! I'll soon make a new release!