Problems with data transfer on Windows

[jnimoth]

For the windows version, data transfer via the put, get and sync commands do not seem to work properly.

Example debug for put local Windows to iRODS:

X:\My Downloads> .\gocmd-v0.4.3-windows-amd64\gocmd.exe -c windows_config.yaml put --progress -d .\RO.zip
time="2023-01-30 16:03:24.030" level=debug msg="use sessionID - 19868" function=ProcessCommonFlags package=commons
time="2023-01-30 16:03:24.030" level=debug msg="reading config file/dir - X:\\My Downloads\\windows_config.yaml" function=loadConfigFile package=commons
time="2023-01-30 16:03:24.034" level=debug msg="reading gocommands YAML config file - X:\\My Downloads\\windows_config.yaml" function=loadConfigFile package=commons
time="2023-01-30 16:03:28.234" level=error msg="error while uploading a local file X:\\My Downloads\\RO.zip to /testZone/home/username/RO.zip" error="CreateFile /X:\\My Downloads\\RO.zip: The specified path is invalid." function=ScheduleUpload package=commons struct=ParallelTransferManager
time="2023-01-30 16:03:28.236" level=error msg="CreateFile /X:\\My Downloads\\RO.zip: The specified path is invalid." function=processPutCommand package=main
CreateFile /X:\My Downloads\RO.zip: The specified path is invalid.

Example debug for get from iRODS to local Windows:

X:\My Downloads> .\gocmd-v0.4.3-windows-amd64\gocmd.exe -c windows_config.yaml get --progress -d test.tar
time="2023-01-30 16:11:24.545" level=debug msg="use sessionID - 19868" function=ProcessCommonFlags package=commons
time="2023-01-30 16:11:24.545" level=debug msg="reading config file/dir - X:\\My Downloads\\windows_config.yaml" function=loadConfigFile package=commons
time="2023-01-30 16:11:24.548" level=debug msg="reading gocommands YAML config file - X:\\My Downloads\\windows_config.yaml" function=loadConfigFile package=commons
time="2023-01-30 16:11:28.803" level=error msg="error while downloading a data object /testZone/home/username/test.tar to X:\\My Downloads\\test.tar" error="CreateFile /X:\\My Downloads\\test.tar: The specified path is invalid." function=ScheduleDownload package=commons struct=ParallelTransferManager
time="2023-01-30 16:11:28.805" level=error msg="CreateFile /X:\\My Downloads\\test.tar: The specified path is invalid." function=processGetCommand package=main
CreateFile /X:\My Downloads\test.tar: The specified path is invalid.

Example debug for sync between a local directory and an iRODS collection:

X:\My Downloads> .\gocmd-v0.4.3-windows-amd64\gocmd.exe -c windows_config.yaml sync -d --progress --no_replication example_folder i:/testZone/home/username/gocommand_sync
time="2023-01-30 16:27:19.189" level=debug msg="use sessionID - 19868" function=ProcessCommonFlags package=commons
time="2023-01-30 16:27:19.189" level=debug msg="reading config file/dir - X:\\My Downloads\\windows_config.yaml" function=loadConfigFile package=commons
time="2023-01-30 16:27:19.219" level=debug msg="reading gocommands YAML config file - X:\\My Downloads\\windows_config.yaml" function=loadConfigFile package=commons
time="2023-01-30 16:27:24.220" level=error msg="CreateFile X:\\My Downloads\\X:\\My Downloads\\example_folder\\Quinine_PROTON-1-1.jdf: The specified path is invalid." function=processSyncCommand package=main
CreateFile X:\My Downloads\X:\My Downloads\example_folder\Quinine_PROTON-1-1.jdf: The specified path is invalid.

The equivalent commands for put, get and sync work with the linux/amd64 Version (tested with v0.4.3).

[iychoi]

I think there is an issue on path manipulation. I'll fix it soon. Thanks for the report!

[iychoi]

Can you try the new release v0.4.5 and verify if it works on your system? It now works on my Windows machine.

[jnimoth]

Tested today a little bit v0.4.5 on Windows. The get and put command now worked without issues for me.

There are still error messages displayed when running sync. For example. I want to sync this local folder:

PS X:\My Downloads> ls .\D\

    Directory: X:\My Downloads\D

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
------        2-11-2022     20:32           9399 acqu
------        2-11-2022     20:32           9399 acqus
------        2-11-2022     20:32         262144 fid
------        2-11-2022     20:32            166 pulseprogram

with a collection /testZone/home/username/gocommand_sync, I tried the following command:

PS X:\My Downloads> .\gocmd-v0.4.5-windows-amd64\gocmd.exe sync -c windows_config.yaml --progress .\D\ i:/testZone/home/username/gocommands_sync
acqus                                              ... done! [9.40KB in 5.983s]
acqu                                               ... done! [9.40KB in 10.939s]
fid                                                ... done! [262.14KB in 15.618s]
pulseprogram                                       ... done! [166B in 19.743s]
time="2023-01-31 13:43:13.332" level=error msg="error while uploading a local file X:\\My Downloads\\D\\fid to /testZone/home/username/gocommands_sync/fid" error=SYS_NOT_ALLOWED function=ScheduleUploadIfDifferent package=commons struct=ParallelTransferManager
time="2023-01-31 13:43:13.365" level=error msg="error while uploading a local file X:\\My Downloads\\D\\acqu to /testZone/home/username/gocommands_sync/acqu" error=SYS_NOT_ALLOWED function=ScheduleUploadIfDifferent package=commons struct=ParallelTransferManager
time="2023-01-31 13:43:14.118" level=error msg="error while uploading a local file X:\\My Downloads\\D\\pulseprogram to /testZone/home/username/gocommands_sync/pulseprogram" error=SYS_NOT_ALLOWED function=ScheduleUploadIfDifferent package=commons struct=ParallelTransferManager
time="2023-01-31 13:43:39.371" level=error msg="error while uploading a local file X:\\My Downloads\\D\\acqus to /testZone/home/username/gocommands_sync/acqus" error=SYS_NOT_ALLOWED function=ScheduleUploadIfDifferent package=commons struct=ParallelTransferManager
time="2023-01-31 13:43:39.371" level=error msg=SYS_NOT_ALLOWED function=processSyncCommand package=main
SYS_NOT_ALLOWED

The files do still appear in the destination location on the iRODS side:

PS X:\My Downloads> .\gocmd-v0.4.5-windows-amd64\gocmd.exe -c windows_config.yaml ls /testZone/home/username/gocommands_sync
  Quinine_PROTON-1-1.jdf
  acqu
  acqus
  fid
  jdf2bruker.zip
  pulseprogram
  C- /testZone/home/username/gocommands_sync/rclone-v1.60.0-windows-amd64

[iychoi]

Can you check if the same command throws the error in Linux environment? It looks like the issue is due to replication configuration in iRODS server side. By default, gocommands requests replication of a new file uploaded, and it may return the error "SYS_NOT_ALLOWED".

[jnimoth]

Yes, that was indeed the issue! I get the same message also under Linux, but the error (on both, Linux and Windows) is gone when using the additional --no_replication flag.

Thanks!

[iychoi]

Thank you for the confirmation.

I'll add the replication issue on README for other users.