search

IBM / aspera-connect-sdk-js

Official Aspera Connect SDK for Javascript
https://ibm.github.io/aspera-connect-sdk-js/
Apache License 2.0
11 stars 3 forks source link

Getting error in Aspera connect when trying to download ~430 files from cloud using Aspera Connect JS SDK #59

Open AakankshaBhatia opened 12 months ago

AakankshaBhatia commented 12 months ago

Dear Team,

I am trying to transfer multiple files(~430) in single transfer spec using Aspera Connect JS SDK. All files have common source path. User is common for this transfer. Authentication mechanism is token based. Transfer is happening from S3 bucket to user's local system. I get error as soon as transfer comes up in transfer list in Connect UI. The error says: "Token doesn't match file-list/file-pair-list" When I transfer less number of files (~230), i dont get error and files are downloaded properly.

TransferSpec: { authentication: token, cookie: BatchId: 4F183A67-6BA2-459C-AED2-44532CCE5201, create_dir: no, direction: receive, downloadTransactionId: 21640, http_fallback: false, lock_min_rate: no, lock_target_rate: no, min_rate_kbps: 5000, paths: [ { destination: 474 Files_16GB-04 Dec\WildLife\Other Extension/Datsun.doc, source: nmdcUCV2DN/20231204/others/328572.doc }, { is_truncated_for_logging: true, reason: to avoid logging too much when a huge number of files is selected } ], paths_count: 430, paths_datafile: C:\Users\abc\AppData\Local\Aspera\Aspera Connect\txfrpaths/72b3e932-825d-478d-8cf6-2aae8b4591f2.json, remote_host: asperaPublicHost, remote_user: asperaabc_USER, resume_check: full_checksum, ssh_port: 33001, target_rate_kbps: 10000, transferType: ASPERA_DOWNLOAD }

Error log in Aspera client aspera-scp-transfer.0.log file: 2023-12-04 22:58:20.713 [235c-000063ec] ERR Internal error: token/path chunk don't match, token_paths 860, actual paths 544, buffer space used 32726 bytes 2023-12-04 22:58:20.713 [235c-000063ec] ERR Token doesn't match file-list/file-pair-list Last message repeated 1 times 2023-12-04 22:58:20.713 [235c-23ec] LOG FASP Session Start uuid=4905284f-cf17-40b2-b43a-af665f4285e5 op=recv status=failed errcode=105 errstr="Token doesn't match file-list/file-pair-list" source= dest=C:/Users/abc/Downloads source_prefix=- local=0.0.0.0:64366 peer=10.xxx.x.xx:8888 tcp_port=33001 os="Microsoft Windows 6.2 " ver=4.0.0.182202 lic=6:1:1 peeros="-" peerver=- peerlic=0:0:0 proto_sess=20003 proto_udp=20000 proto_bwmeas=20000 proto_data=2000f

Transfer Token is generated using below command(which has all 430 filepaths): astokengen --user=S3_USER --path=nmdcUCV2DN/20231204/others/328572.doc --path=nmdcUCV2DN/20231130/others/309815.gch ....--mode=recv

Also checked that the filepaths passed to generate the transfer token are same in number and order as those passed in transfer spec during transfer.

Is there any restriction on the number of files which can be transferred in bulk using Aspera Connect SDK? Any help on the error will be great.

Thanks in advance.

AakankshaBhatia commented 12 months ago

The aspera manifest file in filelist folder has Recipient IP as 0.0.0.0. Though the working one detects recipient IP properly. In screenshot, left side file is the working one, and right side file is non-working one. image

dwosk commented 12 months ago

Hi @AakankshaBhatia, this error is coming directly from the ascp.exe executable bundled within the Aspera Connect application that actually does the transfer. As such, I would suggest you open up a ticket via the IBM Support portal so you can get a SME to help you as this does not seem specific to the Connect SDK.

However, as far as I know, there is no limit to the number of paths that you are able to specify in the transfer spec. This particular error is interesting:

ERR Internal error: token/path chunk don't match, token_paths 860, actual paths 544, buffer space used 32726 bytes

This seems to indicate that the number of paths "encoded" in your token is 860, but the number of actual paths that ascp.exe has received is 544. So there's a mismatch somewhere.

If you take a look at the manifest and paths JSON file located at the path specified in paths_datafile you should be able to see exactly the paths that ascp.exe is getting.

dwosk commented 12 months ago

And for whatever it's worth, on my end I was successfully able to upload 1000 files that were specified in the paths field in the transfer spec so I don't think that's the issue.

AakankshaBhatia commented 12 months ago

Ok thanks @dwosk for the reply. I will check the data path files once again. And will raise ticket with IBM Support, if needed.

AakankshaBhatia commented 12 months ago

Hi @dwosk , please help with below queries:

dwosk commented 12 months ago

It was using token-based authentication and as far as I'm aware there is no limit using the file pair list sent to ascp.

AakankshaBhatia commented 12 months ago

Ok thanks. Is there any memory buffer which aspera connect uses for comparing the files from token with the input files from path?

I got below error logs during failures with different number of files and different length of filenames, hence asking this question:

Line 7278: 2023-12-04 13:12:30.977 [249c-000006a0] ERR Internal error: token/path chunk don't match, token_paths 858, actual paths 556, buffer space used 32679 bytes
    Line 7294: 2023-12-04 13:37:56.035 [5c90-00006b44] ERR Internal error: token/path chunk don't match, token_paths 698, actual paths 656, buffer space used 32706 bytes
    Line 9035: 2023-12-04 14:04:47.079 [1d20-00002264] ERR Internal error: token/path chunk don't match, token_paths 698, actual paths 656, buffer space used 32706 bytes
    Line 9107: 2023-12-04 14:07:02.736 [4308-00005164] ERR Internal error: token/path chunk don't match, token_paths 676, actual paths 654, buffer space used 32727 bytes
    Line 9319: 2023-12-04 16:49:05.161 [732c-00004d64] ERR Internal error: token/path chunk don't match, token_paths 674, actual paths 654, buffer space used 32714 bytes
    Line 9335: 2023-12-04 19:46:39.937 [6550-00004280] ERR Internal error: token/path chunk don't match, token_paths 834, actual paths 558, buffer space used 32656 bytes
    Line 9351: 2023-12-04 20:03:01.369 [5e84-000063dc] ERR Internal error: token/path chunk don't match, token_paths 834, actual paths 558, buffer space used 32656 bytes
    Line 9367: 2023-12-04 20:04:55.124 [77c4-00000acc] ERR Internal error: token/path chunk don't match, token_paths 834, actual paths 548, buffer space used 32645 bytes
    Line 9645: 2023-12-04 20:34:34.108 [26cc-00006150] ERR Internal error: token/path chunk don't match, token_paths 834, actual paths 558, buffer space used 32656 bytes
    Line 9661: 2023-12-04 20:39:17.883 [528-00005214] ERR Internal error: token/path chunk don't match, token_paths 834, actual paths 544, buffer space used 32679 bytes
    Line 10488: 2023-12-04 22:00:45.923 [7198-00006e20] ERR Internal error: token/path chunk don't match, token_paths 866, actual paths 544, buffer space used 32641 bytes
    Line 10550: 2023-12-04 22:29:30.601 [76b8-00007068] ERR Internal error: token/path chunk don't match, token_paths 866, actual paths 544, buffer space used 32641 bytes
    Line 10680: 2023-12-04 22:50:41.774 [1a28-00000114] ERR Internal error: token/path chunk don't match, token_paths 862, actual paths 544, buffer space used 32744 bytes
    Line 10775: 2023-12-04 22:58:20.713 [235c-000063ec] ERR Internal error: token/path chunk don't match, token_paths 860, actual paths 544, buffer space used 32726 bytes
    Line 10849: 2023-12-05 00:06:28.873 [7204-00000ae0] ERR Internal error: token/path chunk don't match, token_paths 860, actual paths 544, buffer space used 32726 bytes
    Line 10865: 2023-12-05 00:06:32.403 [1550-00005564] ERR Internal error: token/path chunk don't match, token_paths 860, actual paths 544, buffer space used 32726 bytes
    Line 16802: 2023-12-06 12:29:35.767 [458-00002a38] ERR Internal error: token/path chunk don't match, token_paths 860, actual paths 544, buffer space used 32726 bytes
    Line 16923: 2023-12-06 13:28:55.088 [7464-0000386c] ERR Internal error: token/path chunk don't match, token_paths 932, actual paths 554, buffer space used 32696 bytes
    Line 16947: 2023-12-06 20:03:05.049 [51f4-00002fac] ERR Internal error: token/path chunk don't match, token_paths 860, actual paths 544, buffer space used 32726 bytes
    Line 17030: 2023-12-06 21:07:45.645 [389c-000061a0] ERR Internal error: token/path chunk don't match, token_paths 860, actual paths 544, buffer space used 32726 bytes
    Line 17686: 2023-12-06 21:11:00.961 [57e8-000005f8] ERR Internal error: token/path chunk don't match, token_paths 674, actual paths 654, buffer space used 32716 bytes
    Line 18811: 2023-12-06 21:18:44.829 [2c78-00001afc] ERR Internal error: token/path chunk don't match, token_paths 670, actual paths 644, buffer space used 32664 bytes
    Line 21013: 2023-12-08 00:10:03.388 [99c-000035f4] ERR Internal error: token/path chunk don't match, token_paths 828, actual paths 784, buffer space used 32732 bytes
    Line 21083: 2023-12-08 00:37:20.906 [6d30-0000506c] ERR Internal error: token/path chunk don't match, token_paths 798, actual paths 786, buffer space used 32705 bytes
    Line 21235: 2023-12-08 00:41:15.134 [4cb0-00004544] ERR Internal error: token/path chunk don't match, token_paths 788, actual paths 786, buffer space used 32672 bytes
    Line 21704: 2023-12-08 00:48:26.147 [5560-00005460] ERR Internal error: token/path chunk don't match, token_paths 788, actual paths 786, buffer space used 32682 bytes
    Line 21766: 2023-12-08 00:49:37.702 [2f90-00000db0] ERR Internal error: token/path chunk don't match, token_paths 788, actual paths 786, buffer space used 32686 bytes
    Line 21911: 2023-12-08 00:55:04.142 [4a1c-00006b2c] ERR Internal error: token/path chunk don't match, token_paths 790, actual paths 788, buffer space used 32755 bytes
    Line 21977: 2023-12-08 01:12:22.569 [236c-0000429c] ERR Internal error: token/path chunk don't match, token_paths 790, actual paths 788, buffer space used 32737 bytes
    Line 22452: 2023-12-08 01:37:59.365 [5cf0-00000ea8] ERR Internal error: token/path chunk don't match, token_paths 792, actual paths 788, buffer space used 32678 bytes
    Line 23580: 2023-12-08 02:03:59.959 [355c-00006bb8] ERR Internal error: token/path chunk don't match, token_paths 794, actual paths 790, buffer space used 32756 bytes
    Line 23764: 2023-12-08 02:05:49.667 [5b6c-00002584] ERR Internal error: token/path chunk don't match, token_paths 792, actual paths 788, buffer space used 32687 bytes
    Line 23835: 2023-12-08 02:06:55.015 [631c-000030e4] ERR Internal error: token/path chunk don't match, token_paths 792, actual paths 790, buffer space used 32716 bytes
    Line 23948: 2023-12-08 02:08:09.469 [4668-00005d88] ERR Internal error: token/path chunk don't match, token_paths 792, actual paths 790, buffer space used 32728 bytes
    Line 24054: 2023-12-08 02:09:50.973 [30fc-00007014] ERR Internal error: token/path chunk don't match, token_paths 792, actual paths 790, buffer space used 32671 bytes

And as can be seen that the buffer space remains almost same, due to which it is not reading the whole file-pair-list file and hence files' comparison fails. The file size of file-pair-list file is going beyond 32 KB sometimes, but the logs say 32KB buffer space.

Thanks in advance.