search

IBM / aspera-cli

IBM Aspera CLI
Apache License 2.0
77 stars 17 forks source link

C++ based ascli capability #150

Closed snps-djigande closed 2 months ago

snps-djigande commented 3 months ago

Hi, I could use ascli to upload and download. One limitation of ascli is that the package is full of harcoded path. Even .so files contains hard coded path making ascli not relocatable meaning I can not move the complete package somewhere else and use it.

Would you please provide a C++ based solution consisting of a C/C++ set header files plus C/C++ sources files (or a precompiled static library) we can compile and link in an executable.

Thanks

laurent-martin commented 3 months ago

I have added C++ examples here: https://github.com/laurent-martin/aspera-api-examples/tree/main/cpp

snps-djigande commented 3 months ago

Hi Laurent-martin We are not looking for an example Asperas SDK with C++. Aspera SDK requires OKta authentication which is not good for for us.

What we want is a C++ library implementing what ascli (Aspera CLI not API) is currently doing. ascli is using LDAP (not OKTA) for authentication which is perfect for us, We are mainly interested with downoload not upload.

Please help.

laurent-martin commented 3 months ago

Hi,

Still, use of the Transfer SDK is the way to go.

Important note: in fact ascli just uses the Shares API, nothing else, and this can be done easily in C++. It will be easier and more reliable to do it purely in C++ rather than executing ascli.

Moreover, the transfer SDK supports the shares API, you don't even have to make a REST call for upload/download.

Also, ascli does not use LDAP, it just uses the Shares API, and internally Shares will request the LDAP server.

So, if you use the transfer SDK, like in this example: https://github.com/laurent-martin/aspera-api-examples/blob/main/cpp/src/shares.cpp

then , the user will be auithenticated with LDAP, if it's a LDAP user.

snps-djigande commented 3 months ago

Hi Laurent-Martin

Thanks a lot for you quick reply.

When you say: "Important note: in fact ascli just uses the Shares API, nothing else, and this can be done easily in C++. It will be easier and more reliable to do it purely in C++ rather than executing ascli." This is exactly what I want. I do not want to run the asci executable but link in some C++ code into to my C++ executable.

Questions:

  1. For the Network configuration we have, OKTA 2-layer authentication is required when using Aspera SDK. IBM then provided us the ascli solution to make transfor work with LDAP authentication via the Shares API. Has IBM meanwhile improved Aspera SDK to support the shares API and have the same capabilities and requirements as ascli?

  2. How are user credentials (username, password) passed in your example? Using the transferSpec json node I suppose.

Thanks

Kola

laurent-martin commented 3 months ago
  1. The transfer SDK supports node api auth since the beginning. And Shares has the same API as node api for transfers.
  2. Yes, credentials are provided in the json:
json::array({{
    {"key", "Authorization"},
    {"value", TestEnvironment::basic_auth_header(test_env.conf_str({"shares", "user"}), test_env.conf_str({"shares", "pass"}))}  //
}}),

shares.user and shares.pass are configuration parameters of the sample codes taken from the yaml file. Basically, the Basic authentication header with user's credentials is provided.

It is also possible to call the shares (node) api directly to create a transfer spec v1, like in the python example: https://github.com/laurent-martin/aspera-api-examples/blob/main/python/src/shares.py

snps-djigande commented 3 months ago

Thanks a lot Laurent-Martin

We are using Aspera Shares web application (okta or ldap) and HSTS architecture. How to access Aspera transfer SDK on our architecture?

But, I will give it a try, Where do I get the C++ sources (or header + static library) of the Transfer SDK objects used in your example so that build and test it (e.g transfersdk) on our server?

From: https://developer.ibm.com/apis/catalog/aspera--aspera-transfer-sdk/downloads/downloads.json ?

Thanks

Kola

laurent-martin commented 3 months ago

Yes, On the link you provide, you can get, for supported platforms:

There is no static library, as the normal way is:

The SDK provides generated code for convenience, though. It might speed up development, but eventually it's best to generate the stub yourself.

The C++ sample in this repo generates the stub files and do not use the ones from SDK.

snps-djigande commented 3 months ago

Hi Laurent-Martin

OK, Thanks I tried using make to build your example and run into issue when unzipping the downloaded transfer_sdk.zip file. I am running on CentOS Linux release 7.3.1611 (Core) Any OS requirements?

I could download an sdk.zip file which I could unzip

Thanks

unzip -qud ./generated/trsdk/ ./generated/trsdk/transfer_sdk.zip
[./generated/trsdk/transfer_sdk.zip]
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of ./generated/trsdk/transfer_sdk.zip or
        ./generated/trsdk/transfer_sdk.zip.zip, and cannot find ./generated/trsdk/transfer_sdk.zip.ZIP, period.
laurent-martin commented 3 months ago

Well, the error message says that it cannot find the file: ./generated/trsdk/transfer_sdk.zip

Here a way to get it:

$ mkdir -p generated/trsdk
$ curl -Lo generated/trsdk/transfer_sdk.zip https://ibm.biz/aspera_transfer_sdk
$ cksum generated/trsdk/transfer_sdk.zip
1142155211 97751681 generated/trsdk/transfer_sdk.zip
$ unzip -d generated/trsdk generated/trsdk/transfer_sdk.zip

I tested that on 2 systems: Linux and macOS.

snps-djigande commented 3 months ago

Hi Laurent-martin The .zip file is present, the error message is about unzip failing on it. ls -ll says: 15510 Aug 20 15:44 transfer_sdk.zip

check sum is indeed strange: cksum ./transfer_sdk.zip 3750263575 15490 ./transfer_sdk.zip

But for this issue I do have a solution, which is to override the downloaded file transfer_sdk.zip file a sdk.zip file I downloaded in the Makefile before unzip is called.

snps-djigande commented 3 months ago

Hi Laurent-martin,

Made some progress building the example. I am now stuck and conan install conan is downloading multiple boost versions in my $HOME/.conan2 and compiling files in $HOME/.conan2 exceeding my disk quota.

Is there is way override $HOME/.conan2 build directory ? Using the conan profile for instance? setting $HOME seems to help but is not nice

Thanks Kola

laurent-martin commented 3 months ago

Well, the answer probably lies in the manual, right ?

https://docs.conan.io/2/reference/environment.html

export CONAN_HOME=$PWD/.conan2
conan profile detect --force
snps-djigande commented 3 months ago

Hi Laurent-Martin

Thanks for the environment variable.

The following lnk is broken for me: https://developer.ibm.com/apis/catalog/aspera--aspera-transfer-sdk/Quick+Start+for+Sample+Code:+C++

Thanks Merci

Kola

snps-djigande commented 3 months ago

Hi Laurent Thanks a lot for your help. I made good progress in using the C++ shares example.

One (I hope final) issue though I am getting the following invalid token error message when trying to upload. Would you please help understand the root cause of this issue and a find a fix.

{"appname":"faspmanagerd","hostname":"127.0.0.1","level":"info","msg":"Endpoint (QueryTransfer) accepts request: (transferId:\"4846c4b7-a099-4455-82cc-5a06f5cd1111\")","time":"2024-08-22T17:11:41+02:00"}
{"appname":"faspmanagerd","hostname":"127.0.0.1","level":"info","msg":"Endpoint (QueryTransfer) returns response: (apiVersion:\"1\" transferId:\"4846c4b7-a099-4455-82cc-5a06f5cd1111\" title:\"send using Node API and ts v2\" transferType:FILE_REGULAR status:FAILED transferInfo:{endTimeUsec:1724339488000 errorCode:\"34\" errorDescription:\" \\n Server aborted session: Authorization refused: invalid token - transfer files don't match\" targetRateKbps:810000 fileChecksumType:\"none\" cookie:\"aspera.shares2:f01cda1e-a96a-478f-9500-9a58c9f6a1d3:S29sYSBEamlnYW5kZQ==:dXBsb2FkIHRvIHNoYXJlIFZwcm90b1Rlc3RfU3lub3BzeXNfMzIyNDMwIHZpYSBBUEk=\" direction:\"send\" tags64:\"eyJhc3BlcmEiOnsic2hhcmVzIjp7ImRlc3Rfc2hhcmUiOiJWcHJvdG9UZXN0X1N5bm9wc3lzXzMyMjQzMCIsImRlc3Rfc2hhcmVfaWQiOjIwNTc0OCwidHJhbnNmZXJfaWQiOiJmMDFjZGExZS1hOTZhLTQ3OGYtOTUwMC05YTU4YzlmNmExZDMiLCJ1c2VyIjoiS29sYSBEamlnYW5kZSIsInVzZXJfaWQiOjIwNTc5Mn0sInhmZXJfaWQiOiI0ODQ2YzRiNy1hMDk5LTQ0NTUtODJjYy01YTA2ZjVjZDExMTEifX0=\"}) after (67.998µs) ","time":"2024-08-22T17:11:41+02:00"}

Thanks

Kola

laurent-martin commented 3 months ago

it says:

invalid token - transfer files don't match

can you provide the content of key: assets you are using in the transfer spec ?

When using transfer spec v2, internally, asperatransferd calls the shares API, like in this python example with transfer spec V1:

https://github.com/laurent-martin/aspera-api-examples/blob/main/python/src/shares.py

That error means that the transfer is started for a list of files that do not match the list provided when the request for token was made. I think it should not happen when using v2, but it does here, so it might have to do with the content of transfer spec.

snps-djigande commented 3 months ago

Hi Laurent

Thanks for your answer. Here is the cout print of the entire transfer spec. I did not change the basic Authorization value you create in src/test_environment.hpp "Basic " + ...

 {
  "assets": {
    "destination_root": "/<DIR>",
    "paths": [
      {
        "destination": "This_is_a_test.txt",
        "source": "<DIR>/aspera-api-examples/tmp/This_is_a_test.txt"
      }
    ]
  },
  "direction": "send",
  "session_initiation": {
    "node_api": {
      "headers": [
        {
          "key": "Authorization",
          "value": "Basic ..."
        }
      ],
      "url": "https://<ADDRESS>/node_api"
    }
  },
  "title": "send using Node API and ts v2"
}

Here are some lines in my config.yaml file are:

misc:
  platform: linux-x86_64
trsdk:
  url: grpc://127.0.0.1:55002
server:
  url: ssh://<OUR SERVER ADDRESS>:33001
  user: _user_here_
  pass: _pass_here_
  file_download: "/aspera-test-dir-small/10MB.1"
  folder_upload: "/Upload"

Here are more lines of aspera-api-examples/tmp/asperatransferd.log

{"appname":"faspmanagerd","hostname":"127.0.0.1","level":"info","msg":"Endpoint (StartTransfer) accepts request: (transferType:FILE_REGULAR config:{logLevel:2} transferSpec:\"{\\\"assets\\\":{\\\"destination_root\\\":\\\"/<DIR>\\\",\\\"paths\\\":[{\\\"destination\\\":\\\"This_is_a_test.txt\\\",\\\"source\\\":\\\"/<DIR>/aspera-api-examples/tmp/This_is_a_test.txt\\\"}]},\\\"direction\\\":\\\"send\\\",\\\"session_initiation\\\":{\\\"node_api\\\":{\\\"headers\\\":[{\\\"key\\\":\\\"Authorization\\\",\\\"value\\\":\\\"*******\\\"}],\\\"url\\\":\\\"<ADDRESS>/node_api\\\"}},\\\"title\\\":\\\"send using Node API and ts v2\\\"}\")","time":"2024-08-23T09:17:41+02:00"}
{"appname":"faspmanagerd","hostname":"127.0.0.1","level":"info","msg":"Info: [API server] StartTransfer TransferSpec: (model.TransferSpec{direction:\"send\", remoteHost:\"\", title:\"send using Node API and ts v2\", initiation:(*model.InitiationSpec)(0xc0003487a0), security:(*model.SecuritySpec)(0xc00031e9c0), tracking:(*model.TrackingSpec)(0xc00031db60), filesystem:(*model.FileSystemSpec)(0xc00033e3c0), transport:(*model.TransportSpec)(0xc000324690), assets:(*model.AssetSpec)(0xc00032f260), extraOpts:map[string]string(nil)}) intent (1)","time":"2024-08-23T09:17:41+02:00"}
{"appname":"faspmanagerd","hostname":"127.0.0.1","level":"debug","msg":"Debug: Trans*******)\n","time":"2024-08-23T09:17:41+02:00"}
{"appname":"faspmanagerd","hostname":"127.0.0.1","level":"info","msg":"Info: FaspServer: startTransfer: started transfer id=c36ba5d0-60b3-4165-8548-8d76f1df81be type=FileRegular","time":"2024-08-23T09:17:41+02:00"}

...

{"appname":"faspmanagerd","hostname":"127.0.0.1","level":"info","msg":"Info: EventManager: done notifying 0 listener(s) about event (model.TransferResponse{ApiVersion:\"\", TransferId:\"c36ba5d0-60b3-4165-8548-8d76f1df81be\", Title:\"send using Node API and ts v2\", TransferType:1, Status:\"FAILED\", TransferEvent:\"Session Error\", Terminated:true, TransferInfo:(*model.TransferInfo)(0xc000147680), SessionTransferInfo:(*model.SessionTransferInformation)(0xc0004d6000), FileTransferInfo:(*model.FileTransferInformation)(nil), Message:\"{\\\"Loss\\\":0,\\\"Code\\\":34,\\\"Description\\\":\\\"Server aborted session: Authorization refused: invalid token - transfer files don't match\\\",\\\"SessionId\\\":\\\"3df9a7c8-e947-42fa-bd3b-fd13cff1866a\\\",\\\"UserStr\\\":\\\"c36ba5d0-60b3-4165-8548-8d76f1df81be_s_70dbabeb-b4ae-48c1-854e-82518f3fd08e\\\"}\", Error:(*model.Error)(nil)}) transferInfo (\u0026{AverageRateKbps:0 BytesLost:0 BytesTransferred:0 BytesWritten:0 DirectoriesCompleted:0 StartTimeUsec:0 ElapsedUsec:0 EndTimeUsec:1724397463000 ErrorCode:34 ErrorDescription: \n Server aborted session: Authorization refused: invalid token - transfer files don't match FilesCompleted:0 TargetRateKbps:810000 MinRateKbps:0 FileChecksumType:none Cookie:aspera.shares2:142e9bf7-e6b5-4211-a3a3-817c8225ad72:S29sYSBEamlnYW5kZQ==:dXBsb2FkIHRvIHNoYXJlIFZwcm90b1Rlc3RfU3lub3BzeXNfMzIyNDMwIHZpYSBBUEk= Direction:send Operation: Tags64:eyJhc3BlcmEiOnsic2hhcmVzIjp7ImRlc3Rfc2hhcmUiOiJWcHJvdG9UZXN0X1N5bm9wc3lzXzMyMjQzMCIsImRlc3Rfc2hhcmVfaWQiOjIwNTc0OCwidHJhbnNmZXJfaWQiOiIxNDJlOWJmNy1lNmI1LTQyMTEtYTNhMy04MTdjODIyNWFkNzIiLCJ1c2VyIjoiS29sYSBEamlnYW5kZSIsInVzZXJfaWQiOjIwNTc5Mn0sInhmZXJfaWQiOiJjMzZiYTVkMC02MGIzLTQxNjUtODU0OC04ZDc2ZjFkZjgxYmUifX0= ArgTransfersAttempted:0 ArgTransfersPassed:0 ArgTransfersSkipped:0 ArgTransfersFailed:0 TransfersSkipped:0 TransfersPassed:0 TransfersFailed:0 TransfersAttempted:0}) sessionInfo (\u0026{ID:70dbabeb-b4ae-48c1-854e-82518f3fd08e SessionID:3df9a7c8-e947-42fa-bd3b-fd13cff1866a User:shares ClientUser:djigande ClientNodeID: ClientClusterID: ServerNodeID:63fd4d93-b07b-4836-aac3-b57d696a1b3c ServerClusterID:e89c67ab-5b68-464a-b125-90dfc7739200 ClientIPAddress: ServerIPAddress: Port:33001 TCPPort:33001 Status:Failed StartTimeUsec:1724397463000 EndTimeUsec:1724397463000 ElapsedUsec:0 BytesTransferred:0 BytesWritten:0 BytesLost:0 FilesCompleted:0 FilesFailed:0 FilesSkipped:0 DirectoriesCompleted:0 TargetRateKbps:810000 MinRateKbps:0 CalcRateKbps:0 NetworkDelayUsec:0 ErrorCode:34 ErrorDesc:Server aborted session: Authorization refused: invalid token - transfer files don't match ManifestFilePath: SourcePathsScanExcluded:0 SourcePathsScanIrregular:0 SourcePathsScanFailed:0 SourcePathsScanAttempted:0 TransfersSkipped:0 TransfersPassed:0 TransfersFailed:0 TransfersAttempted:0 Cookie:aspera.shares2:142e9bf7-e6b5-4211-a3a3-817c8225ad72:S29sYSBEamlnYW5kZQ==:dXBsb2FkIHRvIHNoYXJlIFZwcm90b1Rlc3RfU3lub3BzeXNfMzIyNDMwIHZpYSBBUEk= Direction:Send FileChecksumType:none Operation:Transfer Tags:{\"aspera\":{\"shares\":{\"dest_share\":\"VprotoTest_Synopsys_322430\",\"dest_share_id\":205748,\"transfer_id\":\"142e9bf7-e6b5-4211-a3a3-817c8225ad72\",\"user\":\"MUSER NAME>\",\"user_id\":<USER_ID>},\"xfer_id\":\"c36ba5d0-60b3-4165-8548-8d76f1df81be\"}} ArgTransfersAttempted:0 ArgTransfersPassed:0 ArgTransfersSkipped:0 ArgTransfersFailed:0 Encryption:Yes Adaptive:Trickle Remote:No Destination:/<DIR>Priority:2 TransferID:c36ba5d0-60b3-4165-8548-8d76f1df81be RateCap:4503599627370495 MinRateCap:4503599627370495 PolicyCap: RateLock:No MinRateLock:Yes PolicyLock:No ServerHostname:<SERVER_NAME> RemoteAddress:149.117.73.92 Cipher:aes-128 ResumePolicy:policy_sparse_csum CreatePolicy:0 ManifestPolicy:manifest_none Precalc:No OverwritePolicy:diff RTTAutocorrect:Yes TimePolicy:0 ManifestInprogress:.aspera-inprogress FilesEncrypt:No FilesDecrypt:No DatagramSize:0 VLinkVersion:0 PeerVLinkVersion:0 VLinkLocalEnabled:No VLinkRemoteEnabled:No ReadBlockSize:0 WriteBlockSize:0 ClusterNumNodes:0 ClusterNodeID:0 MoveRange:No Keepalive:No TestLogin:No UseProxy:No RateControlAlgorithm:alg_delay PMTU:0 PreTransferFiles:0 PreTransferBytes:0 PreTransferDirs:0 PreTransferSpecial:0 SourcePathsScanCompleted:0 ArgScansAttempted:0 ArgScansCompleted:0 ArgFaspFileArgIndex:0 DirCreatesAttempted:0 DirCreatesFailed:0 DirCreatesPassed:0 DirScansCompleted:0}) fileInfo (\u003cnil\u003e)","time":"2024-08-23T09:17:56+02:00"}
{"appname":"faspmanagerd","hostname":"127.0.0.1","level":"info","msg":"Endpoint (QueryTransfer) accepts request: (transferId:\"c36ba5d0-60b3-4165-8548-8d76f1df81be\")","time":"2024-08-23T09:17:56+02:00"}
{"appname":"faspmanagerd","hostname":"127.0.0.1","level":"info","msg":"Endpoint (QueryTransfer) returns response: (apiVersion:\"1\" transferId:\"c36ba5d0-60b3-4165-8548-8d76f1df81be\" title:\"send using Node API and ts v2\" transferType:FILE_REGULAR status:FAILED transferInfo:{endTimeUsec:1724397463000 errorCode:\"34\" errorDescription:\" \\n Server aborted session: Authorization refused: invalid token - transfer files don't match\" targetRateKbps:810000 fileChecksumType:\"none\" cookie:\"aspera.shares2:142e9bf7-e6b5-4211-a3a3-817c8225ad72:S29sYSBEamlnYW5kZQ==:dXBsb2FkIHRvIHNoYXJlIFZwcm90b1Rlc3RfU3lub3BzeXNfMzIyNDMwIHZpYSBBUEk=\" direction:\"send\" tags64:\"eyJhc3BlcmEiOnsic2hhcmVzIjp7ImRlc3Rfc2hhcmUiOiJWcHJvdG9UZXN0X1N5bm9wc3lzXzMyMjQzMCIsImRlc3Rfc2hhcmVfaWQiOjIwNTc0OCwidHJhbnNmZXJfaWQiOiIxNDJlOWJmNy1lNmI1LTQyMTEtYTNhMy04MTdjODIyNWFkNzIiLCJ1c2VyIjoiS29sYSBEamlnYW5kZSIsInVzZXJfaWQiOjIwNTc5Mn0sInhmZXJfaWQiOiJjMzZiYTVkMC02MGIzLTQxNjUtODU0OC04ZDc2ZjFkZjgxYmUifX0=\"}) after (66.238µs) ","time":"2024-08-23T09:17:56+02:00"}

Thanks

laurent-martin commented 2 months ago

Hi, Mmm, in fact transfer spec V2 works with node , and should work with shares, but I had the same error.

I have revised the Shares example, to use transfer spec V1 and directly the REST API of Shares:

https://github.com/laurent-martin/aspera-api-examples/blob/main/cpp/src/shares.cpp

snps-djigande commented 2 months ago

Hi Laurent Thanks a lot for updating the share example. Kola

snps-djigande commented 2 months ago

Hi Laurent,

Do you by chance also have an example which use transfer spec V1 and directly the REST API of Shares to download a file too?

Thanks

laurent-martin commented 2 months ago

Yes, I have added download case here:

https://github.com/laurent-martin/aspera-api-examples/blob/main/cpp/src/shares.cpp#L26

snps-djigande commented 2 months ago

Thanks Laurent Merci infiniment!

The example works for me (I could upload and download). Have meanwhile figured out why the example did not work with the new transfer spec V2? Any chance to get work with V2 (No REST api of shares) soon?

Thanks

Kola

laurent-martin commented 2 months ago

Fantastic.

For v2, I have not figured out yet. Basically, when using v2, the transfer daemon does exactly the same REST call to Shares/Node API.

To me, for the time being, it seems to be a bug in the transfer daemon, I have asked to the IBM Aspera dev team, and will let you know when I have any news.

Personally, I prefer to use the v1 transfer spec, as it works for all Aspera applications and leaves the freedom to use the App' REST API with all bells and whistles.

snps-djigande commented 2 months ago

Hi Laurent and all

Thanks a lot for you help and for the update.

Would you please provide a C++ example on how to browse the content of a remote folder (e.g /) and save the result in a destination file? An equivalent of "ascli shares files browse / --url=..." but using aspera C++ api.

Thanks Kola

laurent-martin commented 2 months ago

In fact, run the ascli command in debug or trace2 level and it shows api call... it's one of the possible use for ascli: learn APIs by seeing them working.

snps-djigande commented 2 months ago

Thanks a lot Laurent, Debug log was a good hint to find out the post arguments.

D, [2024-09-15T16:51:34.392916 #36566] DEBUG -- : command=browse
D, [2024-09-15T16:51:34.393127 #36566] DEBUG -- : path=/My Folders
D, [2024-09-15T16:51:34.393149 #36566] DEBUG -- : (value) get query=
D, [2024-09-15T16:51:34.393181 #36566] DEBUG -- : POST [files/browse]