search

apache / airavata-mft

Apache Airavata Managed File Transfer Services
https://airavata.apache.org
Apache License 2.0
32 stars 39 forks source link

File resource download hangs when trying to download file #33

Open machristie opened 3 years ago

machristie commented 3 years ago

Description

I'm running download_http.py to generate a download url for a file resource. When I click on the generated URL to download the file, it just hangs.

Steps to Reproduce

I created a SCP file resource

...
  {
    "type": "SCP",
    "resourceId":  "remote-ssh-resource",
    "resourceMode": "FILE",
    "resourcePath": "/tmp/test1/solution_summary.json",
    "storageId" : "remote-ssh-storage"
  },

Then I generate the download URL with a python script (modified download_http.py):

import grpc
import MFTApi_pb2
import MFTApi_pb2_grpc

channel = grpc.insecure_channel('localhost:7004')
stub = MFTApi_pb2_grpc.MFTApiServiceStub(channel)
download_request = MFTApi_pb2.HttpDownloadApiRequest(
    # sourceStoreId ="remote-ssh-storage",
    sourceResourceId = "remote-ssh-resource",
    # sourceResourceId =  "remote-ssh-dir-resource",
    # sourceStoreId ="remote-ssh-dir-resource",
    # sourcePath= "/tmp/test1/foo",
    sourceToken = "local-ssh-cred",
    sourceType= "SCP",
    targetAgent = "agent0",
    # mftAuthorizationToken = ""
    )

result = stub.submitHttpDownload(download_request)
print(result)

When I run the script I get:

$ python download_http.py 
url: "http://localhost:3333/4978023d-c807-46fc-989b-8f550d18021d"
targetAgent: "agent0"

When I load the URL in a browser, the browser just spins and never finishes loading the file.

Expected Behaviour

Loading the URL in the browser would result in a file download.

Your Environment

Additional Context

By using logging statements, I know that it gets to trying to trying to create a session in SCPReceiver.java

https://github.com/apache/airavata-mft/blob/develop/transport/scp-transport/src/main/java/org/apache/airavata/mft/transport/scp/SCPReceiver.java#L103

Apparently it is stuck on this line in SCPTransportUtil:

https://github.com/apache/airavata-mft/blob/develop/transport/scp-transport/src/main/java/org/apache/airavata/mft/transport/scp/SCPTransportUtil.java#L37

because I can log the line before but none of my logging after this line prints.

Also, I know that the SCP configuration is good since I'm able to fetch metadata for the resource:

request = MFTApi_pb2.FetchResourceMetadataRequest(

    resourceId= "remote-ssh-resource",
                                        resourceType = "SCP",
                                        resourceToken = "local-ssh-cred",
                                        targetAgentId = "agent0",
                                        )
response = stub.getFileResourceMetadata(request)
print(response)
machristie commented 3 years ago

I've done some debugging and I've found that for some reason the private key is invalid. But also, when the exception is thrown it is causing the thread pool mechanism to just hang, but I'm not sure why. I'll continue investigating.

machristie commented 3 years ago

It turns out it was a private key format issue. Downloading a file uses the SCPReceiver/SCPSender classes and these classes use the JSch library instead of the ssh4j library. JSch doesn't support the newer format of OpenSSH private keys, see https://stackoverflow.com/questions/53134212/invalid-privatekey-when-using-jsch.

I was able to convert my private key to the older format by using this:

ssh-keygen -p -f local-ssh-cred -m pem

The -m pem argument tells openssh to write private key in the classic PEM format.

This private key is now useable with ssh4j and JSch.

machristie commented 3 years ago

@DImuthuUpe I have this fixed for now but I think there are still a few issues here: