gfal-bringonline does not work with StoRM Tape REST API

andrearendina commented 2 months ago

Hi all,

as described in the releseas notes, https://github.com/cern-fts/gfal2/blob/c07d4d63df06f25050cc6c9fa623acd7f4247ef6/RELEASE-NOTES#L118C55-L118C61 from 2.21.0, Gfal2 supports HTTP tape operations.

At INFN-T1, we have the StoRM Tape REST API deployment on our endpoints.

With

$ gfal-bringonline --version
gfal2-util version 1.8.1 (gfal2 2.22.1)
        dcap-2.22.1
        file-2.22.1
        gridftp-2.22.1
        http-2.22.1
        lfc-2.22.1
        rfio-2.22.1
        sftp-2.22.1
        srm-2.22.1
        xrootd-2.22.1

the following command tells Gfal2 successfully discovered the Tape endpoint:

$ gfal-xattr https://transfer-test.cr.cnaf.infn.it:8443/tape
taperestapi.version = v1
taperestapi.uri = https://storm-tape.cr.cnaf.infn.it:8443/api/v1
taperestapi.sitename = StoRM@CNAF

However, the below command tells it does well an /archiveinfo request (see log in attachment*), but it returns error:

$ gfal-ls https://transfer-test.cr.cnaf.infn.it:8443/tape
2-ddtest10GB
[...]
$ gfal-xattr https://transfer-test.cr.cnaf.infn.it:8443/tape/2-ddtest10GB user.status
gfal-xattr error: 22 (Invalid argument) - [Tape REST API] Archive polling call failed: HTTP 400 : Server Error : <html>
<head><title>400 The SSL certificate error</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The SSL certificate error</center>
<hr><center>nginx/1.24.0</center>
</body>
</html>

On the other hand, gfal-xattr works properly with the srm protocol:

$ gfal-xattr srm://storm-test.cr.cnaf.infn.it:8444/tape/2-ddtest10GB user.status
NEARLINE

How does Gfal2 redirect the request to the StoRM Tape REST API endpoint? Does it perform a curl command to the StoRM Tape REST API with the same X509 voms proxy used for the AuthN/Z to StoRM WebDAV like the following?

$ curl --capath /etc/grid-security/certificates --cacert /tmp/x509up_u$ID --cert /tmp/x509up_u$ID --key /tmp/x509up_u$ID https://storm-tape.cr.cnaf.infn.it:8443/api/v1/archiveinfo/ -d @paths.json
[{"path":"/tape/2-ddtest10GB","locality":"TAPE"}]
$ cat paths.json
{
  "paths": ["/tape/2-ddtest10GB"]
}

The same happens with gfal-bringonline:

$ gfal-bringonline https://transfer-test.cr.cnaf.infn.it:8443/tape/2-ddtest10GB
https://transfer-test.cr.cnaf.infn.it:8443/tape/2-ddtest10GB => FAILED: [Tape REST API] Stage call failed: HTTP 400 : Server Error : <html>
<head><title>400 The SSL certificate error</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The SSL certificate error</center>
<hr><center>nginx/1.24.0</center>
</body>
</html>

You can find the logs attached** also for this command.

Should we open a ticket also to the StoRM developers?

Thanks, Andrea

gfa_xattr_user.status_log.txt * gfal_bringonline_log.txt **

mpatrascoiu commented 2 months ago

Hello Andrea,

Is the endpoint transfer-test.cr.cnaf.infn.it:8443 closed to outside connections? I cannot reach it from CERN.

Your Gfal2 version looks rather old, as well. I wonder if you're not hitting this problem: DMC-1397. Could you upgrade to the latest gfal2 v2.23.0? (available in EPEL)

Could you redo your commands, but this time also add the -D"HTTP PLUGIN:LOG_CONTENT=True" option?

$ gfal-xattr -vvv --log-file=gfal2-xattr-issue22.log \
             -D"HTTP PLUGIN:LOG_CONTENT=True" \
       https://transfer-test.cr.cnaf.infn.it:8443/tape/2-ddtest10GB user.status

Cheers, Mihai

mpatrascoiu commented 2 months ago

Hello,

In fact, although the StoRM instance takes a long time, it eventually replies. The request worked for me:

$ gfal-xattr https://transfer-test.cr.cnaf.infn.it:8443/tape/2-ddtest10GB user.status
NEARLINE

Maybe it's a site issue that was fixed?

Cheers, Mihai

andrearendina commented 1 month ago

Hello Mihai,

with gfal 2.23.0 I can use gfal-xattr and gfal-bringonline properly:

[arendina@ui9-tier1 ~]$ gfal-xattr -D"HTTP PLUGIN:LOG_CONTENT=True" https://transfer-test.cr.cnaf.infn.it:8443/tape/2-ddtest10GB user.status
NEARLINE
[arendina@ui9-tier1 ~]$ gfal-xattr --version
gfal2-util version 1.9.0 (gfal2 2.23.0)
        dcap-2.23.0
        file-2.23.0
        gridftp-2.23.0
        http-2.23.0
        sftp-2.23.0
        srm-2.23.0
        xrootd-2.23.0
[arendina@ui9-tier1 ~]$ gfal-bringonline https://transfer-test.cr.cnaf.infn.it:8443/tape/2-ddtest10GB
Bringonline token: e0766f4e-9dc2-4bb3-b588-0e6e9af6ac6d
https://transfer-test.cr.cnaf.infn.it:8443/tape/2-ddtest10GB QUEUED

I mark the ticket as closed.

Thank you very much, cheers, Andrea

mpatrascoiu commented 1 month ago

Hello,

So in the end, the root of the problem really was DMC-1397! Glad to see upgrading gfal2 unblocked this issue.

Cheers, Mihai

cern-fts / gfal2

gfal-bringonline does not work with StoRM Tape REST API #22