Native iRODS user mapping - gridmap callout

[vladimir-mencl-eresearch]

Hi Roberto,

This is the feature I discussed in #5.

I have it working on my test server, with reasonable test cases tried out.

As discussed, the feature is not activated by default, and the deployer would have to add an extra line to /etc/gridftp.conf to start loading the gridmap_iRODS_callout module.

I have also added another feature (also discussed in #5) to the DSI module directly, changing how the iRODS connection is made - if an environment variable called "$irodsConnectAsAdmin" is set, the code would switch to a different form of the rcConnect call, passing rods as proxyUser and the actual user name as clientUser. If the variable is not set, the original form of rcConnect is used.

I have also made a few fixes to the DSI module:

• Check for NULL in globus_l_gfs_iRODS_destroy (the module was segfaulting when GridFTP server called the destroy method without having created a connection first)
• Use strdup to make copies of strings on the stack
• homeDir: use parsed username without zone (after the mapping module started returning full usernames including zone, the homeDir was being set to "wrong" directory "//home/#/". Use the parsed username without zone instead.

Note that in the gridmap callout code, I've reused the stub from globus_gridmap_verify_myproxy_callout - but as it comes from the same source as the DSI module skeleton, I would not see it as a licensing issue.

Please let me know if the pull request is looking all right to you.

Cheers, Vlad

[vladimir-mencl-eresearch]

Hi Roberto,

[not sure why your last post isn't showing here - I only got it by email and cannot see it here]:

/devel/products/globus/3rdparty/GIT/DSI/B2STAGE-GridFTP/gridmap_iRODS_callout.c:19:42: error: globus_gridmap_callout_error.h: No such file or directory

Yes, to compile the gridmap, module, you need additional Globus devel packages.

I have added that to the requirements in README.md:

+- libglobus-gridmap-callout-error-dev (.deb) or globus-gridmap-callout-error-devel (.rpm)

Installing this package should get you past this compile error.

Cheers, Vlad

[muccix]

Hi Vlad,

sorry, my fault: as soon as I posted the comment, I realized that you added the information related to the package in the README.md. I deleted the comment hoping you didn't receive the notification.

I am now waiting for the system administrator to install the package

Cheers, Roberto

[vladimir-mencl-eresearch]

Hi Roberto,

No worries - my fault I did not figure out that the reason I do not see the comment is that you deleted it :-)

Please let me know if everything is working for you all fine.

Cheers, vlad

[muccix]

Hi Vlad,

I am testing the second feature that would make the DSI module authenticate as the rods admin user.

So far I am not able to make it work.

Here are some logs that may help to understand why.

Gridftp logs:

[15003] Tue Mar 17 10:49:18 2015 :: Configuration read from /devel/products/globus/etc/gridftp.conf. [15003] Tue Mar 17 10:49:18 2015 :: Server started in inetd mode. [15003] Tue Mar 17 10:49:18 2015 :: New connection from: cli.globusonline.org:43850 [15003] Tue Mar 17 10:49:19 2015 :: cli.globusonline.org:43850: [CLIENT]: USER :globus-mapping: [15003] Tue Mar 17 10:49:19 2015 :: cli.globusonline.org:43850: [SERVER]: 331 Password required for :globus-mapping:. [15003] Tue Mar 17 10:49:19 2015 :: cli.globusonline.org:43850: [CLIENT]: PASS dummy [15003] Tue Mar 17 10:49:19 2015 :: iRODS: calling _rcConnect(fec03.cineca.it,1248,proirod1,CINECA, rmucci00, CINECA) [15003] Tue Mar 17 10:49:19 2015 :: cli.globusonline.org:43850: [CLIENT]: PASS dummy [15003] Tue Mar 17 10:49:19 2015 :: cli.globusonline.org:43850: [SERVER]: 530-Login incorrect. : iRODS Error: 'clientLogin' failed.. KRB_NOT_COMPILED_IN: . status: -942000. 530- 530 End. [15003] Tue Mar 17 10:49:19 2015 :: Closed connection from cli.globusonline.org:43850 [15003] Tue Mar 17 10:49:19 2015 :: iRODS: disconnected. [14992] Tue Mar 17 10:49:19 2015 :: Child process 15003 ended with rc = 0

iRODS logs: Mar 17 10:51:50 pid:22046 NOTICE: Agent process 15256 started for puser=proirod1 and cuser=rmucci00 from 130.186.16.3 Mar 17 10:51:51 pid:15256 DEBUG: Mar 17 10:51:51 pid:15256 NOTICE: readAndProcClientMsg: received disconnect msg from client

As you can see, the connection to iRODS is done through the _rcConnect. It is worth noticing that I receive the same error whether I add the server DN to the proirod1 user or not and the same error appears using the "standard" authentication method (invoking rcConnect) if I remove the server DN associated to my user.

It looks like that with the new method the server DN associated to the irods admin user (proirod1 in my case) is not "seen".

Any idea?

Thanks, Roberto

[vladimir-mencl-eresearch]

Hi Roberto,

Ah, OK, troubleshooting time...

Here are my thoughts on what to check:

• the 'proirod1' account you are refering to: that is a rodsAdmin type account? And the .irodsEnv file that '$irodsEnvFile' points to is explicitly setting 'irodsUserName' to 'proirod1'?
• what version of iRODS are you using - client side and server side? This feature only works with 3.3+ (and breaks with 3.2).
• Do you have GSI support compiled in (server-side + client side) ? (But the GSI login works in other cases...so this should not be the case.)

None of these is a rock-solid explanation to me, but are the things I'd check ...

Looking at the logs again, I see the parameters being passed to _rcConnect and they look correct to me - passing 'proirod1' as the rodsAdmin proxy account and 'rmucci00' as the user account.

So I'm suspecting the iRODS version(s) at the moment. Can you please let me know the exact versions? I'd try to replicate it here.

Cheers, Vlad

[muccix]

Hi Vlad,

I double-checked:

• proirod1 is a rodsAdmin user: user_name: proirod1 user_type_name: rodsadmin
• in the .irodsEnv of the user who run the GridFTP server: irodsUserName 'proirod1'
• iRODS server version: rodsServer Release version rods3.3 (actually 3.3.1 with GSI support)
• iRODS client version (not sure why this is needed): $ ienv NOTICE: Release Version = rods3.3, API Version = d

Hope this could help.

Thanks.

Roberto

[muccix]

Hi Vlad,

some news: I tried to "attach" the same gridftp server to our production iRODS instance and it works (replacing the irodsHost and irodsPort info in the .irodsEnv of the user who run the gridftp).

This production iRODS instance should be identical to the devel one, except that in the latter the PAM module is not enabled.

Also, the OS are different: Red Hat Enterprise 6.3 in production, Debian 6 in devel.

I can't see any other difference in the configuration of the two iRODS instances.

Cheers, Roberto

[vladimir-mencl-eresearch]

Hi Roberto,

I have seen some "weird" behavior with the GSI login with the same version of iRODS, 3.3.1.

I saw iRODS was ignoring the username I passed in the proxy user and was instead looking the username up by the DN of the GSI credential.

So could you please check what other accounts is the GridFTP server DN associated with - and remove it from those accounts and leave it only on the proirods1 account?

That might also explain why you get it with the DEV server but not with the PROD server - difference in whether there are other accounts that also accept the GridFTP server DN...

Hope this helps!

Cheers, Vlad

[muccix]

Hi Vlad,

now the GridFTP server DN is associated only to proirod1:

$ iadmin lua | grep /C=IT/O=INFN/OU=Host/L=CINECA-SAP/CN=fec03.cineca.it proirod1 /C=IT/O=INFN/OU=Host/L=CINECA-SAP/CN=fec03.cineca.it

But still receive the same error.

KRB_NOT_COMPILED_IN: . status: -942000

Cheers, Roberto

[vladimir-mencl-eresearch]

Hi Roberto,

This is starting to get very, very confusing.

I started compiling a list of things to check.

And when searching for the the error code you got in the iRODS 3.3.1 source code, I got:

$grep -r KRB_NOT_COMPILED_IN .
./lib/core/src/rodsLog.c:    KRB_NOT_COMPILED_IN, 
./lib/core/src/rodsLog.c:    "KRB_NOT_COMPILED_IN", 
./lib/core/include/rodsErrorTable.h:#define KRB_NOT_COMPILED_IN              -951000

• It only found the error definitions, but no use of them.
• But more surprisingly, the numeric ID of the error is different from the one you got.

When I search for the numeric ID you got, I instead find:

#define GSI_MULTIPLE_MATCHING_DN_FOUND   -942000

Which might possibly mean that in your version of iRODS, the two error tables in lib/core/src/rodsLog.c that map error names to error codes got offset by one - maybe due to a patch being applied that added a row to only one of the tables. I've sanity-checked mine (vanilla 3.3.1) and the rows seem to be aligned properly.

The GSI_MULTIPLE_MATCHING_DN_FOUND code (which is my last night's guess) could be a potentential explanation - it can be returned in ./server/api/src/rsGsiAuthRequest.c.

I'm staring into that file now.

The obvious reason why this error would be returned is when the DN maps to multiple account - but you've just ruled that out.

The code that handles the login via an admin proxy account starts at about line 201 (for 3.3.1).

It first finds a rodsadmin account that matches the DN (and it doesn't even need to see the results, just happy to see the query succeed).

And then it searches for a user account that matches the requested clientUserName.

And then it returns to the generic code.

I think what is happening is that the last query run (find an account by clientUserName) returns multiple results, and that triggers the GSI_MULTIPLE_MATCHING_DN_FOUND. The code otherwise expects to see the results of a query searching for a mapping of the DN - and throws an error if the query output has more than one row. In this case, more than one row is also a problem, just the error code does not fit anymore...

To check this, can you please run the two queries for me that the above code would otherwise run:

iquest "select USER_ID, USER_TYPE, USER_NAME, USER_ZONE where USER_DN = '/C=IT/O=INFN/OU=Host/L=CINECA-SAP/CN=fec03.cineca.it' and USER_TYPE = 'rodsadmin'"
iquest "select USER_ID, USER_TYPE, USER_NAME, USER_ZONE where USER_NAME = 'rmucci00' "

I'm guessing you get two results in the second query. Was that a zone federation experiment? If so, I guess the iRODS server code needs to be patched to add the local zone into the condition...

That's my wild guess for now. I did I get it?

Removing the outside zone account should make it work for now.

Cheers, Vlad

PS: If the above guess is wrong, can you please send me instead:

• Output of the following commands:

  imiscsvrinfo iadmin lu proirod1

• Contents of the .irodsEnv file used by the DSI module
• Run the server with '$spLogLevel' in '$IRODS_HOME/scripts/perl/irodsctl.pl' set to 10 and send me the rodsLog output (please check for any passwords that get logged at that level).

But I hope these won't be needed....

[muccix]

Hi Vlad,

I think you got it!

Look: $ iquest "select USER_ID, USER_TYPE, USER_NAME, USER_ZONE where USER_DN = '/C=IT/O=INFN/OU=Host/L=CINECA-SAP/CN=fec03.cineca.it' and USER_TYPE = 'rodsadmin'" USER_ID = 10007 USER_TYPE = rodsadmin USER_NAME = proirod1 USER_ZONE = CINECA

$ iquest "select USER_ID, USER_TYPE, USER_NAME, USER_ZONE where USER_NAME = 'rmucci00' " USER_ID = 1381022 USER_TYPE = rodsadmin USER_NAME = rmucci00 USER_ZONE = CINECA

USER_ID = 5694373 USER_TYPE = rodsuser USER_NAME = rmucci00 USER_ZONE = CINECA01

I have 2 users defined in the DEV iRODS: $ iadmin lu | grep rmucci00 rmucci00#CINECA rmucci00#CINECA01

Removing rmucci00#CINECA01 everything works!

So we need to patch the iRODS code or send a request to the iRODS developers (in the PROD environment we don't currently have duplicated users).

Thanks a lot; I will proceed with the testing of the third option

Cheers, Roberto

[vladimir-mencl-eresearch]

Hi Roberto,

FYI: just posted this as a question on irod-chat:

https://groups.google.com/forum/#!topic/irod-chat/dOchfGA3ogY

Let's see what kind of response I get there.

Cheers, Vlad

[vladimir-mencl-eresearch]

Hi Roberto,

The above discussion has concluded that:

• this indeed is an iRODS bug in GSI proxy authentication
• the fix I proposed would be acceptable (or at least would not create a security hole)
  • This fix would work for the 3.x branch - and you can get it from http://technical.bestgrid.org/images/8/83/Irods-3.3.1-gsi-proxy-auth-fed.patch.txt
• the bug we discovered does not directly apply to iRODS 4.x - because the GSI proxy authentication code is just not present at all in that version.
  • This means the connectAsAdmin feature would not work with 4.x as such
  • The iRODS developers (Terrell) invited/encouraged me to re-introduce the GSI proxy authentication into the iRODS 4.x code
  • After this is done (and assuming I include the above fix in the code), the connectAsAdmin feature would work also with 4.x.

Would you be happy to merge this pull request in the current state?

Cheers, Vlad

[muccix]

Hi Vlad,

I have been out of office last week. I'm going to start the testing of the gridmap callout module and then merge the pull request.

With respect to the lack of GSI proxy support in iRODS 4: could it be possible that it is included in the GSI Auth Plugin ( http://irods.org/2014/07/gsi-auth-plugin-v1-0-released/ )

Best, Roberto

[vladimir-mencl-eresearch]

Hi Roberto,

The GSI support as such is present in the plugin, but the proxy authentication support got somehow dropped in 4.x - see Terrell's post from March 28th in https://groups.google.com/forum/#!topic/irod-chat/dOchfGA3ogY

I'm now looking into re-adding it back. (It got actually added into 3.x when requested by me :-) )

Cheers, Vlad

PS: No worries about the timing ...

[muccix]

Hi Vlad,

I am testing the authentication through the callout, but unfortunately I'm experiencing some troubles.

The test works: ./testirodsmap "/C=IT/O=INFN/OU=Personal Certificate/L=CINECA-SAP/CN=Roberto Mucci" Level(16) libirodsmap_connect: connected to iRODS server (fec03.cineca.it:1248) Level(16) libirodsmap_query_dn: rcGenQuery returned user rmucci00 (rc=0) Level(16) libirodsmap_query_dn: rcGenQuery returned zone CINECA (rc=0) Mapping for /C=IT/O=INFN/OU=Personal Certificate/L=CINECA-SAP/CN=Roberto Mucci is rmucci00#CINECA

But I am not able to perform the authentication trough the GridFTP server if a remove the grid-mapfile.

In the gridftp.conf i added:

$GSI_AUTH_CONF "/devel/products/globus/3rdparty/GIT/DSI/B2STAGE-GridFTP/callout/gridmap_iRODS_callout.conf"

as well as

$irodsConnectAsAdmin "proirod1"

The file "gridmap_iRODS_callout.conf" contains: globus_mapping libgridmap_iRODS_callout_gcc64pthr gridmap_iRODS_callout

The error message is: Command Failed: Error (login) Server: rmucci00#fec03 (fec03.cineca.it:2812) Message: Login Failed --- 530-Login incorrect. : globus_gss_assist: Gridmap lookup failure: Could not map /C=IT/O=INFN/OU=Personal Certificate/L=CINECA-SAP/CN=Roberto Mucci 530- 530 End.

Also, even though I am not sure it is correlated, I had to modify the CMakeList.txt in the following way to succeed with the compilation:

51 set(gridmap_callout_library_LINK_FLAGS "-lglobus_gridmap_callout_error")
52 #set(gridmap_callout_library_LINK_FLAGS "-lglobus_gridmap_callouterror$ENV{FLAVOR}")

Any idea?

Cheers, Roberto

[vladimir-mencl-eresearch]

Hi Roberto,

On the first look, I'm not sure why it's failing, so could you please try setting full debug by adding these lines to /etc/gridftp.conf:

log_level ALL
log_module stdio:buffer=0
log_single /var/log/globus-gridftp-server.log

and sending me what you get in '/var/log/globus-gridftp-server.log' when attempting to log in?

Wait ... I think I might have an explanation, that would be related to the change you made to CMakeList.txt.

A library is happy to link with unresolved symbols - hoping they would be matched at runtime when the library is loaded.

However, the globus_gridmap_callout_error library would not be loaded independently in a running globus-gridftp-server and has to be explicitly added to the link flags.

If not done, the mapping library libgridmap_iRODS_callout.so fails to load at runtime - and I think that's the problem you are getting.

I did run into the same issue when developing the module.

I'm now not sure what change you made in the CMakeList.txt.

Did you make comment the 'set(gridmap_callout_library_LINK_FLAGS...' line out?

If so, you need to again uncomment the LINK_FLAGS line in CMakeList.txt.

Or did you just remove the '_$ENV{FLAVOR}' part out?

If the build was failing for you, I think that could be because you are setting the FLAVOR environment variable, but you actually have FLAVOR-less Globus installed from RPMs. Is that the case ? Did you install Globus Toolkit from the pre-built RPM binaries? If so, you should unset FLAVOR and the alternative form (at line 56) would get used.

In summary:

• If you were compiling without the LINK_FLAGS, I think I have the explanation - .so failing to load at runtime.
• If you were compiling WITH the LINK_FLAGS, but modified to remove the FLAVOUR, then I have an explanation just for the build error (unset FLAVOR as per above), but I don't know why the mapping is failing - I'd try troubleshooting further from the logs if could please send me that.

Thanks for all the patience with this!

Cheers, Vlad

[muccix]

Hi Vlad,

I just removed "_$ENV{FLAVOR}". The line:

set(gridmap_callout_library_LINK_FLAGS "-lglobus_gridmap_callouterror$ENV{FLAVOR}")

has changed to:

set(gridmap_callout_library_LINK_FLAGS "-lglobus_gridmap_callout_error")

Unfortunately, I have no root permissions so it is not easy for me to make many tests; anyway, yes, Globus was installed from RPMs but the DSI lib has the FLAVOR:

/usr/lib64/libglobus_gridftp_server_iRODS_gcc64pthr.so

Here are the Globus logs (not much interesting as I can understand):

[13917] Wed Apr 1 09:57:16 2015 :: GFork functionality not enabled.: globus_gfork: GFork error: Env not set

[13917] Wed Apr 1 09:57:16 2015 :: Configuration read from /devel/products/globus/etc/gridftp.conf. [13917] Wed Apr 1 09:57:16 2015 :: Server started in inetd mode. [13917] Wed Apr 1 09:57:16 2015 :: New connection from: cli.globusonline.org:47138 [13917] Wed Apr 1 09:57:17 2015 :: cli.globusonline.org:47138: [CLIENT]: USER :globus-mapping: [13917] Wed Apr 1 09:57:17 2015 :: cli.globusonline.org:47138: [SERVER]: 331 Password required for :globus-mapping:. [13917] Wed Apr 1 09:57:17 2015 :: cli.globusonline.org:47138: [CLIENT]: PASS dummy [13917] Wed Apr 1 09:57:17 2015 :: cli.globusonline.org:47138: [CLIENT]: PASS dummy [13917] Wed Apr 1 09:57:17 2015 :: cli.globusonline.org:47138: [SERVER]: 530-Login incorrect. : globus_gss_assist: Gridmap lookup failure: Could not map /C=IT/O=INFN/OU=Personal Certificate/L=CINECA-SAP/CN=Roberto Mucci 530- 530 End. [13917] Wed Apr 1 09:57:17 2015 :: Closed connection from cli.globusonline.org:47138 [13912] Wed Apr 1 09:57:17 2015 :: Child process 13917 ended with rc = 0

What is not clear to me: who and how triggers the callout?

Thanks again Best, Roberto

[vladimir-mencl-eresearch]

Hi Roberto,

The absence of anything from the module in the logs makes me think the callout module is failing to load.

The configuration is triggered by setting the GSI_AUTH_CONF environment variable - and then having a line in that file that gives the library name and a method name under the key globus_mapping - so the line:

globus_mapping libgridmap_iRODS_callout_gcc64pthr gridmap_iRODS_callout

Now, that also means the library has to be in LD_LIBRARY_PATH. I did not add any special instructions about that, as I assume the library gets installed in the same directory as the DSI module and LD_LIBRARY_PATH already gets set for that.

I'm now not sure about how the flavoured/non-flavoured versions of Globus interleave on your system ....

I see the line you have in the gridmap_iRODS_callout.conf file uses the flavoured name of the library.

So can you please:

• run ldd /path/to/libgridmap_iRODS_callout_gcc64pthr.so (and send me the output)
• check that this path is in LD_LIBRARY_PATH ?

Somehow hoping the issue is somewhere in loading the library... please let me know what you find.

Cheers, Vlad

[muccix]

Here I am.

ldd /devel/products/irods/3rdparty/gridftp/globus_gridftp_server-6.19/dsi_bones/DSI/libgridmap_iRODS_callout_gcc64pthr.so

linux-vdso.so.1 =>  (0x00007ffff17ff000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f90a42b0000)
libglobus_gridmap_callout_error.so.0 => /usr/lib/libglobus_gridmap_callout_error.so.0 (0x00007f90a40af000)
libc.so.6 => /lib/libc.so.6 (0x00007f90a3d4c000)
libm.so.6 => /lib/libm.so.6 (0x00007f90a3aca000)
/lib64/ld-linux-x86-64.so.2 (0x00007f90a4843000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f90a38b4000)
libglobus_gssapi_error.so.2 => /usr/lib/libglobus_gssapi_error.so.2 (0x00007f90a36b1000)
libglobus_gss_assist.so.3 => /usr/lib/libglobus_gss_assist.so.3 (0x00007f90a34a3000)
libglobus_gssapi_gsi.so.4 => /usr/lib/libglobus_gssapi_gsi.so.4 (0x00007f90a3284000)
libglobus_gsi_proxy_core.so.0 => /usr/lib/libglobus_gsi_proxy_core.so.0 (0x00007f90a3072000)
libglobus_gsi_credential.so.1 => /usr/lib/libglobus_gsi_credential.so.1 (0x00007f90a2e61000)
libglobus_gsi_callback.so.0 => /usr/lib/libglobus_gsi_callback.so.0 (0x00007f90a2c56000)
libglobus_oldgaa.so.0 => /usr/lib/libglobus_oldgaa.so.0 (0x00007f90a2a4c000)
libglobus_gsi_sysconfig.so.1 => /usr/lib/libglobus_gsi_sysconfig.so.1 (0x00007f90a2840000)
libglobus_gsi_cert_utils.so.0 => /usr/lib/libglobus_gsi_cert_utils.so.0 (0x00007f90a263b000)
libglobus_openssl.so.0 => /usr/lib/libglobus_openssl.so.0 (0x00007f90a2438000)
libglobus_openssl_error.so.0 => /usr/lib/libglobus_openssl_error.so.0 (0x00007f90a2234000)
libglobus_callout.so.0 => /usr/lib/libglobus_callout.so.0 (0x00007f90a202f000)
libglobus_proxy_ssl.so.1 => /usr/lib/libglobus_proxy_ssl.so.1 (0x00007f90a1e29000)
libglobus_common.so.0 => /usr/lib/libglobus_common.so.0 (0x00007f90a1bdf000)
libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8 (0x00007f90a1988000)
libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 (0x00007f90a15e6000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007f90a13ca000)
libltdl.so.7 => /usr/lib/libltdl.so.7 (0x00007f90a11c0000)
libdl.so.2 => /lib/libdl.so.2 (0x00007f90a0fbc000)
libz.so.1 => /usr/lib/libz.so.1 (0x00007f90a0da4000)

Since I don't have root permissions we have linked the files in /usr/lib64/ to a folder I can control, so under /usr/lib64/ I have:

lrwxrwxrwx 1 root root 123 Jul 16 2013 libglobus_gridftp_server_iRODS_gcc64pthr.so -> /devel/products/irods/3rdparty/gridftp/globus_gridftp_server-6.19/dsi_bones/DSI/libglobus_gridftp_server_iRODS_gcc64pthr.so

and

lrwxrwxrwx 1 root root 117 Mar 31 10:48 libgridmap_iRODS_callout_gcc64pthr.so -> /devel/products/irods/3rdparty/gridftp/globus_gridftp_server-6.19/dsi_bones/DSI/libgridmap_iRODS_callout_gcc64pthr.so

Notice that the LD_LIBRARY_PATH env var is not defined. I tried to export it but nothing changes.

I know my configuration is not so convenient to make tests, so I will try to set up test env in which I am more free to change things

Cheers, Roberto

[vladimir-mencl-eresearch]

Hi Roberto,

At this point, I'm not sure what to try next ...

When I was troubleshooting the issue with loading the library, I ended up using both strace to watch the system calls, and setting flags that extract debug output out of ld.so.

I ran

LD_WARN=1 LD_DEBUG=bindings LD_VERBOSE=1 /usr/sbin/globus-gridftp-server -c /etc/gridftp.conf -no-detach -p 2812 -C /etc/gridftp.d -no-fork 

which produced lots of output so I filtred out the gridmap callout library:

LD_WARN=1 LD_DEBUG=bindings LD_VERBOSE=1 /usr/sbin/globus-gridftp-server -c /etc/gridftp.conf -no-detach -p 2812 -C /etc/gridftp.d -no-fork 2>&1 | grep gridmap_iRODS_callout

That in my case revealed unresolved symbols - and I fixed that by adding the callout library to the LINK_FLAGS - but this is already taken care of in your environment.

Just to be sure: in the symlinks from /usr/lib, you are linking both the versioned and unversioned name of the library? Typically, .so files are linked as:

libglobus_gridftp_server_iRODS.so -> libglobus_gridftp_server_iRODS.so.1.6.3

• and I think the dlopen call would be searching for the unversioned .so name.

You may try checking that with

strace /usr/sbin/globus-gridftp-server -c /etc/gridftp.conf -no-detach -p 2812 -C /etc/gridftp.d -no-fork 2>&1 | grep 'open('

Hope this gets us somewhere....

Cheers, Vlad

[muccix]

I perfomed the strace: maybe it is strange that I can't find any reference to "libgridmap_iRODS_callout" but only to "libglobus_gridftp_server_iRODS"

open("/usr/lib/libglobus_gridftp_server_iRODS.la", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/devel/products/irods/3rdparty/gridftp/globus_gridftp_server-6.19/dsi_bones/DSI/libglobus_gridftp_server_iRODS.la", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/lib/libglobus_gridftp_server_iRODS.la", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/libglobus_gridftp_server_iRODS.la", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libglobus_gridftp_server_iRODS.la", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libglobus_gridftp_server_iRODS.la", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libglobus_gridftp_server_iRODS.la", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/libglobus_gridftp_server_iRODS.so", O_RDONLY) = 10

The only references to "callout" are the following:

open("/usr/lib/libglobus_openssl_error.so.0", O_RDONLY) = 3
open("/devel/products/irods/3rdparty/gridftp/globus_gridftp_server-6.19/dsi_bones/DSI/libglobus_callout.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/libglobus_callout.so.0", O_RDONLY) = 3
open("/devel/products/irods/3rdparty/gridftp/globus_gridftp_server-6.19/dsi_bones/DSI/libglobus_gsi_authz_callout_error.so.0", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/lib/libglobus_gsi_authz_callout_error.so.0", O_RDONLY) = 3

Roberto

[vladimir-mencl-eresearch]

Hi Roberto,

The callout library would only be loaded after a connection is received - on the sample command line, GridFTP server would get started at port 2812 - can you also try connecting to the server from a client?

That should trigger the loading for the callout library and you should see more output from the strace...

Cheers, Vlad

[muccix]

Here is the strace, trying to list a folder:

open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 12 open("/etc/grid-security/myproxy/hostcert.pem", O_RDONLY) = 12 open("/etc/grid-security/myproxy/hostkey.pem", O_RDONLY) = 12 open("/etc/grid-security/certificates//2f3fadf6.0", O_RDONLY) = 12 open("/etc/grid-security/certificates//2f3fadf6.signing_policy", O_RDONLY) = 12 open("/etc/grid-security/grid-mapfile", O_RDONLY) = -1 ENOENT (No such file or directory)

Cheers, Roberto

[muccix]

Sorry, I had to modify the path to the gridftp.conf.

The actual output is:

open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 12 open("/etc/grid-security/myproxy/hostcert.pem", O_RDONLY) = 12 open("/etc/grid-security/myproxy/hostkey.pem", O_RDONLY) = 12 open("/etc/grid-security/certificates//2f3fadf6.0", O_RDONLY) = 12 open("/etc/grid-security/certificates//2f3fadf6.signing_policy", O_RDONLY) = 12 open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 12 open("/etc/grid-security/grid-mapfile", O_RDONLY) = -1 ENOENT (No such file or directory)

Roberto

[muccix]

If you think it could be useful, we could organize a skype or hangout call (maybe not now, since I suppose it is quite late there!)

Roberto

[vladimir-mencl-eresearch]

Hi Roberto,

Big apologies about the time wasted.

I've just been looking into why your strace log has no mention of the mapping callout at all - and then checked the documentation against my actual settings and I found I miss-spelled the environment variable name - should have been GSI_AUTHZ_CONF, I wrote it into the documentation as GSI_AUTH_CONF. If you search for the wrong spelling, the only page Google finds is this pull request :-(

Can you please try again with the correct spelling?

Apologies again about wasting so much time on a single typo.... :-(

Cheers, Vlad

[muccix]

Oh Yes!

Now it works! Damn Z!!!!

Very good, I'm going to merge the pull request in a while

Cheers, Roberto