search

UPPMAX / irods

Project for implementing an iRODS infrastructure on UPPMAX / SciLifeLab
8 stars 3 forks source link

Access Swestore National Storage via the iRODS SRM Module #1

Closed samuell closed 12 years ago

samuell commented 12 years ago

Compile and install the iRODS SRM module available at: http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/forschung/grid_computing/iRODS ... and use it to connect to SweStore National Storage.

samuell commented 12 years ago

@jhagberg: I'm realizing more and more how hard it is for me to find time to work on these things ... Do you think you could have a look on this one, as soon as we get up a test node, where you have sudo access?

Feel free to tell if you think we should split our work in another way. I just think this is kind of the main stumbling block right now before we can start to play with things in practice, thus it would be nice to put some focus on getting this fixed first/asap ...

samuell commented 12 years ago

@jhagberg: ... I will see if I find some time to get started on this though, but feel free to dig in if you feel like it! :) ... the sooner the better ...

jhagberg commented 12 years ago

To work with this we need a new server grid certificate.

Perhaps we need a new test volume on swestore and a server proxy cert for u5. Samuel can you fix this?

jhagberg commented 12 years ago

@samuell Can you also check with Swestore if it is possible to use gsiftp?

Does it exist a gsiftp://swestore.se?

or is arccp the only way?

samuell commented 12 years ago

@jhagberg: You are very much right about a test volume. Always thought it feels scary to play with the full uppnex account ... especially since there is not even backup on Swestore, only simple mirroring.

Regarding GsiFTP, I have been talking to the Swestore guys about this earlier. They say that the SRM protocol (which is used by the ng* commands) actually wraps the GsiFTP, which in turn is told to be some kind of subset of GridFTP.

But, further down on the GridFTP module page (http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/forschung/grid_computing/iRODS) , they actually have an "SRM" module too, so my thought was to try that module directly!

samuell commented 12 years ago

(Have now mailed Tom and Swestore Support, for a test volume, and server cert for u5 ...)

jhagberg commented 12 years ago

Yes but it say that the SRM with gsftp module make it possible to load files on and off dCACHE Den 14 dec 2011 17:32 skrev "Samuel Lampa" < reply@reply.github.com

:

@jhagberg: You are very much right about a test volume. Always thought it feels scary to play with the full uppnex account ... especially since there is not even backup on Swestore, only simple mirroring.

Regardgin GsiFTP, I have been talking to the Swestore guys about this earlier. They say that the SRM protocol (which is used by the ng* commands) actually wraps the GsiFTP, which in turn is told to be some kind of subset of GridFTP.

But, further down on the GridFTP module page ( http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/forschung/grid_computing/iRODS) , they actually have an "SRM" module too, so my thought was to try that module directly!

Reply to this email directly or view it on GitHub: https://github.com/UPPMAX/irods/issues/1#issuecomment-3143850

jhagberg commented 12 years ago

Server cert you need to fix yorself for the server. Den 14 dec 2011 17:40 skrev "Samuel Lampa" < reply@reply.github.com

:

(Have now mailed Tom and Swestore Support, for a test volume, and server cert for u5 ...)

Reply to this email directly or view it on GitHub: https://github.com/UPPMAX/irods/issues/1#issuecomment-3144127

samuell commented 12 years ago

@jhagberg Ah, right .... but, "GridFTP" you mean? (

"[...] Combinded with the GridFTP module SRM based file transfers are possible [...]" http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/forschung/grid_computing/iRODS

Well, I'll raise the question with the SweStore guys again!

jhagberg commented 12 years ago

Yes and in the module readme they use gsiftp:// Den 14 dec 2011 17:46 skrev "Samuel Lampa" < reply@reply.github.com

:

@jhagberg Ah, right .... but, "GridFTP" you mean? (

"[...] Combinded with the GridFTP module SRM based file transfers are possible [...]"

http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/forschung/grid_computing/iRODS

Well, I'll raise the question with the SweStore guys again!

Reply to this email directly or view it on GitHub: https://github.com/UPPMAX/irods/issues/1#issuecomment-3144222

samuell commented 12 years ago

Ah, I see!

Mattias Wadenstein that there is gsiftp addresses, but no permanent ones ... instead one has to talk the SRM protocol in order to get a GsiFTP dynamically. (I guess this is part of how they do the "round-robin" functionality of spreading things on different storage pools ...).

Excerpt from the chat:

----------Swestore chat-room----------- samuel.lampa 05:47:26 PM Hmm, alltså, jag har frågat förr, men vad var skillnaden mellan GsiFTP och GridFTP? ... borde en mjukvara som stöder SRM+GridFTP funka bra för att ansluta till nationella lagringen? (... även om den ju sägs ha SRM+GsiFTP?) 05:50:57 PM Har förstått att den ena ska vara ett subset av den andra, men har inte riktigt kläm på vad det innebär ... 05:51:26 PM

samuel.lampa 06:02:49 PM Aha, vid närmare koll så använder modulen till och med "gsiftp/" i URL:en, så uppenbarligen borde det fungera, i teorin.

maswan 06:02:55 PM gridftp och gsiftp är samma sak

samuel.lampa 06:03:00 PM Ok! Då blir frågan om man kan ansluta direkt med gsiftp på nått sätt (finns det fasta gsiftp URL:er), eller måste man få dem via SRM? 06:03:43 PM

maswan 06:04:19 PM det finns, ja. men de är inte uppsatta med några permanenta namn eller ens namespace

samuel.lampa 06:04:52 PM Ok, hmm ... men om man förstår hur man pratar SRM så får man en dynamiskt?

maswan 06:04:59 PM precis

samuel.lampa 06:05:36 PM Ok, vi får kika närmare på hur mjukvaran ifråga (SRM modul för iRODS) fungerar då. Danke för feedbacken!

maswan 06:07:09 PM om du har något som bara funkar med gridftp och du vill göra rena mjukvarutester så kan du ju ta den gridftp-dörr som du får i ett srm-svar (arccp med lämplig debugflagga etc), men det är inget som är uppsatt att köras så permanent. om ni sedan kommer fram till att ni behöver en gridftp-dörr med stabil URL etc så får du nog förhandla med tom/jens/nikke/foo

samuel.lampa 06:09:51 PM Ja, Ok! ----------End: Swestore chat-room-----------

samuell commented 12 years ago

Have applied for a TCS server cert for u5 now.

Also, Tom said that we can create a testfolder under the the "ops" folder, if we become member of (or let the server cert become a member of) the ops_test voms group.

Just gotta wait for the cert paper work to get through ...

jhagberg commented 12 years ago

@samuell Could you check if "Globus GSI Libraries" are on place on u5 and Globus Toolkit libraries I am missing srm_types.h when trying to complie srm module

jhagberg commented 12 years ago

Hej Need to have srm-ifce installed

I do not know yet if it is bundled in any rpm but it looks like it is a part of lcgutil https://svnweb.cern.ch/trac/lcgutil/browser/srm-ifce?rev=4651

samuell commented 12 years ago

Ah, there we go ... bit closer to the solution ... will have a look ...

samuell commented 12 years ago

Aarrgh ... seem to only be rpms available for sl5 ... :( http://etics-repository.cern.ch/repository/download/registered/emi/emi.lcgutil.srm-ifce/1.1.0

(Otherwise multiple srm stuff here: http://etics-repository.cern.ch/repository/download/registered/emi )

jhagberg commented 12 years ago

Should we close this case?

samuell commented 12 years ago

@jhagberg I guess, yes, focusing on lftp/univMSSdriver instead, I guess ... (closing now)