search

EUDAT-B2SAFE / B2SAFE-core

B2SAFE service core code for EUDAT project
Other
14 stars 20 forks source link

Functional differences between iRODS v3 and v4 regarding data streaming and acPostProcForPut #18

Closed janiheikkinen closed 8 years ago

janiheikkinen commented 9 years ago

iRODS v3 and v4 behave differently (in calling hooks/actions) when the following test rule is run:

writeFile {
    msiDataObjCreate("*file", "forceFlag=", *filePointer);
    msiDataObjWrite(*filePointer, "*contents", *bytesWritten);
    msiDataObjClose(*filePointer, *outStatus);
}
INPUT *file="/zone/replicate/test.replicate",
*contents="Is the acPostProcForPut hook triggered?"
OUTPUT ruleExecOut
janiheikkinen commented 9 years ago

https://github.com/irods/irods/issues/2055

cookie33 commented 9 years ago

Hi,

I did some hacking and got it to go further with the modification of the writeFile rule:

writeFile(_file, contents) { msiDataObjCreate("_file.tmp", "forceFlag=", _filePointer); logDebug("[writeFile] Created object: file"); msiDataObjWrite(_filePointer, "_contents", *bytesWritten); msiDataObjClose(_filePointer, _outStatus);

msiDataObjCopy("_file.tmp","_file","forceFlag=",_Status);

msiSplitPath("_file.tmp",_Coll,_FileName);
msiDataObjGet("_file.tmp","localPath=/tmp/_FileName++++forceFlag=",_Status);
logDebug("[getFile] File _file.tmp is retrieved from the data grid");
msiDataObjPut(_file,"demoResc","localPath=/tmp/_FileName++++forceFlag=",_Status);
logDebug("[putFile] File *file is put to the data grid");

}

I created the file as before, with the extension .tmp. Than I get with iget. And I put it with iput with the correct extension. The problem is with the resource which I have to fill in.

Than the acPostProcForPut is triggered.

But than it fails miserably on an other place.

But this is a nasty workaround and the process becomes even slower.

Greetings,

Robert

On 04 Feb 2015, at 12:59, janiheikkinen notifications@github.com wrote:

iRODS v3 and v4 behave differently (in calling hooks/actions) when the following test rule is run:

writeFile { msiDataObjCreate("file", "forceFlag=", filePointer); msiDataObjWrite(filePointer, "contents", bytesWritten); msiDataObjClose(filePointer, outStatus); } INPUT file="/zone/replicate/test.replicate", *contents="Is the acPostProcForPut hook triggered?" OUTPUT ruleExecOut

— Reply to this email directly or view it on GitHub https://github.com/EUDAT-B2SAFE/B2SAFE-core/issues/18.

Met vriendelijke groet,

Robert Verkerk systeem programmeur Data Services

SURFsara heeft een nieuw algemeen telefoonnummer: 020 800 1300

Robert Verkerk | Dataservices | SURFsara | Science Park 140 | 1098 XG Amsterdam | T +31 (0)20 8001300 | robert.verkerk@surfsara.nl | www.surfsara.nl

cookie33 commented 9 years ago

Hi,

The following also did not work:

A next attempt: create createFile.sh and call it. writeFile(_file, contents) { logDebug("[writeFile] Created object: file"); msiSplitPath("_file.tmp",_Coll,_FileName); msiExecCmd("createFile.sh","_FileName '_contents'", "null", "null", "null", _out); msiDataObjPut(_file,"demoResc","localPath=/tmp/_FileName++++forceFlag=",_Status); logDebug("[putFile] File *file is put to the data grid"); } and createFile.sh:

10:05 irodstest1.storage.sara.nl:/home/rods rods$ cat /var/lib/irods/iRODS/server/bin/cmd/createFile.sh

/bin/bash

set -x

echo "$2" > /tmp/$1 NO GO on the remote side.. Locally it works..

Greetings,

Robert

On 06 Feb 2015, at 09:26, Robert Verkerk robert.verkerk@surfsara.nl wrote:

Hi,

I did some hacking and got it to go further with the modification of the writeFile rule:

writeFile(_file, contents) { msiDataObjCreate("_file.tmp", "forceFlag=", _filePointer); logDebug("[writeFile] Created object: file"); msiDataObjWrite(_filePointer, "_contents", *bytesWritten); msiDataObjClose(_filePointer, _outStatus);

msiDataObjCopy("_file.tmp","_file","forceFlag=",_Status);

msiSplitPath("_file.tmp",_Coll,_FileName);
msiDataObjGet("_file.tmp","localPath=/tmp/_FileName++++forceFlag=",_Status);
logDebug("[getFile] File _file.tmp is retrieved from the data grid");
msiDataObjPut(_file,"demoResc","localPath=/tmp/_FileName++++forceFlag=",_Status);
logDebug("[putFile] File *file is put to the data grid");

}

I created the file as before, with the extension .tmp. Than I get with iget. And I put it with iput with the correct extension. The problem is with the resource which I have to fill in.

Than the acPostProcForPut is triggered.

But than it fails miserably on an other place.

But this is a nasty workaround and the process becomes even slower.

Greetings,

Robert

On 04 Feb 2015, at 12:59, janiheikkinen <notifications@github.com mailto:notifications@github.com> wrote:

iRODS v3 and v4 behave differently (in calling hooks/actions) when the following test rule is run:

writeFile { msiDataObjCreate("file", "forceFlag=", filePointer); msiDataObjWrite(filePointer, "contents", bytesWritten); msiDataObjClose(filePointer, outStatus); } INPUT file="/zone/replicate/test.replicate", *contents="Is the acPostProcForPut hook triggered?" OUTPUT ruleExecOut

— Reply to this email directly or view it on GitHub https://github.com/EUDAT-B2SAFE/B2SAFE-core/issues/18.

Met vriendelijke groet,

Robert Verkerk systeem programmeur Data Services

SURFsara heeft een nieuw algemeen telefoonnummer: 020 800 1300

Robert Verkerk | Dataservices | SURFsara | Science Park 140 | 1098 XG Amsterdam | T +31 (0)20 8001300 | robert.verkerk@surfsara.nl mailto:robert.verkerk@surfsara.nl | www.surfsara.nl http://www.surfsara.nl/

Met vriendelijke groet,

Robert Verkerk systeem programmeur Data Services

SURFsara heeft een nieuw algemeen telefoonnummer: 020 800 1300

Robert Verkerk | Dataservices | SURFsara | Science Park 140 | 1098 XG Amsterdam | T +31 (0)20 8001300 | robert.verkerk@surfsara.nl | www.surfsara.nl

andyjenkinson commented 9 years ago

I noticed that the hook is fired when overwriting a file that already exists:

Test rule:

writeFile {
   msiDataObjCreate("*file", "forceFlag=", *filePointer);
   logDebug("TEST writeFile, created object: *file");
   msiDataObjWrite(*filePointer, "*contents", *bytesWritten);
   msiDataObjClose(*filePointer, *outStatus);
   logDebug("TEST writeFile, outStatus: *outStatus"); 
}
INPUT *file="/ebi/pride/test.replicate", *contents="justsomecontent"
OUTPUT ruleExecOut

Run it:

aj@irods-eudat:~$ irule -vF test-replicate.r
rcExecMyRule: writeFile {
   msiDataObjCreate("*file", "forceFlag=", *filePointer);
   logDebug("TEST writeFile, created object: *file");
   msiDataObjWrite(*filePointer, "*contents", *bytesWritten);
   msiDataObjClose(*filePointer, *outStatus);
   logDebug("TEST writeFile, outStatus: *outStatus"); 
}

outParamDesc: ruleExecOut
ExecMyRule completed successfully.    Output

Logs:

Feb  6 11:29:46 pid:16111 NOTICE: Agent process 16263 exited with status 0
Feb  6 11:29:46 pid:16111 NOTICE: Agent process 16308 started for puser=pride and cuser=pride from 127.0.0.1
Feb  6 11:29:46 pid:16308 NOTICE: +Testing Rule Number:0 for Action:acChkHostAccessControl
Feb  6 11:29:46 pid:16308 NOTICE: +Testing Rule Number:0 for Action:acSetPublicUserPolicy
Feb  6 11:29:47 pid:16308 NOTICE: +Testing Rule Number:0 for Action:acSetRescSchemeForCreate
Feb  6 11:29:47 pid:16308 NOTICE: +Testing Rule Number:0 for Action:acRescQuotaPolicy
Feb  6 11:29:47 pid:16308 NOTICE: +Testing Rule Number:0 for Action:acPreprocForDataObjOpen
Feb  6 11:29:47 pid:16308 NOTICE: +Testing Rule Number:0 for Action:logDebug
Feb  6 11:29:47 pid:16308 NOTICE: writeLine: inString = DEBUG: TEST writeFile, created object: /ebi/pride/test.replicate
Feb  6 11:29:47 pid:16308 NOTICE: +Testing Rule Number:0 for Action:acPreProcForModifyDataObjMeta
Feb  6 11:29:47 pid:16308 NOTICE: +Testing Rule Number:0 for Action:acPostProcForModifyDataObjMeta
Feb  6 11:29:47 pid:16308 NOTICE: +Testing Rule Number:0 for Action:acPostProcForOpen
Feb  6 11:29:47 pid:16308 NOTICE: +Testing Rule Number:0 for Action:acPostProcForPut
Feb  6 11:29:47 pid:16308 NOTICE: +Testing Rule Number:0 for Action:logDebug
Feb  6 11:29:47 pid:16308 NOTICE: writeLine: inString = DEBUG: TEST writeFile, outStatus: 0
Feb  6 11:29:47 pid:16308 NOTICE: readAndProcClientMsg: received disconnect msg from client
Feb  6 11:29:47 pid:16308 NOTICE: Agent exiting with status = 0
ccacciari commented 8 years ago

the control files have been removed from B2SAFE v3.0.0 so there in no need anymore to trigger a write operation.